JP2019153053A - Image processing device, image processing method and program - Google Patents

Abstract

To enable re-detection of a letter string by a further simple input operation when a desired letter string is undetectable from an image.SOLUTION: An image processing device according to the embodiment includes a receiving unit, a determining unit, a display control unit, and a display unit. The receiving unit receives input information input to the image. The determining unit specifies an input trace on the input information until a present time after the receiving of the input information starts, and determines a first letter string region that contains the input trace. The display control unit displays, while the receiving unit is receiving the input information, information indicating the first letter string region output by the determining unit for each unit time on the display unit.SELECTED DRAWING: Figure 1

Description

  Embodiments described herein relate generally to an image processing apparatus, an image processing method, and a program.

  2. Description of the Related Art Conventionally, a technique for detecting a character string described in a signboard, a sign, a restaurant menu, or the like from an image acquired by photographing with a camera built in a smartphone, a tablet, or the like is known. For example, a technique for detecting a character string from an image based on a user input operation on the image is conventionally known.

JP, 2006-004553, A JP, 2006-045877, A US Patent Application Publication No. 2011/0090253 JP 2006-119842 A International Publication No. 2017/029737

Tonouchi Y. , Suzuki K. Osada K .; , A Hybrid Approach to Detect Texts in Natural Scenes by Integration of a Connected-Component Method and a Sliding-Window Method, IWRR 2014 (SWR 2014) Shigeru Nagasawa, “Now Visual Konjac,“ Google Translate ”App Powers Up with“ Word Lens ””, [online], January 15, 2015, [Search September 4, 2017], Internet < URL: http://internet.watch.impress.co.jp/docs/news/6838329.html> “I tried using the powerful Google translation app!”, [Online], January 16, 2015, [September 4, 2017 search], Internet <URL: http: // diy -Ilands. com / 2015/01/16 / google-translate-usage /> “Discrimination analysis method (Otsu's binarization)”, [online], February 9, 2009, [Search September 4, 2017], Internet <URL: http: // imaginingsolution. blog107. fc2. com / blog-entry-113. html>

  However, in the conventional technique, when a desired character string cannot be detected from an image, the character string cannot be detected again with a simple operation. For example, when the user cannot detect a desired character string from the image, the user has to start the input operation again from the beginning.

  The image processing apparatus according to the embodiment includes a reception unit, a determination unit, a display control unit, and a display unit. The reception unit receives input information input to the image. The determination unit specifies an input locus of the input information from the start of reception of the input information to the current time, and determines a first character string region including the input locus. The display control unit displays information indicating the first character string region output at each unit time by the determination unit on the display unit while the input information is received by the reception unit.

FIG. 3 is a diagram illustrating an example of a functional configuration of the image processing apparatus according to the embodiment. The figure which shows the example of a function structure of the determination part of embodiment. The figure which shows the example of the input locus | trajectory and initial determination area | region of embodiment. The figure which shows the example of the result of the binarization process of the initial determination area | region of embodiment. The figure which shows the example of the peripheral region of embodiment. The figure which shows the example of the result of the binarization process of the peripheral region of embodiment. The figure which shows the example 1 of the display information of embodiment. The figure which shows the example 2 of the display information of embodiment. The figure which shows the example of a function structure of the determination part of the modification of embodiment. FIG. 3 is a diagram illustrating an example of a hardware configuration of the image processing apparatus according to the embodiment.

  Hereinafter, embodiments of an image processing apparatus, an image processing method, and a program will be described in detail with reference to the accompanying drawings.

  In conventional character recognition, a scanner captures a document image printed on paper as image data, and detects character strings arranged in a substantially horizontal or vertical direction and in a right-angle parallel and undistorted state. On the other hand, in the case of a character string in a camera image, the character string in the camera image may be greatly inclined or distorted depending on the relative positional relationship between the character string included as the subject and the camera. The character string in the camera image is significantly different from the character string in the image captured by the conventional scanner. Therefore, it is difficult to automatically detect a character string in a camera image using only image information.

  Furthermore, even if the entire character string region can be detected from the camera image, if there are a plurality of character strings in the camera image, it is difficult to specify the character string to which the user pays attention from the plurality of detection regions. is there. For this reason, an instruction indicating the position of the character string that the user is interested in is often received from the user by some means. For example, in a smartphone and a tablet with a touch panel, the user performs an operation of tracing a region where a character string is roughly displayed along a finger along the character string direction on a display screen displaying a camera image. After the user's instruction, a character string region of interest to the user is detected from the camera image based on the user's instruction information. When the user's instruction is a tracing operation, the position and line direction of the target character string can be roughly estimated.

  However, even in this case, since the detection process is performed after the user designates and the detection result is output, if the character string desired by the user cannot be detected by the detection process, the user again indicates the position of the character string. I have to do it.

  Hereinafter, embodiments of an image processing apparatus, an image processing method, and a program that can re-detect a character string with a simpler input operation even when a desired character string cannot be detected from the image will be described.

[Example of functional configuration]
FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing apparatus 10 according to the embodiment. The image processing apparatus 10 according to the embodiment includes a reception unit 1, a determination unit 2, a display control unit 3, and a display unit 4.

  The receiving unit 1 receives input information input to an image. The image may be arbitrary. The image is an image taken with a camera or the like, for example. Moreover, the input method of input information may be arbitrary. The input information input method is, for example, a user tap or mouse operation. The receiving unit 1 acquires the indicated position (x (k), y (k)) of the display unit 4 when input information is obtained in time series from a touch pad, a tablet, or the like. Here, k indicates the current time. The input trajectory from the time 0 when the input by the user is started to the current time k is a broken line connecting (x (i), y (i)) (i = 0,..., K).

  The determination unit 2 specifies an input trajectory of input information from the start of reception of input information to the current time, and determines a character string region (first character string region) including the input trajectory. The character string area includes, for example, characters, numbers, symbols, and the like. Details of the determination unit 2 will be described later with reference to FIG.

  While the input information is received by the receiving unit 1, the display control unit 3 displays information indicating the character string area output for each unit time by the determination unit 2 on the display unit 4. The unit time may be arbitrary. The unit time is, for example, 0.1 second, 0.5 second, 1 second, or the like. Moreover, the display part 4 may be arbitrary. The display unit 4 is a liquid crystal touch panel, for example.

  FIG. 2 is a diagram illustrating an example of a functional configuration of the determination unit 2 according to the embodiment. The determination unit 2 according to the embodiment includes a setting unit 21, an initial determination region determination unit 22, and a peripheral region determination unit 23. The determination unit 2 may be realized as one determination unit 2 without being divided into a plurality of functional blocks as shown in FIG. In the embodiment, in order to make the description of the operation of the determination unit 2 easy to understand, the determination unit 2 will be described by being divided into a plurality of functional blocks.

  The setting unit 21 sets an initial determination area including an input locus.

  FIG. 3 is a diagram illustrating an example of the input locus 101 and the initial determination region 102 according to the embodiment. The setting unit 21 sets an initial determination area 102 including the input locus 101. An input trajectory 101 indicates an input trajectory from the input start to the current time k. The shape of the initial determination area 102 may be arbitrary. In the example of FIG. 3, the initial determination area 102 is a rectangular area including the input locus 101. For example, the initial determination area 102 may be an area in a quadrangle in which a margin is added to a quadrilateral circumscribing at least one of the start point and the end point of the input trajectory 101.

  Returning to FIG. 2, the initial determination region determination unit 22 executes a process of binarizing pixels included in the initial determination region 102 based on the luminance value (pixel value) of the pixel. Specifically, first, the initial determination region determination unit 22 converts an image into a grayscale image. Next, the initial determination area determination unit 22 calculates a threshold value t using a binarization method such as Non-Patent Document 4 or the like. Next, the initial determination area determination unit 22 compares the luminance value of the pixel with a threshold value for each pixel in the initial determination area 102. When the pixel value is larger than the threshold value t, the initial determination area determination unit 22 determines that the pixel is a black pixel, that is, a character, and sets the pixel value to 1. On the other hand, when the pixel value is equal to or less than the threshold value t, the initial determination region determination unit 22 determines that the pixel is not a white pixel, that is, a character, and sets the pixel value to 0.

  FIG. 4 is a diagram illustrating an example of a result of the binarization process in the initial determination area 102 according to the embodiment. In the example of FIG. 4, the character string area (second character string area) included in the initial determination area 102 is represented by black pixels, and areas other than the character string areas are represented by white pixels. In addition, pixels in an area where the binarization processing has not been performed (an area outside the initial determination area 102) are represented with the pixel values of an image captured by a camera or the like.

  Returning to FIG. 2, the peripheral region determination unit 23 binarizes pixels in the peripheral region including the initial determination region 102 by executing the following processes 1 to 4.

[Process 1. The surrounding area determination unit 23 sets all pixels in the initial determination area 102 as the determined set C. Next, the process proceeds to process 2.

[Processing 2. The surrounding area determination unit 23 sets the pixels in the character string area in the initial determination area 102 as a determination target character pixel set M. Hereinafter, a pixel included in the character pixel set M to be determined is referred to as a character pixel p. Next, the process proceeds to process 3.

[Process 3. The surrounding area determination unit 23 extracts the character pixel p from the character pixel set M to be determined, and determines all the pixels q not included in the determined set C among the eight pixels around the character pixel p. In addition, the pixel determination process of the following process 3.1 is executed for each pixel q. The character pixel p extracted from the character pixel set M to be determined is deleted from the character pixel set M to be determined as a determined pixel. The surrounding area determination unit 23 executes the following process 4 when all eight pixels around the character pixel p are already included in the determined set C and there is no pixel q to be newly added to the determined set C. To do.

[Process 3.1. The peripheral area determination unit 23 determines whether or not the pixel q is a character pixel by comparing the luminance of the pixel q with a threshold value t. When the pixel q is determined to be a character pixel, the surrounding area determination unit 23 adds the pixel q to the determination target character pixel set M as a new character pixel p. Next, the process proceeds to process 4.

[Process 4. The peripheral area determination unit 23 ends the process when the character pixel set M to be determined is an empty set, and executes the above-described process 3 when it is not an empty set.

  FIG. 5 is a diagram illustrating an example of the peripheral region 103 according to the embodiment. The peripheral area determination unit 23 binarizes the pixels in the peripheral area 103 of the initial determination area 102 by the above-described processes 1 to 4, so that the area other than the character string area (second character string area) overlapping the initial determination area 102 It is determined whether or not a pixel representing the character string area (third character string area) is in the peripheral area 103.

  FIG. 6 is a diagram illustrating an example of a result of the binarization processing of the peripheral area 103 according to the embodiment. In the example of FIG. 6, the character string region is represented by black pixels, and the region other than the character string region is represented by white pixels. In addition, pixels in an area where the binarization processing has not been performed (an area outside the peripheral area 103) are represented with the pixel values of an image captured by a camera or the like.

  The example of FIG. 6 shows the result of the binarization process at the current time k. In the determination process at the current time k, the determination result obtained by the determination process up to time k−1 may be used. When the determination result obtained by the determination process up to time k−1 is used, the surrounding area determination unit 23 sets the character pixel set N () indicating the pixels in the character string area obtained by the determination process at time k−1. k-1) and the determined set C (k-1) at the end of the determination process at time k-1. Then, in the determination process at the current time k, the surrounding area determination unit 23 uses the determination result at the time k−1 for pixels included in the determined set C (k−1) among the pixels in the surrounding area 103. Pixels included in a certain character pixel set N (k−1) are determined as character pixels, and pixels not included in the character pixel set N (k−1) are determined as pixels other than characters. In addition, the peripheral region determination unit 23 binarizes the pixels in the peripheral region 103 that are not included in the determined set C (k−1) by the above-described processes 1 to 4, thereby generating the peripheral region. In step 103, it is determined whether there is a pixel representing a character string area.

  With the configuration of the image processing apparatus 10 described above, a character string area (for example, a value) with respect to a local area including the tracing position at the current time k while the user performs a tracing operation on the area of the character string of interest in the image. 1) and a background area (for example, value 0) can be binarized.

<Example of display information>
FIG. 7 is a diagram illustrating an example 1 of display information according to the embodiment. The display control unit 3 changes the display color of the character string region and the display color of the region other than the character based on the binary image output from the determination unit 2, thereby causing the display unit 4 to process the determination unit 2. Display the results. A binary image is an image in which, for example, the pixel value of a pixel in a character string region is set to 1, and the pixel value of a pixel that is not a character string region is set to 0. In the example of FIG. 7, the character string area is a pixel area representing “kil”. In the example of FIG. 7, the display control unit 3 displays the character string region on the display unit 4 in a first color (for example, black), and the region other than the character string region in the peripheral region 103 is displayed in the second color ( For example, white is displayed on the display unit 4.

  FIG. 8 is a diagram illustrating a second example of display information according to the embodiment. In the example of FIG. 8, the display control unit 3 further displays the input locus 101 on the display unit 4 in a third color (for example, red).

  The first to third colors described above may be arbitrary. Further, the display control unit 3 may display the display information in which the color is changed only in a part of the area on the display unit 4. For example, the display control unit 3 may display the display information in which only the color of the character string area is changed on the display unit 4. For example, the display control unit 3 may display display information in which only the color of the region other than the character string region in the peripheral region 103 is changed on the display unit 4.

  As described above, in the image processing apparatus 10 according to the embodiment, the reception unit 1 receives input information input to an image. The determination unit 2 specifies the input trajectory 101 of input information from the start of acceptance of input information to the current time k, and determines a character string region (first character string region) including the input trajectory 101. Then, while the input information is received by the receiving unit 1, the display control unit 3 displays information indicating the character string area output for each unit time by the determination unit 2 on the display unit 4.

  Thereby, according to the image processing apparatus 10 of the embodiment, when a desired character string cannot be detected from the image, the character string can be detected again by a simpler input operation. Specifically, for example, the user can check the progress of the process in real time even during the tracing operation. Further, for example, when the user notices that the character string region desired by the user has not been correctly detected, desirable detection processing is performed by sequentially correcting the stroking position or interrupting the operation and restarting the operation. In this way, corrections can be made during the operation.

(Modification)
Next, a modification of the embodiment will be described. In the description of the modification of the embodiment, the description similar to that of the above-described embodiment will be omitted, and different points from the above-described embodiment will be described.

  In the modification of the embodiment, a case will be described in which an inverted character may be included in an image, unlike a normal case. In an area including inverted characters, a pixel having a low luminance value (for example, a black pixel) indicates the background, and a pixel having a high luminance value (for example, a white pixel) indicates the character (inverted character).

  FIG. 9 is a diagram illustrating an example of a functional configuration of the determination unit 2-2 according to a modification of the embodiment. The determination unit 2-2 of the embodiment includes a setting unit 21, an initial determination region determination unit 22, a peripheral region determination unit 23, and an inversion determination unit 24. That is, in the modification of the embodiment, the inversion determination unit 24 is further added to the configuration of the above-described embodiment (see FIG. 2).

  The inversion determination unit 24 determines whether a pixel whose luminance value is higher than the luminance value of surrounding pixels is a pixel indicating a character string region, and has a higher luminance value than the luminance value of surrounding pixels. When the pixel is a pixel indicating a character string area, the luminance value of the image is inverted before the binarization process is executed.

  Specific processing of the inversion determination unit 24 will be described by taking as an example a case where a pixel having a low luminance value is a black pixel and a pixel having a high luminance value is a white pixel. In other words, the inversion determination unit 24 indicates that the character indicated by the pixel having a low luminance value is a black character, the pixel having a high luminance value is a white background (white background), or the character indicated by the pixel having a high luminance value is a white character. An example of determining whether a pixel having a low luminance value is a black background (black background) (whether it is an inverted character) will be described.

  Specifically, the inversion determination unit 24 first determines whether or not the area including the initial determination area 102 includes inverted characters. The inversion determination unit 24 inverts the luminance value of the image when an inversion character is included in the area including the initial determination area 102. When the luminance value of the image is inverted, the brighter pixels become darker and the darker pixels become brighter.

  Here, a method for determining whether the background is a white background and black characters or an inverted character (background is black and white characters) will be described. The inversion determination unit 24 obtains the ratio of white pixels (number of white pixels / total number of pixels) included in each of the initial determination region 102 and the surrounding region of the initial determination region 102. The surrounding area here is, for example, an area obtained by removing the initial determination area 102 from an area obtained by enlarging the initial determination area 102 at a constant magnification around the center of gravity of the initial determination area 102. When the ratio of white pixels is greater in the initial determination area 102 than in the surrounding area, the inversion determination unit 24 determines that the area including the initial determination area 102 includes inverted characters.

  According to the image processing apparatus 10 of the modified example of the embodiment, the same effect as that of the above-described embodiment can be obtained even when an inverted character is included in the image.

  Finally, an example of a hardware configuration of the image processing apparatus 10 according to the embodiment will be described.

[Example of hardware configuration]
FIG. 10 is a diagram illustrating an example of a hardware configuration of the image processing apparatus 10 according to the embodiment. The image processing apparatus 10 according to the embodiment includes a control device 301, a main storage device 302, an auxiliary storage device 303, a display device 304, an input device 305, a communication device 306, and an imaging device 307. A control device 301, a main storage device 302, an auxiliary storage device 303, a display device 304, an input device 305, a communication device 306, and an imaging device 307 are connected via a bus 310.

  The control device 301 executes the program read from the auxiliary storage device 303 to the main storage device 302. The control device 301 is one or more processors such as a CPU, for example. The reception unit 1, the determination unit 2, and the display control unit 3 described above are realized by the control device 301, for example. The main storage device 302 is a memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The auxiliary storage device 303 is a memory card, an HDD (Hard Disk Drive), or the like.

  The display device 304 displays information. The display device 304 is a liquid crystal display, for example. The above-described display unit 4 is realized by the display device 304, for example. The input device 305 receives input of information. The input device 305 is, for example, a keyboard and a mouse. Note that the display device 304 and the input device 305 may be a liquid crystal touch panel having both a display function and an input function. The communication device 306 communicates with other devices. The imaging device 307 captures an image such as a scene image.

  A program executed by the image processing apparatus 10 according to the embodiment is a file in an installable format or an executable format, and is read by a computer such as a CD-ROM, a memory card, a CD-R, and a DVD (Digital Versatile Disk). It is stored in a possible storage medium and provided as a computer program product.

  The program executed by the image processing apparatus 10 according to the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed by the image processing apparatus 10 according to the embodiment may be provided via a network such as the Internet without being downloaded.

  Further, the program executed by the image processing apparatus 10 according to the embodiment may be configured to be provided by being incorporated in advance in a ROM or the like.

  The program executed by the image processing apparatus 10 according to the embodiment has a module configuration including functions that can be realized by the program among the functional configurations of the image processing apparatus 10 according to the embodiment.

  Functions realized by the program are loaded into the main storage device 302 by the control device 301 reading the program from a storage medium such as the auxiliary storage device 303 and executing the program. That is, the function realized by the program is generated on the main storage device 302.

  Note that some of the functions of the image processing apparatus 10 according to the embodiment may be realized by hardware such as an IC (Integrated Circuit). The IC is a processor that executes dedicated processing, for example.

  When each function is realized using a plurality of processors, each processor may realize one of the functions or two or more of the functions.

  The operation mode of the image processing apparatus 10 according to the embodiment may be arbitrary. The image processing apparatus 10 according to the embodiment may be operated as a cloud system on a network, for example.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 1 Reception part 2 Determination part 3 Display control part 4 Display part 10 Image processing apparatus 21 Setting part 22 Initial determination area | region determination part 23 Peripheral area | region determination part 24 Inversion determination part 301 Control apparatus 302 Main storage apparatus 303 Auxiliary storage apparatus 304 Display apparatus 305 Input device 306 Communication device 307 Imaging device 310 Bus

Claims (9)

A reception unit for receiving input information input to the image;
A determination unit that identifies an input locus of the input information from the start of reception of the input information to a current time, and determines a first character string region including the input locus;
A display control unit that displays on the display unit information indicating the first character string region that is output every unit time by the determination unit while the input information is received by the reception unit;
An image processing apparatus comprising: The determination unit determines an area other than the first character string area and the first character string area based on a binarization process that indicates the pixel as a binary value based on a luminance value of a pixel included in the image. To
The image processing apparatus according to claim 1. The determination unit sets an initial determination region including the input locus, determines the second character string region included in the initial determination region by executing the binarization process in the initial determination region, and By performing the binarization process on a peripheral area of the second character string area, a third character string area included in the peripheral area is determined, and the second character string area, the third character string area, To determine the first character string region,
The image processing apparatus according to claim 2. The determination unit determines whether a pixel whose luminance value is higher than the luminance value of surrounding pixels is a pixel indicating a character string region, and has a higher luminance value than the luminance value of surrounding pixels. Is a pixel indicating a character string region, the luminance value of the image is inverted before executing the binarization process.
The image processing apparatus according to claim 2. The display control unit displays the first character string region on the display unit in a first color.
The image processing apparatus according to claim 2. The display control unit displays a region other than the first character string region in the peripheral region including the first character string region on the display unit in a second color.
The image processing apparatus according to claim 5. The display control unit displays the input locus in a third color on the display unit;
The image processing apparatus according to claim 6. Receiving input information entered in the image;
Identifying an input trajectory of the input information from the start of acceptance of the input information to the current time, and determining a first character string region including the input trajectory;
Displaying the information indicating the first character string area output at each unit time on the display unit while the input information is received;
An image processing method including: Computer
A reception unit for receiving input information input to the image;
A determination unit that identifies an input locus of the input information from the start of reception of the input information to a current time, and determines a first character string region including the input locus;
While the input information is received by the receiving unit, a display control unit that displays on the display unit information indicating the first character string region that is output at every unit time by the determination unit;
Program to function as.