JP2014186549A - Image processor, image processing program and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a search window with high accuracy.SOLUTION: An image processor 100 acquires image data of a television program or the like, and generates edge image data 122 by extracting an edge from the image data. The image processor 100 extracts horizontal straight lines from the edge image data 122. The image processor 100 extracts a black pixel connection area obtained by connecting edges in the edge image data 122, and determines, as a search window, a combination of straight lines having the black pixel connection area within a predetermined range between combinations of the plurality of straight lines.

Description

  The present invention relates to an image processing apparatus and the like.

  A search window may be displayed in a TV program CM (Commercial Message) or the like. For example, the user can acquire more information regarding the contents of the CM by remembering the keyword displayed in the search window and searching the keyword using a PC (Personal Computer) or the like. Can do.

  In recent years, there are various techniques for detecting a search window from a still image and recognizing a character string in the search window. Here, as an example, the prior art 1 and the prior art 2 will be described.

  Prior art 1 extracts an edge from an input image, and extracts a rectangular region surrounded by the outline of the edge. The conventional technique 1 determines an area where the variance of pixel values within a rectangular area is equal to or less than a threshold value as a search window, and binarizes the area of the search window to perform character recognition.

  Prior art 2 extracts line segments of various angles by performing Radon transform on the edge image, and extracts a line segment set that forms a closed region using the adjacency relation of the line segments. Then, the related art 2 approximates the line segment set forming the closed region to a polygon, and determines the region corresponding to the polygon as a search window when the polygon satisfies a predetermined condition. For example, the conventional technique 2 selects the longest two of the sides constituting the polygon, the selected line segment is parallel to the image, and the entire polygon is a horizontally long region. Let the corresponding area be the search window.

JP 2012-160023 A JP 2002-015283 A JP-A-8-317195 JP 2009-44658 A JP 2009-188886 A JP 2010-152800 A

  However, the above-described conventional technique has a problem that the search window cannot be detected with high accuracy.

  For example, in the conventional techniques 1 and 2, if only a strong edge is extracted, the edge outline may be interrupted, the closed region cannot be extracted, and the search window cannot be extracted. In addition, when edge extraction is performed including not only strong edges but also weak edges, edges are extracted not only from edges corresponding to the search window but also from keywords in the background area and search window, and the correct search window is identified. I can't.

  An object of one aspect is to provide an image processing apparatus, an image processing program, and an image processing method that can detect a search window with high accuracy.

  In the first plan, the image processing apparatus includes a straight line extraction unit and a determination unit. The straight line extraction unit extracts a plurality of horizontal lines based on an edge image obtained by extracting edges from the image. The determination unit extracts a black pixel connection region obtained by connecting edges in the edge image, and determines a combination of straight lines having a black pixel connection region within a predetermined range between a plurality of straight line combinations.

  According to the embodiment of the present invention, the image processing apparatus can detect the search window with high accuracy.

FIG. 1 is a diagram illustrating an example of a configuration of a TV according to the present embodiment. FIG. 2 is a diagram illustrating the configuration of the image processing apparatus according to the present embodiment. FIG. 3 is a diagram illustrating an example of image data. FIG. 4 is a diagram illustrating an example of edge image data. FIG. 5 is a diagram illustrating an example of the data structure of the straight line table. FIG. 6 is a diagram illustrating an example of the data structure of the black pixel connection area table. FIG. 7 is a diagram for explaining processing of the straight line extraction unit. FIG. 8 is a diagram illustrating an example of each straight line extracted by the straight line extraction unit. FIG. 9 is a diagram for explaining the processing of the black pixel connection region extraction unit. FIG. 10 is a diagram (1) for explaining the processing of the determination unit. FIG. 11 is a diagram illustrating an example of correspondence management information generated by the determination unit. FIG. 12 is a diagram (2) for explaining the processing of the determination unit. FIG. 13 is a diagram (3) for explaining the processing of the determination unit. FIG. 14 is a diagram (4) for explaining the processing of the determination unit. FIG. 15 is a diagram (5) for explaining the processing of the determination unit. FIG. 16 is a diagram (6) for explaining the processing of the determination unit. FIG. 17 is a diagram (7) for explaining the processing of the determination unit. FIG. 18 is a diagram (8) for explaining the process of the determination unit. FIG. 19 is a diagram (9) for explaining the processing of the determination unit. FIG. 20 is a diagram (10) for explaining the processing of the determination unit. FIG. 21 is a flowchart illustrating the processing procedure of the image processing apparatus according to the present embodiment. FIG. 22 is a flowchart illustrating a processing procedure of search window area detection processing according to the present embodiment. FIG. 23 is a flowchart illustrating a processing procedure of straight line pair extraction processing. FIG. 24 is a flowchart illustrating a processing procedure for selecting a straight line pair based on the positional relationship between the black pixel connection region and the straight line pair. FIG. 25 is a flowchart showing a processing procedure for executing character recognition on an area between straight line pairs and selecting a straight line pair based on the certainty of character recognition. FIG. 26 is a flowchart showing a processing procedure for selecting a straight line pair based on a specific character. FIG. 27 is a diagram for explaining another embodiment of the image processing apparatus. FIG. 28 is a diagram illustrating an example of a computer that executes a control program.

  Embodiments of an image processing apparatus, an image processing program, and an image processing method disclosed in the present application will be described below in detail with reference to the drawings. Note that the present invention is not limited to the embodiments.

  An example of the configuration of the TV according to the present embodiment will be described. FIG. 1 is a diagram illustrating an example of a configuration of a TV according to the present embodiment. As shown in FIG. 1, the TV 50 includes a tuner 51, a monitor 52, a communication unit 53, a remote control reception unit 54, a TV control unit 55, and an image processing device 100. The tuner 51, the monitor 52, the communication unit 53, the remote control receiving unit 54, the TV control unit 55, and the image processing apparatus 100 are electrically connected to each other via a connection line 60. The TV 50 is connected to various servers (not shown) via the network 10. Since other configurations are the same as those of the well-known TV, description thereof is omitted here.

  The tuner 51 is a device that receives a television broadcast using the antenna 5. The tuner 51 outputs TV broadcast information to the TV control unit 55.

  The monitor 52 is a display device that displays information on the television broadcast output from the TV control unit 55. The monitor 52 displays information output from the image processing apparatus 100. The information output from the image processing apparatus 100 includes a Web search result of a keyword in a search window displayed in a TV program CM or the like.

  The communication unit 53 is a processing unit that performs data communication with various servers (not shown) via the network 10. The image processing apparatus 100 described later exchanges data with an external server or the like via the communication unit 53. The communication unit 53 corresponds to a communication device such as a NIC (Network Interface Card).

  The remote control receiving unit 54 is a device that receives information from a remote controller (remote controller) operated by a user. The remote control receiving unit 54 outputs the received information to the TV control unit 55. When receiving a search request regarding the keyword displayed in the search window, the remote control receiving unit 54 outputs the search request to the image processing apparatus 100.

  The TV control unit 55 is a device that controls the entire TV 50. For example, the TV control unit 55 outputs the program of the channel designated by the remote controller from the information of each TV program received by the tuner 51 to the monitor 52 for display. Further, the TV control unit 55 outputs the video data of the television program displayed on the monitor 52 to the image processing apparatus 100.

  The image processing apparatus 100 is an apparatus that extracts a search window displayed in a television program. Further, the image processing apparatus 100 performs a Web search for keywords included in the search window and causes the monitor 52 to display the search results.

  An example of the configuration of the image processing apparatus 100 illustrated in FIG. 1 will be described. FIG. 2 is a diagram illustrating the configuration of the image processing apparatus according to the present embodiment. As illustrated in FIG. 2, the image processing apparatus 100 includes an interface unit 110, a storage unit 120, and a control unit 130.

  The interface unit 110 is a processing unit that executes data communication with the TV control unit 55, for example. The interface unit 110 performs data communication with a server on the network 10 via the communication unit 53. The control unit 130 described later exchanges data with the TV control unit 55, the server, and the like via the interface unit 110.

  The storage unit 120 stores image data 121, edge image data 122, a straight line table 123, a black pixel connection area table 124, and search window area data 125. The storage unit 120 corresponds to a storage device such as a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), and a flash memory.

  The image data 121 is still image data extracted from video data of a television program. FIG. 3 is a diagram illustrating an example of image data.

  The edge image data 122 is image data generated by extracting an edge from the image data 121. FIG. 4 is a diagram illustrating an example of edge image data. The image 122b is an enlargement of the region 122a of the edge image 122.

  The straight line table 123 is a table that holds horizontal straight line information extracted from the edge image data 122. FIG. 5 is a diagram illustrating an example of the data structure of the straight line table. As shown in FIG. 5, the straight line table 123 associates straight line identification information, start point coordinates, and end point coordinates. The straight line identification information is information that uniquely identifies a horizontal straight line extracted from the edge image data 122. The start point coordinates indicate the start point coordinates of a straight line in the horizontal direction. The end point coordinate indicates the end point coordinate of a straight line in the horizontal direction.

  The black pixel connection area table 124 is a table that holds information on black pixel connection areas in which edges of edge image data are connected. For example, the black pixel connection area is approximated to a rectangle. FIG. 6 is a diagram illustrating an example of the data structure of the black pixel connection area table. As shown in FIG. 6, this black pixel connection area table 124 associates black pixel identification information, upper left rectangular coordinates, and lower right rectangular coordinates. Black pixel identification information is information that uniquely identifies a black pixel connection region. The upper left rectangular coordinates indicate the coordinates of the upper left corner of the black pixel connection area. The lower right rectangular coordinates indicate the coordinates of the lower right corner of the black pixel connection area.

  The search window area data 125 includes information indicating a search window area on the image data 121 detected by the search window detection unit 133 described later. For example, the search window area data 125 corresponds to the coordinates of the search window on the screen data 121.

  The control unit 130 includes an acquisition unit 131, a conversion unit 132, a search window detection unit 133, a character recognition processing unit 134, and a web search unit 135. The control unit 130 corresponds to an integrated device such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 130 corresponds to an electronic circuit such as a CPU or MPU (Micro Processing Unit).

  Note that the control unit 130 may start the process when the search request is acquired from the remote control reception unit 54.

  The acquisition unit 131 is a processing unit that acquires video data from the TV control unit 55. The acquisition unit 131 extracts image data to be a still image from the acquired video data, and stores the extracted image data in the storage unit 120.

  The conversion unit 132 is a processing unit that generates the edge image data 122 by extracting an edge from the image data 121. For example, the conversion unit 132 converts the image data 121 into grayscale image data. Then, the conversion unit 132 generates the edge image data 122 by applying an algorithm such as Canny Edge to the grayscale image data.

  The search window detection unit 133 is a processing unit that detects a search window. The search window detection unit 133 includes a straight line extraction unit 133a, a black pixel connection region extraction unit 133b, and a determination unit 133c.

  The straight line extraction unit 133 a is a processing unit that extracts a straight line in the horizontal direction from the edge image data 122. The straight line extraction unit 133 a registers the extracted straight line information in the straight line table 123.

  The process of the straight line extraction unit 133a will be specifically described. The straight line extraction unit 133a obtains a horizontal straight line by integrating the small line segments after extracting the small line segments from the edge image data 122. A process for extracting a small line segment will be described.

  FIG. 7 is a diagram for explaining processing of the straight line extraction unit. The straight line extraction unit 133 a sets a strip area having a width of 32 pixels for the edge image data 122. The straight line extraction unit 133a creates a horizontal projection histogram p (i) of black pixels for each y coordinate of each strip area. i represents the y coordinate. For example, for the y coordinate (y1) of the strip area, when the number of black pixels is 30, the value of p (y1) is 30.

  The straight line extraction unit 133a extracts a short line segment having a strip width of y-coordinate i when any one of the following straight line extraction conditions 1 to 3 is satisfied. Line extraction condition 1 is a condition that the value of p (i) is 30 or more. The straight line extraction condition 2 is a condition that the value obtained by adding the value of p (i) and the value of p (i + 1) is 30 or more. The straight line extraction condition 3 is a condition that the value obtained by adding the value of p (i), the value of p (i + 1), and the value of p (i−1) is 30 or more. Even if the straight line of the edge is slightly deviated from the horizontal direction, the straight line can be relieved by the straight line extraction conditions 2 and 3.

  For example, for a certain y coordinate i in the strip area 1a of FIG. 7, when any one of the straight line extraction conditions 1 to 3 is satisfied, the straight line extraction unit 133a sets the straight line 1b on the y coordinate i in the strip area 1a. To extract. The straight line extraction unit 133a extracts a small line segment by repeatedly executing the same processing for other strip regions.

  The straight line extracting unit 133a obtains a horizontal straight line by extracting the small line segments and then integrating the small line segments in the horizontal direction. For example, the straight line extraction unit 133a compares the start point coordinates and end point coordinates of each small line segment, and integrates each small line segment that is continuous in the horizontal direction. FIG. 8 is a diagram illustrating an example of each straight line extracted by the straight line extraction unit. The straight line extraction unit 133 a registers the information of each integrated straight line in the straight line table 123.

  Returning to the description of FIG. The black pixel connection region extraction unit 133b is a processing unit that extracts black pixel connection regions by connecting the edges included in the edge image data 122. For example, the black pixel connection region extraction unit 133b performs labeling on the edge image data 122 and obtains a circumscribed rectangle of the same label lump. The black pixel connection area extraction unit 133 b registers information on the black pixel connection area approximated by a circumscribed rectangle in the black pixel connection area table 124.

  FIG. 9 is a diagram for explaining the processing of the black pixel connection region extraction unit. As shown in FIG. 9, the black pixel connection area extraction unit 133b executes a labeling process, whereby a plurality of circumscribed rectangles are extracted.

  The determination unit 133c is a processing unit that determines the region of the search window based on the straight line table 123 and the black pixel connection region table 124. The determination unit 133c registers the determined search window area information in the search window area data 125. Hereinafter, the processing of the determination unit 133c will be specifically described. The determination unit 133c executes processing for associating a straight line with a black pixel connection region, processing for determining a straight line pair, and processing for determining a straight line pair constituting the search window. In addition, the determination unit 133 executes processing for extracting candidate regions.

  A process in which the determination unit 133c associates a straight line with a black pixel connection region will be described. FIG. 10 is a diagram (1) for explaining the processing of the determination unit. First, the determination unit 133c determines a set of black pixel connection regions that have the same distance from the straight line, and associates them with each other. For example, in the example shown in FIG. 10, the distances between the straight line 2a and the black pixel connection regions 3a, 3b, 3c, 3d, 3e, and 3f are the same. For this reason, the determination unit 133c associates the straight line 2a with the black pixel connection regions 3a to 3f. The distances between the straight line 2a and the black pixel connection regions 4a, 4b, 4c, 4d, and 4e are the same. For this reason, the determination unit 133c associates the straight line 2a with the black pixel connection regions 4a to 4f.

  The determination unit 133c registers the associated straight line identification information and the black pixel identification information of each black pixel connection region in association management information in association with each other. FIG. 11 is a diagram illustrating an example of correspondence management information generated by the determination unit.

  A process in which the determination unit 133c determines a straight line pair will be described. For example, based on the straight line table 123 and the black pixel connection region table 124, the determination unit 133c overlaps in the y direction when the length difference between the straight lines is less than a predetermined threshold, and connects the black pixels between the straight lines. When a region exists, each straight line is paired. The determination unit 133c registers information on each straight line pair in the pair management information. Although illustration is omitted here, the pair management information is information associated with the straight line identification information of each straight line forming a pair.

  A process in which the determination unit 133c determines a pair of straight lines constituting the search window will be described. FIG. 12 is a diagram (2) for explaining the processing of the determination unit. The determination unit 133c selects the pair of straight lines associated with each other, and when the black pixel connection region associated with the selected straight line satisfies a predetermined condition, the determination unit 133c selects the pair of straight lines that form the search window. Judge as a pair. Specifically, when the condition shown in Expression (1) is satisfied, the selected straight line pair is determined as a straight line pair constituting the search window. The threshold value of the formula (1) is a value set by the user as appropriate.

  Of the black pixel connection areas associated with the straight line pairs, the number of black pixel connection areas common to the straight line pairs / the number of black pixel connection areas associated with the straight line pairs (however, the overlapping black pixel connection areas are Count as 1)> Threshold ... (1)

  The contents of equation (1) will be described with reference to FIG. In FIG. 12, the straight line 2a and the straight line 2b form a straight line pair. The black pixel connection areas 4a, 4b, 4c, 4d, and 4e are associated with the straight line 2a. The black pixel connection areas 4a, 4b, 4c, 4e, and 4f are associated with the straight line 2b. Further, the black pixel connection areas common to the straight line pairs are black pixel connection areas 4a, 4b, 4c, and 4e. For this reason, in the example illustrated in FIG. 12, among the black pixel connection regions associated with the straight line pair, the number of black pixel connection regions common to the straight line pair is “4”, and the black pixels associated with the straight line pair The number of connected areas is “6”. That is, the black pixel connection regions 4f and 4d are further associated with the straight line pair.

  A process in which the determination unit extracts candidate areas will be described. FIG. 13 is a diagram (3) for explaining the processing of the determination unit. It is assumed that the straight line pairs 2a and 2b shown in FIG. 13 are determined as straight line pairs constituting the search window by the above processing. The determination unit 133c extracts a region 3a surrounding the black pixel connection regions 4a to 4f in the region in the straight line pairs 2a and 2b as a candidate region.

  FIG. 14 is a diagram (4) for explaining the processing of the determination unit. The determination unit 133c integrates the two candidate areas when the two extracted candidate areas have substantially the same y-coordinate, are adjacent, and the y-direction positions of the line segment and the black pixel connection area are substantially the same. For example, in the example shown in FIG. 14, the difference between the upper left and lower right y coordinates of the two candidate areas 3a and 3b is less than the threshold, and the difference between the y coordinates of the straight line 2a and the straight line 2c is less than the threshold. The difference between the y-coordinates of the straight line 2b and the straight line 2d is less than the threshold value. Further, it is assumed that the difference between the y-direction position of the center of gravity of the black pixel connection areas 4a to 4f and the y-direction position of the center of gravity of the black pixel connection areas 4g to 4l is less than the threshold value. In this case, the determination unit 133c integrates the candidate areas 3a and 3b to generate the candidate area 3c.

  FIG. 15 is a diagram (5) for explaining the processing of the determination unit. Here, a description will be given using the candidate areas 3d and 3e. The determination unit 133c searches for a peak point with a black pixel projection histogram in the left direction from the left end of the candidate area 3d, starting from a point of 32 pixels leftward, and sets the peak point as the left end of the candidate area. Further, a point of 32 pixels in the right direction from the right end of the candidate area 3d is used as a start point, a peak point is searched for in the black pixel projection histogram in the vertical direction in the left direction, and the peak point is set as the right end of the candidate area. By executing the processing related to the determination unit 133c, the candidate area 3d becomes the candidate area 3f. The determination unit 133c performs the same process on the candidate region 3e, so that the candidate region 3e becomes the candidate region 3g.

  FIG. 16 is a diagram (6) for explaining the processing of the determination unit. Here, a description will be given using the candidate areas 3h and 3i. The determination unit 133c scans the area inside one pixel of the four sides of the candidate area, and counts the number of times of change from the white pixel to the black pixel or from the black pixel to the white pixel. The determination unit 133c deletes the corresponding candidate area when the number of changes is equal to or greater than the threshold. For example, regarding the candidate area 3h, when the area inside one pixel on the four sides of the candidate area is scanned, the number of changes is less than the threshold. For this reason, the determination unit 133c leaves the candidate area 3h. On the other hand, for the candidate area 3i, when the area inside one pixel of the four sides of the candidate area is scanned, the number of changes becomes equal to or greater than the threshold. For this reason, the determination unit 133c deletes the candidate area 3i.

  FIG. 17 is a diagram (7) for explaining the processing of the determination unit. The determination unit 133c integrates candidate areas with similar coordinates. Here, a description will be given using candidate areas 3j and 3k. Also, among the x coordinates of the candidate area 3j, the minimum x coordinate is xmin1, and the maximum x coordinate is xmax1. Among the y coordinates of the candidate area 3j, the minimum y coordinate is ymin1, and the maximum y coordinate is ymax1. Further, among the x coordinates of the candidate area 3k, the minimum x coordinate is xmin2, and the maximum x coordinate is xmax2. Among the y coordinates of the candidate area 3j, the minimum y coordinate is ymin2, and the maximum y coordinate is ymax2.

  The determination unit 133c integrates the candidate areas 3j and 3k when the condition shown in Expression (2) is satisfied. For example, the determination unit 133c generates the candidate region 3l by integrating the candidate regions 3j and 3k.

  | Xmin1-xmin2 | + | xmax1-xmax2 | + | ymin1-ymin2 | + | ymax1-ymax2 | <threshold value (2)

  FIG. 18 is a diagram (8) for explaining the process of the determination unit. The determination unit 133c divides the candidate region when the peak point of the black pixel projection histogram in the vertical direction exists at a certain distance from the left and right end points in the candidate region. Here, a description will be given using the candidate area 3m. For example, when the peak point 6a of the black pixel projection histogram in the vertical direction exists in the candidate area 3m, the determination unit 133c determines the candidate area 3m as a reference based on the position of the peak point 6a. Divide into

  FIG. 19 is a diagram (9) for explaining the processing of the determination unit. The determination unit 133c performs character recognition on each black pixel connection area included in the candidate area. The determination unit 133c deletes the candidate area when the average value of the certainty factor of each character recognition result is less than the threshold value. Here, description will be made using candidate regions 3p and 3q. Note that any conventional technique may be used as the technique for performing character recognition on the black pixel connection area and the technique for calculating the certainty factor. For example, the determination unit 133c performs character recognition on the candidate region 3p, and leaves the candidate region 3p when the average value of the certainty factor of each character recognition result is equal to or greater than a threshold value.

  Note that the determination unit 133c deletes the candidate area if the candidate area includes a specific character even if the certainty factor of each character recognition result is equal to or greater than the threshold. For example, the candidate area is deleted when the characters recognized by character recognition include the characters “verification”, “search”, or “Q”. For example, the character recognition result for the magnifying glass mark indicating the meaning of the search may be “Q”. When character recognition is performed on the candidate area 3q, since the characters “verification” and “search” are included, the determination unit 133c deletes the candidate area 3q. Note that the determination unit 133c records the coordinates of the candidate area including “check”, “search”, or “Q”.

  FIG. 20 is a diagram (10) for explaining the processing of the determination unit. The determination unit 133c may execute the process described below when there are a plurality of candidate areas by the above process. When there is a region including the characters “detect”, “search”, or “Q”, the determination unit 133c detects the closest region on the left and right of the region as a search window. For example, in the example shown in FIG. 10, “search” is included in the region 3r, and the region closest to the left and right of the region 3r is the candidate region 3s. Therefore, the determination unit 133c detects the candidate area 3r as a search window area. The determination unit 133c registers the detected search window information in the search window area data 125.

  Returning to the description of FIG. The character recognition processing unit 134 is a processing unit that performs character recognition on the search window area and outputs a keyword that is a character recognition result to the Web search unit 135. The character recognition processing unit 134 specifies a search window area on the edge image data based on the search window coordinates registered in the search window area data, and performs character recognition on the specified area. Any conventional technique may be used for performing character recognition on the area.

  The Web search unit 135 is a processing unit that executes a Web search for a keyword acquired from the character recognition processing unit 134. For example, the web search unit 135 accesses a server connected to the network 10 and executes a web search. The web search unit 135 displays the search result on the monitor 52.

  Next, a processing procedure of the image processing apparatus 100 according to the present embodiment will be described. FIG. 21 is a flowchart illustrating the processing procedure of the image processing apparatus according to the present embodiment. For example, the process shown in FIG. 21 is executed when a search request is accepted.

  As shown in FIG. 21, the image processing apparatus 100 acquires one frame of image data from video data (step S101), and executes search window area detection processing (step S102).

  The image processing apparatus 100 executes a character recognition process on the search window area (step S103), and executes a web search (step S104). Then, the image processing apparatus 100 outputs the search result (step S105).

  Next, the search window area detection process shown in FIG. 21 will be described. FIG. 22 is a flowchart illustrating a processing procedure of search window area detection processing according to the present embodiment. As shown in FIG. 22, the image processing apparatus 100 generates edge image data 122 from the image data 121 (step S110). The image processing apparatus 100 extracts a black pixel connection area (step S111).

  The image processing apparatus 100 extracts a horizontal straight line from the edge image data (step S112). The image processing apparatus 100 executes straight line pair extraction processing (step S113). The image processing apparatus 100 selects only a pair in which black pixels exist between the straight line pairs among the straight line pairs (step S114). The image processing apparatus 100 determines whether or not the selected straight line pair is one (step S115). If the selected straight line pair is one (step S115, Yes), the image processing apparatus 100 proceeds to step S121.

  On the other hand, if there are two or more selected straight line pairs (step S115, No), the image processing apparatus 100 proceeds to step S116. The image processing apparatus 100 executes a process of selecting a straight line pair based on the positional relationship between the black pixel connection region and the straight line pair (step S116). The image processing apparatus 100 determines whether or not the selected straight line pair is one (step S117). When the selected straight line pair is one (step S117, Yes), the image processing apparatus 100 proceeds to step S121.

  On the other hand, when the number of selected straight line pairs is two or more (step S117, No), the image processing apparatus 100 proceeds to step S118. The image processing apparatus 100 performs character recognition on the area between the straight line pairs, and executes a process of selecting a straight line pair based on the certainty of character recognition (step S118). The image processing apparatus 100 determines whether there is one selected straight line pair (step S119). When the selected straight line pair is one (step S119, Yes), the image processing apparatus 100 proceeds to step S121.

  On the other hand, when there are two or more selected straight line pairs (step S119, No), the image processing apparatus 100 selects a straight line pair based on the specific character (step S120). The image processing apparatus 100 extracts the region coordinates of the straight line pair (step S121).

  Next, the straight line pair extraction process shown in step S113 of FIG. 22 will be described. FIG. 23 is a flowchart illustrating a processing procedure of straight line pair extraction processing. As shown in FIG. 23, the image processing apparatus 100 sets 1 to i (step S130). The image processing apparatus 100 sets a value obtained by adding 1 to the value of i to j (step S131).

  The image processing apparatus 100 selects the i-th straight line and the j-th straight line (step S132). The image processing apparatus 100 determines whether or not the i-th straight line and the j-th straight line have substantially the same length and overlap in the y direction (step S133). If the i-th straight line and the j-th straight line are not substantially the same length or do not overlap in the y direction (step S133, No), the image processing apparatus 100 proceeds to step S136.

  On the other hand, when the i-th straight line and the j-th straight line have substantially the same length and overlap in the y direction (step S133, Yes), the image processing apparatus 100 proceeds to step S134. The image processing apparatus 100 determines whether there is a black pixel between the straight line pair (step S134). If there is no black pixel between the straight line pairs (step S134, No), the image processing apparatus 100 proceeds to step S136.

  On the other hand, if there is a black pixel between the straight line pair (step S134, Yes), the image processing apparatus 100 records the straight line pair (step S135).

  The image processing apparatus 100 determines whether or not the value of j is less than the value of “N−1” (step S136). N is the total number of straight lines. If the value of j is less than the value of “N−1” (step S136, Yes), the image processing apparatus 100 sets a value obtained by adding 1 to the value of j to j (step S137). The process proceeds to step S132.

  On the other hand, when the value of j is not less than the value of “N−1” (step S136, No), the image processing apparatus 100 determines whether the value of i is less than the value of “N−1”. (Step S138). When the value of i is less than the value of “N−1” (step S138, Yes), the image processing apparatus 100 sets a value obtained by adding 1 to the value of i to i (step S139). The process proceeds to step S131.

  On the other hand, when the value of i is not less than the value of “N−1” (No in step S138), the image processing apparatus 100 ends the straight line pair extraction process.

  Next, the process of selecting a straight line pair based on the positional relationship between the black pixel connection region and the straight line pair shown in step S116 of FIG. 22 will be described. FIG. 24 is a flowchart illustrating a processing procedure for selecting a straight line pair based on the positional relationship between the black pixel connection region and the straight line pair.

  As shown in FIG. 24, the image processing apparatus 100 sets 1 to i (step S150). The image processing apparatus 100 sets a value obtained by adding 1 to the value of i to j (step S151). The image processing apparatus 100 selects the i-th straight line and the j-th straight line (step S152).

  The image processing apparatus 100 selects a rectangle having a constant distance between the i-th straight line and the j-th straight line (step S153). The image processing apparatus 100 determines whether or not the condition of Expression (1) is satisfied (step S154). When the condition of formula (1) is not satisfied (step S155, No), the process proceeds to step S157.

  On the other hand, when the condition of the expression (1) is satisfied (step S155, Yes), the image processing apparatus 100 records a straight line pair that satisfies the condition (step S156). The image processing apparatus 100 determines whether or not the value of j is less than the value of “N−1” (step S157). N is the total number of straight lines. If the value of j is less than the value of “N−1” (step S157, Yes), the image processing apparatus 100 sets a value obtained by adding 1 to the value of j to j (step S158), The process proceeds to step S152.

  On the other hand, when the value of j is not less than the value of “N−1” (No at Step S157), the image processing apparatus 100 proceeds to Step S159. The image processing apparatus 100 determines whether or not the value of i is less than the value of “N−1” (step S159). If the value of i is less than the value of “N−1” (Yes in step S159), the image processing apparatus 100 sets a value obtained by adding 1 to the value of i to i (step S160). The process proceeds to step S151.

  On the other hand, when the value of i is not less than “N−1” (No in step S159), the image processing apparatus 100 selects a straight line pair based on the positional relationship between the black pixel connection region and the straight line pair. The process ends.

  Next, a process of performing character recognition on the area between straight line pairs shown in step S118 of FIG. 22 and selecting straight line pairs based on the certainty of character recognition will be described. FIG. 25 is a flowchart showing a processing procedure for executing character recognition on an area between straight line pairs and selecting a straight line pair based on the certainty of character recognition.

  As shown in FIG. 25, the image processing apparatus 100 sets 1 to i (step S180). The image processing apparatus 100 performs character recognition on the inside of the i-th straight line pair, and calculates an average value of the certainty factor of each character (step S181).

  The image processing apparatus 100 determines whether the recognition result character is not a noise character and the average value is greater than or equal to a threshold value (step S182). If the recognition result character is a noise character, or if the average value is less than the threshold value (No in step S182), the image processing apparatus 100 proceeds to step S184.

  On the other hand, if the recognition result character is not a noise character and the average value is equal to or greater than the threshold (Yes in step S182), the image processing apparatus 100 records the i-th straight line pair (step S183). The image processing apparatus 100 determines whether or not the value of i is less than the value of “N−1” (step S184). When the value of i is less than the value “N−1” (step S184, Yes), the image processing apparatus 100 sets a value obtained by adding 1 to the value of i to i (step S185), The process proceeds to S181.

  On the other hand, when the value of i is not less than the value of “N−1” (No in step S184), the image processing apparatus 100 ends the process.

  Next, processing for selecting a straight line pair based on the specific character shown in step S120 of FIG. 22 will be described. FIG. 26 is a flowchart showing a processing procedure for selecting a straight line pair based on a specific character.

  As shown in FIG. 26, the image processing apparatus 100 sets 1 to i (step S190). The image processing apparatus 100 recognizes characters in the area surrounded by the extension lines at the upper and lower ends of the i-th straight pair area and the area other than the i-th straight pair (step S191). The image processing apparatus 100 determines whether or not a specific character such as “search” exists in the character recognition result (step S192).

  If there is no specific character such as “search” in the character recognition result (step S192, No), the image processing apparatus 100 proceeds to step S195. On the other hand, when a specific character such as “search” is present in the character recognition result (step S192, Yes), the image processing apparatus 100 proceeds to step S193.

  The image processing apparatus 100 determines whether the distance from the candidate area to a specific character such as “search” is the shortest (step S193). If the distance from the candidate area to a specific character such as “search” is not the shortest (step S193, No), the image processing apparatus 100 proceeds to step S195.

  When the distance from the candidate area to a specific character such as “search” is the shortest (step S193, Yes), the image processing apparatus 100 stores the closest candidate area (step S194).

  The image processing apparatus 100 determines whether or not the value of i is less than the value of “N−1” (step S195). When the value of i is less than “N−1” (Yes in step S195), the image processing apparatus 100 sets a value obtained by adding 1 to the value of i to i (step S196). The process proceeds to step S191. On the other hand, if the value of i is not less than “N−1” (No in step S195), the image processing apparatus 100 ends the process.

  Next, effects of the image processing apparatus 100 according to the present embodiment will be described. The image processing apparatus 100 extracts a plurality of horizontal straight lines based on the edge image data 122 and determines a combination of straight lines having a pixel connection region within a predetermined range between the plurality of straight line sets. Thus, it can be said that the combination of straight lines determined by the image processing apparatus 100 is the upper straight line and the lower straight line of the rectangle constituting the search window, and the search window can be detected with high accuracy. Further, since the image processing apparatus 100 does not extract straight lines other than in the horizontal direction, it is possible to prevent the extraction of straight lines that become noise and to reduce the processing load.

  In addition, the image processing apparatus 100 associates a plurality of black pixel connection regions having the same distance from the straight line for each straight line, compares the black pixel connection regions associated with each straight line, and compares each black pixel. A combination of straight lines that match a part of the connected area is determined. Thus, the search window including the keyword can be detected with high accuracy by making the image processing apparatus 100 make a determination using the characteristic that the search window has a black pixel corresponding to the keyword.

  Further, the image processing apparatus 100 further performs character recognition using a region between the pair of straight lines as a candidate region, and determines whether the candidate region is a search window based on the certainty of the character recognition result. To do. For this reason, it is possible to accurately extract a region corresponding to the search window from a plurality of candidate regions.

  When the candidate area includes a specific character, the image processing apparatus 100 selects another candidate area that exists in the horizontal direction with respect to the candidate area including the specific character, and the selected candidate area is displayed in the search window. It is determined whether or not there is. Usually, a specific character such as “search” may be displayed beside the search window. For this reason, by selecting another candidate area that exists in the horizontal direction with respect to the candidate area including the specific character, an area corresponding to the search window can be accurately extracted from the plurality of candidate areas. .

  Further, the image processing apparatus 100 searches for information related to characters included in the search window from the network and displays the search result, so that the user can perform a TV program CM without performing complicated operations. Detailed information about the keywords displayed in the search window can be notified to the user.

  By the way, the embodiment of the image processing apparatus 100 described above is not limited to the above. For this reason, other embodiments of the image processing apparatus 100 will be described below. For example, the image processing apparatus 100 may acquire image data including a search window captured by a digital camera, and detect a search window area from the acquired image data. The process for detecting the search window from the image data is almost the same as that of the image processing apparatus described above, but the process for associating the line segment with the black pixel connection area is partially different, and will be described below.

  FIG. 27 is a diagram for explaining another embodiment of the image processing apparatus. The image processing apparatus 100 extracts an edge image from image data of a television image captured by a digital camera, and extracts a horizontal straight line. For example, the image processing apparatus 100 extracts a straight line whose angle is within a predetermined angle with respect to the horizontal direction. This is a process that takes into account that when a digital camera captures image data, the horizontal straight line of the search window appears obliquely. In the example shown in FIG. 27, the image processing apparatus 100 extracts straight lines 2c and 2d. In addition, when the image processing apparatus 100 selects a straight line pair, the difference in length between the straight lines is less than a predetermined threshold and overlaps in the y direction, and a black pixel connection region exists between the straight lines. Each straight line is a pair.

  When the straight line and the black pixel connection area are associated with each other, the image processing apparatus 100 associates the black corner connection area where the distance between the rectangular corner of the black pixel connection area and the straight line is the same. For example, the distance between the upper left corner of the black pixel connection areas 4h, 4i, 4j, 4k, and 4m and the straight line 2c is the same. Therefore, the straight line 2c is associated with the black pixel connection areas 4h, 4i, 4j, 4k, and 4m. The distances between the lower left corners of the black pixel connection areas 4h, 4i, 4j, 4l, and 4m and the straight line 2d are the same. Therefore, the straight line 2d is associated with the black pixel connection areas 4h, 4i, 4j, 4l, and 4m.

  When the relationship between the straight line 2c and the straight line 2d satisfies the condition of the expression (1) shown in the above embodiment, the image processing apparatus 100 uses the straight line pair of the straight line 2c and the straight line 2d as a search window candidate area. judge. Other processes are the same as in the above embodiment. By executing such processing by the image processing apparatus, it is possible to detect the search window with high accuracy even from an image captured by a digital camera.

  In the above embodiment, the case where the image processing apparatus 100 is built in the TV 50 has been described. However, the present invention is not limited to this. For example, a function similar to that of the image processing apparatus 100 may be provided in the server apparatus of the network 10. For example, when the search request is acquired, the TV control unit 55 transmits the video data to the server, and the server device executes the same processing as the image processing device 100 to extract the search window, and The keyword search result may be notified to the TV control unit 55. The TV control unit 55 causes the monitor 52 to display the notified search result. Further, for example, in a device including a portable camera such as a digital camera, a smartphone, or a mobile phone, a function similar to that of the image processing device 100 may be provided. In this case, the hardware configuration is an image acquisition unit, an image processing device, a communication unit, and a monitor. The image acquired by the image acquisition unit is input to the image processing device, and the text in the search window is acquired by the image processing device. Then, it is sent to the communication unit, Internet search is performed, the result is acquired by the communication unit, and displayed on the monitor.

  Next, an example of a computer that executes an image processing program that realizes the same function as the image processing apparatus 100 described in the above embodiment will be described. FIG. 28 is a diagram illustrating an example of a computer that executes a control program.

  As illustrated in FIG. 28, the computer 200 includes a CPU 201 that executes various arithmetic processes, an input device 202 that receives input of data from a user, and a display 203. The computer 200 also includes a reading device 204 that reads a program and the like from a storage medium, and an interface device 205 that exchanges data with other computers via a network. The computer 200 also includes a RAM 206 that temporarily stores various information and a hard disk device 207. The devices 201 to 207 are connected to the bus 208.

  The hard disk device 207 includes, for example, a line extraction program 207a and a determination program 207b. The CPU 201 reads out each program 207a, 207b and develops it in the RAM 206.

  The straight line extraction program 207a functions as a straight line extraction process 206a. The determination program 207b functions as a determination process 206b.

  For example, the line extraction process 206a corresponds to the line extraction unit 133a. The determination process 206b corresponds to the determination unit 133c.

  Note that the programs 207a and 207b are not necessarily stored in the hard disk device 207 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, and an IC card inserted into the computer 200. Then, the computer 200 may read and execute the programs 207a and 207b from these.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Appendix 1) A straight line extraction unit that extracts a plurality of horizontal straight lines based on an edge image obtained by extracting edges from an image;
A determination unit that extracts a black pixel connection region obtained by connecting edges in the edge image, and determines a combination of straight lines having a black pixel connection region within a predetermined range between the plurality of straight line combinations. An image processing apparatus.

(Additional remark 2) The said determination part matches the some black pixel connection area | region with the same distance with this straight line for every straight line, compares each black pixel connection area matched for every straight line, and compares each black The image processing apparatus according to appendix 1, wherein a combination of straight lines in which a part of the pixel connection area matches is determined.

(Additional remark 3) The said determination part matches the some black pixel connection area | region where the distance of this straight line and the predetermined | prescribed corner of a black pixel connection area | region becomes the same for every straight line, The image processing of Additional remark 2 characterized by the above-mentioned apparatus.

(Additional remark 4) The said determination part further performs character recognition by making the area | region between the group of straight lines into a search window candidate area | region, and based on the certainty of a character recognition result, the said search window candidate area | region is a search window. 4. The image processing apparatus according to appendix 1, 2, or 3, wherein it is determined whether or not there is any.

(Supplementary Note 5) When the search window candidate area includes a specific character, the determination unit selects another search window candidate area that exists in a horizontal direction with respect to the search window candidate area including the specific character, The image processing apparatus according to appendix 4, wherein it is determined whether or not the selected search window candidate area is a search window.

(Supplementary note 6) The image processing apparatus according to supplementary note 4 or 5, further comprising a search unit that searches the network for information related to characters included in the search window determined by the determination unit.

(Appendix 7)
Based on the edge image extracted from the image, to extract a plurality of horizontal straight lines,
Extracting a black pixel connection region obtained by connecting edges in the edge image, and executing each process for determining a combination of straight lines having a black pixel connection region within a predetermined range between the plurality of straight line combinations. An image processing program characterized by the above.

(Supplementary note 8) The determination process is performed by associating a plurality of black pixel connection regions having the same distance from the straight line for each straight line, comparing the black pixel connection regions associated with each straight line, and comparing The image processing program according to appendix 7, wherein a combination of straight lines in which a part of the black pixel connection region matches is determined.

(Supplementary note 9) The image according to supplementary note 7, wherein the determining process associates, for each straight line, a plurality of black pixel connection regions having the same distance between the straight line and a predetermined corner of the black pixel connection region. Processing program.

(Additional remark 10) The said determination process further performs character recognition by making the area | region between the group of straight lines into a search window candidate area | region, and based on the certainty of a character recognition result, the said search window candidate area | region is a search window. 10. The image processing program according to appendix 7, 8 or 9, wherein it is determined whether or not.

(Supplementary Note 11) When the search window candidate area includes a specific character, the determination process selects another search window candidate area that exists in a horizontal direction with respect to the search window candidate area including the specific character. The image processing program according to appendix 10, wherein it is determined whether or not the selected search window candidate area is a search window.

(Additional remark 12) The image processing program as described in additional remark 10 or 11 which further performs the process which searches the information relevant to the character contained in the determined search window from a network.

(Supplementary note 13) An image processing method executed by a computer,
Based on the edge image extracted from the image, to extract a plurality of horizontal straight lines,
Extracting a black pixel connection region obtained by connecting edges in the edge image, and determining each combination of straight lines having a black pixel connection region within a predetermined range between the plurality of straight line combinations. An image processing method characterized by the above.

(Supplementary Note 14) The determination process is performed by associating a plurality of black pixel connection regions having the same distance from the straight line for each straight line, comparing the black pixel connection regions associated with each straight line, and comparing The image processing method according to appendix 13, wherein a combination of straight lines in which a part of the black pixel connection region matches is determined.

(Supplementary note 15) The image according to supplementary note 13, wherein the determining process associates, for each straight line, a plurality of black pixel connection regions having the same distance between the straight line and a predetermined corner of the black pixel connection region. Processing method.

(Additional remark 16) The said determination process further performs character recognition by making the area | region between the group of straight lines into a search window candidate area | region, and based on the certainty of a character recognition result, the said search window candidate area | region is a search window. 16. The image processing method according to appendix 13, 14 or 15, wherein it is determined whether or not.

(Supplementary Note 17) When the search window candidate area includes a specific character, the determination process selects another search window candidate area that exists in the horizontal direction with respect to the search window candidate area including the specific character. The image processing method according to appendix 16, wherein it is determined whether or not the selected search window candidate area is a search window.

(Supplementary note 18) The image processing method according to supplementary note 16 or 17, further comprising a process of searching the network for information related to the character included in the determined search window.

10 network 50 TV
100 Image processing apparatus

Claims (8)

A line extraction unit that extracts a plurality of horizontal lines based on an edge image obtained by extracting edges from the image;
A determination unit that extracts a black pixel connection region obtained by connecting edges in the edge image, and determines a combination of straight lines having a black pixel connection region within a predetermined range between the plurality of straight line combinations. An image processing apparatus.   The determination unit associates, for each straight line, a plurality of black pixel connection regions having the same distance from the straight line, compares the black pixel connection regions associated with each straight line, and compares each of the compared black pixel connection regions. The image processing apparatus according to claim 1, wherein a combination of straight lines that partially match each other is determined.   The image processing apparatus according to claim 2, wherein the determination unit associates, for each straight line, a plurality of black pixel connection regions having the same distance between the straight line and a predetermined corner of the black pixel connection region.   The determination unit further performs character recognition using a region between the pair of straight lines as a search window candidate region, and based on the certainty of the character recognition result, whether or not the search window candidate region is a search window. The image processing apparatus according to claim 1, wherein:   When the search window candidate area includes a specific character, the determination unit selects another search window candidate area that exists in a horizontal direction with respect to the search window candidate area including the specific character, and selects the selected search window The image processing apparatus according to claim 4, wherein it is determined whether the candidate area is a search window.   The image processing apparatus according to claim 4, further comprising a search unit that searches the network for information related to characters included in the search window determined by the determination unit. On the computer,
Based on the edge image extracted from the image, to extract a plurality of horizontal straight lines,
Extracting a black pixel connection region obtained by connecting edges in the edge image, and executing each process for determining a combination of straight lines having a black pixel connection region within a predetermined range between the plurality of straight line combinations. An image processing program characterized by the above. An image processing method executed by a computer,
Based on the edge image extracted from the image, to extract a plurality of horizontal straight lines,
Extracting a black pixel connection region obtained by connecting edges in the edge image, and determining each combination of straight lines having a black pixel connection region within a predetermined range between the plurality of straight line combinations. An image processing method characterized by the above.

Images