JP7018372B2 - Image processing device and image processing method - Google Patents

Description

The disclosed technique relates to an image processing apparatus and an image processing method.

The image taken by the camera (hereinafter sometimes referred to as "captured image") is an image of the medium (hereinafter sometimes referred to as "target medium") taken by the camera (hereinafter referred to as "target medium"). (Sometimes referred to as "medium image") is included. When characters are printed on the surface of the target medium, character recognition may be performed on the medium image by using, for example, OCR (Optical Character Recognition). For example, when the target medium is a driver's license, personal information such as name, date of birth, address, license number, etc. printed on the surface of the driver's license is OCR from the image of the driver's license taken by the camera. May be obtained using.

International Publication No. 2017/208367 Japanese Unexamined Patent Publication No. 2011-035942 Special Table 2018-500703 Gazette

Since there are various situations in which characters are printed on the surface of the target medium by the camera, it may be difficult to set the target area for character recognition on the captured image. For example, in a situation where the target medium is held in the hand for shooting, it may be difficult to identify the medium image in the shot image because a part of the medium image may be hidden by the image of the finger of the hand. .. When it becomes difficult to specify the medium image in the captured image, it becomes difficult to set the target area for character recognition on the captured image. If it becomes difficult to set the target area for character recognition on the captured image, the recognition rate of character recognition will decrease.

The disclosed technique has been made in view of the above, and an object thereof is to accurately set a target area in a captured image.

In the disclosed aspect, the image processing apparatus has a storage unit and an image processing unit. The storage unit stores a photographed image. The image processing unit detects a plurality of line segments in the captured image, detects a plurality of candidates for a combination of four side lines of a quadrilateral based on the mutual relationship of the plurality of line segments, and detects the plurality of candidates. A plurality of scores for each of the candidates are calculated, one combination is selected from the plurality of candidates based on the plurality of scores, and each of the line segments on the four sides included in the selected combination. The target area is determined based on the quadrilateral area having each side, and the target area is set on the captured image.

According to the disclosed aspect, it is possible to accurately set the target area in the captured image.

FIG. 1 is a diagram showing a configuration example of the image processing apparatus of the first embodiment. FIG. 2 is a diagram showing an example of a processing flow of the image processing apparatus of the first embodiment. FIG. 3 is a diagram showing an example of a processing flow of the image processing apparatus of the first embodiment. FIG. 4 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 5 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 6 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 7 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 8 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 9 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 10 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 11 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 12 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 13 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 14 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 15 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 16 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 17 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 18 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 19 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 20 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment. FIG. 21 is a diagram for explaining an operation example of the image processing apparatus of the first embodiment.

Hereinafter, examples of the image processing apparatus and the image processing method disclosed in the present application will be described with reference to the drawings. It should be noted that this embodiment does not limit the image processing apparatus and image processing method disclosed in the present application. Further, in the embodiment, the same reference numerals are given to the configurations having the same functions and the steps for performing the same processing.

[Example 1]
<Configuration of image processing device>
FIG. 1 is a diagram showing a configuration example of the image processing apparatus of the first embodiment. In FIG. 1, the image processing device 10 has a control unit 11, a storage unit 13, and an image processing unit 15. The image processing unit 15 operates under the control of the control unit 11. The image processing device 10 is mounted and used, for example, in a mobile terminal provided with a camera or a stationary terminal provided with a camera.

The captured image is stored in the storage unit 13.

The image processing unit 15 acquires a photographed image stored in the storage unit 13 from the storage unit 13 according to an instruction from the control unit 11, and performs the image processing described below on the acquired photographed image.

<Processing and operation of image processing equipment>
2 and 3 are diagrams showing an example of the processing flow of the image processing apparatus of the first embodiment. 4 to 21 are diagrams for explaining an operation example of the image processing apparatus of the first embodiment.

In the first embodiment, a case where the target medium is a driver's license will be described as an example.

The processing flow shown in FIG. 2 is started when the image processing unit 15 acquires a captured image from the storage unit 13. Further, the acquisition of the captured image from the storage unit 13 by the image processing unit 15 is performed according to the instruction from the control unit 11. For example, when the processing start button displayed on the touch panel of the mobile terminal including the camera is touched, the control unit 11 gives an instruction to acquire the captured image from the storage unit 13 to the image processing unit 15. FIG. 4 shows an example of a captured image PI acquired by the image processing unit 15 from the storage unit 13. In FIG. 4, the captured image PI is, for example, a driver's license image (hereinafter sometimes referred to as “license image”) DLI, which is a medium image, and an image of a finger holding the driver's license (hereinafter, referred to as “license image”). It includes an FI (sometimes referred to as a "finger image") and a background image (hereinafter sometimes referred to as a "background image") BGI. In FIG. 4, in the captured image PI, a part of the license image DLI is hidden by the finger image FI.

In FIG. 2, in step S101, the image processing unit 15 extracts edge pixels in the horizontal and vertical directions from the captured image acquired from the storage unit 13, for example, by using the proximity gradation difference, and the edge pixels in the horizontal direction. An image formed from (hereinafter sometimes referred to as a "horizontal edge image") EIH (FIG. 5) and an image formed from vertical edge pixels (hereinafter referred to as a "vertical edge image"). There is) EIV (Fig. 6) is generated.

Next, in step S103, the image processing unit 15 performs huff detection and line segment detection using the minimum square method for the horizontal edge image EIH (FIG. 5) and the vertical edge image EIV (FIG. 6). An image formed from a plurality of detected line segments (hereinafter, may be referred to as a "line segment image") LSD (FIG. 7) is generated. As shown in FIG. 7, the line segment image LSD is referred to as an image formed from a plurality of horizontal line segments (hereinafter sometimes referred to as "horizontal line segment") (hereinafter referred to as "horizontal line segment image"). (Sometimes called) and an image formed from multiple vertical line segments (sometimes called "vertical line segments" below) (sometimes called "vertical line segment images" below). Is included. For example, the image processing unit 15 connects edge pixels that are horizontally adjacent to each other in the horizontal edge image EIH to generate a horizontal line segment image, and is vertically adjacent to each other in the vertical edge image EIV. Edge pixels are connected to generate a vertical line segment image.

Next, in step S105, the image processing unit 15 performs combination candidate detection processing.

FIG. 3 shows an example of the processing flow of the combination candidate detection process.

Here, FIG. 8 shows an example of a ruled line selected based on the layout of a plurality of ruled lines drawn on the surface of the driver's license DL, which is the target medium (hereinafter, may be referred to as “selected ruled line”). show. A plurality of ruled lines are laid out and drawn on the surface of the driver's license DL in both the horizontal direction and the vertical direction, and one of these multiple ruled lines is based on the layout of these multiple ruled lines. Ruled line is selected in advance. For example, as shown in FIG. 8, in the horizontal direction, seven ruled lines LH1 to LH7 are selected in advance, and in the vertical direction, two ruled lines LV1 and LV2 are selected in advance.

Further, the lower side of the driver's license DL is also regarded as a ruled line, and the lower side of the driver's license DL is selected in advance as the ruled line LH8. Further, in the ruled lines LH1 to LH8, the ruled lines LH1 and LH2 are selected as "reference lines", and the ruled lines LH3 to LH8 are selected as "auxiliary lines" having a lower priority than the reference lines. That is, the selection ruled line includes a reference line and an auxiliary line having a lower priority than the reference line.

In addition, in a situation where a driver's license DL is held in the hand for shooting, a part of the selection ruled lines LH5, LH6, and LH8, which are auxiliary lines, may be hidden. The selection ruled lines LH5 and LH6, which are lines, were set, and the selection ruled lines LH8, which was an auxiliary line, were set below the selection ruled line LH2, which was the reference line.

If the background color of the driver's license DL and the background color of the driver's license DL are similar, the upper side of the driver's license DL is selected as the selection ruled line LH1 and the lower side of the driver's license DL is selected as the ruled line LH2. If set to, the reference line may be unknown. Therefore, the selection ruled line LH1 is set below the upper side of the driver's license DL, and the selection ruled line LH2 is set above the lower side of the driver's license DL. That is, the selection ruled lines LH5, LH6, and LH8 are set in a region other than the region sandwiched between the selection ruled line LH1 and the selection ruled line LH2.

The reference line is sometimes called a "first ruled line", and the auxiliary line is sometimes called a "second ruled line".

Further, FIG. 9 shows an example of the positional relationship between the ruled lines LH1 to LH8. As shown in FIG. 9, the ruled line LH1 and the ruled line LH2 are located apart by a distance L1, the ruled line LH1 and the ruled line LH3 are located separated by a distance L2, and the ruled line LH1 and the ruled line LH4 are located separated by a distance L3. do. Further, the ruled line LH3 and the ruled line LH4 are located apart by a distance L4, the ruled line LH1 and the ruled line LH5 are located separated by a distance L5, and the ruled line LH3 and the ruled line LH5 are located separated by a distance L6. It is located at a distance L7 from the ruled line LH6. Further, the ruled line LH5 and the ruled line LH6 are located apart by a distance L8, the ruled line LH3 and the ruled line LH7 are located separated by a distance L9, and the ruled line LH4 and the ruled line LH7 are located separated by a distance L10. It is located at a distance L11 from the ruled line LH8.

Further, FIG. 10 shows an example of the positional relationship table TA in which the correspondence relationship between the positional relationship of the ruled lines and the distance between the ruled lines is set. The distances L1 to L11 between the ruled lines in FIG. 10 correspond to the distances L1 to L11 in FIG. Further, in the distances L1 to L11 between the ruled lines of the positional relationship table TA, a margin of "± Δl" is added to the distances L2 to L11 between the ruled lines. Δl is, for example, "1 mm". The positional relationship table TA is stored in advance in the storage unit 13 and is referred to by the image processing unit 15.

Returning to FIG. 3, in step S131, the image processing unit 15 selects a horizontal line candidate. The image processing unit 15 divides the selection of the horizontal line candidate into the selection of the reference line candidate (step S133) and the selection of the auxiliary line candidate (step S135), and selects after selecting the two reference line candidates. Select an auxiliary line candidate based on the reference line candidate.

Next, in step S137, the image processing unit 15 selects two vertical line candidates.

Then, in step S139, the image processing unit 15 combines the two reference line segment candidates selected in step S133 and the two vertical line segment candidates selected in step S137 to have four sides as shown in FIG. Candidates for combinations of four-sided line segments (upper side line segment: LH1C, lower side line segment: LH2C, left side line segment: LV1C, right side line segment: LV2C) (hereinafter referred to as "combination candidates") Yes) Detect multiple QRCs. Each line segment of the combination candidate forms a part or all of any of the four sides of the quadrilateral.

For example, in step S133, the image processing unit 15 has lined lines LH1 and LH2 (that is, "reference lines") from among the plurality of horizontal line segments included in the horizontal line segment image in the line segment image LSD (FIG. 7). The horizontal line segment corresponding to is selected as the reference line candidates LH1C and LH2C. For example, the image processing unit 15 selects a combination of two horizontal line segments satisfying all of the following conditions 1 to 3 as a combination of reference line candidates LH1C and LH2C among all the line segments in the horizontal direction.
<Condition 1> The reference line candidate LH1C exists in the upper half region UPH (FIG. 4) of the captured image PI.
<Condition 2> The reference line candidate LH2C exists in the region LOH (FIG. 4) in the lower half of the captured image PI.
<Condition 3> The inclination of the reference line candidate LH2C with respect to the reference line candidate LH1C is ± 3 ° or less.

After the processing of step S133, in step S135, the image processing unit 15 selects the ruled lines LH3 to LH8 (that is, from among the plurality of horizontal line segments included in the horizontal line segment image in the line segment image LSD (FIG. 7)). The horizontal line segments corresponding to each of the "auxiliary lines") are selected as the auxiliary line candidates LH3C to LH8C.

In step S135, the image processing unit 15 first satisfies all of the following conditions 4 and 5 among all the line segments in the horizontal direction based on the reference line candidates LH1C and LH2C selected according to the conditions 1 to 3. The horizontal line segment is selected as the auxiliary line candidates LH3C to LH8C. The image processing unit 15 selects auxiliary line candidates in the order of LH3C → LH4C → LH5C → LH6C → LH7C → LH8C.
<Condition 4> The slope with respect to all neighboring candidates is ± 1 ° or less.
<Condition 5> The deviation of the relative distance with respect to all neighboring candidates is ± Δl mm or less (for example, Δl = 1 mm).

Here, the neighborhood candidate for the auxiliary line candidate LH3C is the reference line candidate LH1C, the neighborhood candidate for the auxiliary line candidate LH4C is the reference line candidate LH1C and the auxiliary line candidate LH3C, and the neighborhood candidate for the auxiliary line candidate LH5C is the reference line candidate LH1C. And the auxiliary line candidate LH3C, the neighborhood candidates for the auxiliary line candidate LH6C are the reference line candidate LH1C and the auxiliary line candidate LH5C, and the neighborhood candidates for the auxiliary line candidate LH7C are the auxiliary line candidate LH3C and the auxiliary line candidate LH4C, and the auxiliary line candidate LH4C. The neighbor candidate for the candidate LH8C is the reference line candidate LH2C.

Further, the image processing unit 15 calculates the "relative distance" in the condition 5 based on the reference line candidates LH1C and LH2C selected in step S133 and the positional relationship table TA (FIG. 10). For example, since the neighborhood candidate for the auxiliary line candidate LH3C is the reference line candidate LH1C, if the distance between the reference line candidate LH1C and the reference line candidate LH2C is "Nmm", which is equivalent to the actual size in the driver's license DL, the position. Based on the distances L1 and L2 between the ruled lines of the relation table TA, the relative distance for the auxiliary line candidate LH3C is calculated as "N × (4/40) = N / 10 mm". Further, for example, since the neighborhood candidates for the auxiliary line candidate LH4C are the reference line candidate LH1C and the auxiliary line candidate LH3C, the distance between the reference line candidate LH1C and the reference line candidate LH2C is equivalent to the actual size in the driver's license DL and is "Nmm". If, the relative distances for the auxiliary line candidate LH4C are "N × (8/40) = N / 5 mm" and "N × (N ×) based on the distances between the ruled lines L1, L3, L4 of the positional relationship table TA. 4/40) = N / 10 mm ”is calculated.

Further, in step S135, the image processing unit 15 then determines whether or not the horizontal line segment to be selected as the auxiliary line candidate LH8C is erroneously selected as the reference line candidate LH2C. The image processing unit 15 determines that the horizontal line segment to be selected as the auxiliary line candidate LH8C is erroneously selected as the reference line candidate LH2C when all of the following conditions 6 to 9 are satisfied. Then, the image processing unit 15 excludes the combination including the reference line candidate LH2C satisfying all of the following conditions 6 to 9 from the combination candidates.
<Condition 6> A horizontal line segment exists within the upward relative distance from the reference line candidate LH2C selected according to conditions 1 to 3.
<Condition 7> The inclination of the auxiliary line candidate LH8C with respect to the reference line candidate LH2C is ± 1 ° or less.
<Condition 8> The deviation of the relative distance of the auxiliary line candidate LH8C with respect to the reference line candidate LH2C is ± Δl mm or less (for example, Δl = 1 mm).
<Condition 9> The total number of edge pixels on the auxiliary line candidate LH8C is 70% or more of the total number of edge pixels on the reference line candidate LH2C.

Here, the image processing unit 15 calculates the "relative distance" in the conditions 6 and 8 based on the reference line candidates LH1C and LH2C selected in step S133 and the positional relationship table TA (FIG. 10). For example, when the distance between the reference line candidate LH1C and the reference line candidate LH2C is "N mm", which is equivalent to the actual size in the driver's license DL, the condition 6 is based on the distances between the ruled lines L1 and L11 of the positional relationship table TA. , 8 is calculated as "N × (3/40) = 3N / 40 mm".

Of the above conditions 1 to 9, conditions 3 to 9 define the interrelationship between a plurality of horizontal line segments included in the horizontal line segment image in the line segment image LSD (FIG. 7).

Next, in step S137, the image processing unit 15 selects the vertical line segments corresponding to the ruled lines LV1 and LV2 from among the plurality of vertical line segments included in the vertical line segment image in the line segment image LSD (FIG. 7). Is selected as the vertical line segment candidates LV1C and LV2C. For example, the image processing unit 15 selects a combination of two vertical line segments satisfying all of the following conditions 10 to 13 as a combination of the vertical line candidates LV1C and LV2C among all the line segments in the vertical direction.
<Condition 10> The vertical line candidate LV1C exists in the left half region LEH (FIG. 4) of the captured image PI.
<Condition 11> The vertical line candidate LV2C exists in the region RIH (FIG. 4) in the right half of the captured image PI.
<Condition 12> The inclination of the vertical line candidate LV1C with respect to the reference line candidate LH1C is 90 ° ± 25 ° or less.
<Condition 13> The inclination of the vertical line candidate LV2C with respect to the reference line candidate LH1C is 90 ° ± 25 ° or less.

Then, in step S139, the image processing unit 15 is based on the reference line candidates LH1C and LH2C selected in step S133 and the auxiliary line candidates selected in step S135 (that is, the reference line candidates LH1C and LH2C selected in step S133). By combining the selected auxiliary line candidate) and the vertical line candidates LV1C and LV2C selected in step S137, a plurality of combination candidate QRCs are detected.

Here, FIGS. 12 to 14 show an example of the reference line candidates LH1C and LH2C and the auxiliary line candidates LH3C to LH8C selected according to the conditions 1 to 9.

The horizontal line candidate group LHC1 shown in FIG. 12 includes two reference line candidates LH1C and LH2C and six auxiliary line candidates LH3C to LH8C.

Further, the horizontal line candidate group LHC2 shown in FIG. 13 includes two reference line candidates LH1C and LH2C and five auxiliary line candidates LH3C to LH7C, but does not include the auxiliary line candidate LH8C. Since the reference line candidate LH2C in the horizon candidate group LHC2 satisfies all of the conditions 6 to 9, the image processing unit 15 excludes the horizon candidate group LHC2 from the combination candidates.

Further, the horizontal line candidate group LHC3 shown in FIG. 14 includes two reference line candidates LH1C and LH2C and three auxiliary line candidates LH3C to LH5C, but does not include auxiliary line candidates LH6C to LH8C. .. The image processing unit 15 excludes combinations of the reference line candidates LH1C, LH2C and the auxiliary line candidates LH3C to LH8C that do not include any of the reference line candidates LH1C, LH2C or the auxiliary line candidates LH3C to LH7C from the combination candidates. Therefore, the image processing unit 15 excludes the horizontal line candidate group LHC3 that does not include the auxiliary line candidates LH6C and LH7C from the combination candidates.

Further, FIGS. 15 to 17 show an example of a combination candidate of the reference line candidates LH1C and LH2C included in the horizontal line candidate group LHC1 (FIG. 12) and the vertical line candidates LV1C and LV2C satisfying all of the conditions 10 to 13. The combination candidate QRC1 shown in FIG. 15, the combination candidate QRC2 shown in FIG. 16, and the combination candidate QRC3 shown in FIG. 17 have the same horizontal line segment selected as the reference line candidates LH1C and LH2C, but the vertical line candidate LV1C. , The vertical line segments selected as LV2C are different from each other.

As described above, the image processing unit 15 detects, for example, three combination candidates QRC1, QRC2, and QRC3 in step S139.

Returning to FIG. 2, in step S107, the image processing unit 15 calculates a plurality of scores for the plurality of combination candidates detected in step S139.

Next, in step S109, the image processing unit 15 selects any one combination as the optimum combination from the plurality of combination candidates detected in step S139 based on the score calculated in step S107. For example, the image processing unit 15 selects the combination candidate having the highest score as the optimum combination from the plurality of combination candidates detected in step S139.

Here, the calculation of the score in step S107 will be described with reference to calculation examples 1 to 5.

<Calculation example 1>
The image processing unit 15 calculates the total number of edge pixels on each of the line segments LH1C, LH2C, LV1C, and LV2C on the four sides of the combination candidate QRC (FIG. 11) as a "reference score".

<Calculation example 2>
In the line segments LH1C, LH2C, LV1C, and LV2C on the four sides of the combination candidate QRC (FIG. 11), the image processing unit 15 between the end points of the line segments and the other line segments (or extension lines of the other line segments). The total value of the distances is calculated as a correction score, and the calculated correction score is subtracted from the reference score.

For example, as shown in FIG. 18, when the combination candidate QRC includes the line segment LH1C on the upper side and the line segment LV1C1 on the left side, the image processing unit 15 has the end point EP of the line segment LH1C on the upper side and the line segment LV1C1 on the left side. The distance d1 between the extension line and the extension line is calculated as the correction score. Further, for example, as shown in FIG. 18, when the combination candidate QRC includes the line segment LH1C on the upper side and the line segment LV1C2 on the left side, the distance d2 between the end point EP of the line segment LH1C on the upper side and the line segment LV1C2 on the left side. Is calculated as a correction score.

<Calculation example 3>
The image processing unit 15 multiplies the reference score by a coefficient k according to the presence or absence of the auxiliary line candidate included in the combination candidate QRC. For example, the image processing unit 15 sets the coefficient k to “1” when all of the five auxiliary line candidates of the auxiliary line candidates LH3C to LH7C are included in the combination candidate QRC, while the auxiliary line candidates LH3C to If any one of LH7C is not included in the combination candidate QRC, the coefficient k is set to "0".

For example, since the combination candidates QRC1, QRC2, and QRC3 (FIGS. 15, 16, and 17) are generated based on the horizon candidate group LHC1, each of the combination candidates QRC1, QRC2, and QRC3 has an auxiliary line candidate LH3C. -All of the five auxiliary line candidates of LH7C are included. Therefore, the image processing unit 15 sets the coefficient k to "1" for each of the combination candidates QRC1, QRC2, and QRC3.

<Calculation example 4>
The image processing unit 15 calculates the total number of edge pixels on the auxiliary line candidates included in the combination candidate QRC as a correction score, and adds the calculated correction score to the reference score.

<Calculation example 5>
Importance kl3 to kl7 are given in advance to each of the five selection ruled lines LH3 to LH7. In the combination candidate QRC, the image processing unit 15 multiplies the number of edge pixels on the auxiliary line candidates LH3C to LH7C corresponding to the selected ruled lines LH3 to LH7 by the importance kl3 to kl7, and the auxiliary line candidate. The total value of each multiplication result of LH3C to LH7C is calculated as a correction score, and the calculated correction score is added to the reference score.

The calculation examples 1 to 5 have been described above for the calculation of the score. The image processing unit 15 may use Calculation Example 1 alone to calculate the score, or may combine any one or more of Calculation Examples 2 to 5 with Calculation Example 1 to calculate the score. good. For example, the image processing unit 15 calculates the score according to the following equation (1). In the equation (1), the coefficient b is a weighting coefficient and is set to, for example, "2".
Score = standard score x coefficient k- (correction score x coefficient b) ... (1)

Here, for the combination candidate QRC1 (FIG. 15), the detection positions of the reference line candidates LH1C and LH2C are the same as the positions of the selection ruled lines LH1 and LH2 (FIG. 8), and the detection positions of the vertical line candidates LV1C and LV2C are selected. It is the same as the position of the ruled lines LV1 and LV2 (FIG. 8). Regarding the combination candidate QRC2 (FIG. 16), the detection positions of the reference line candidates LH1C and LH2C are the same as the positions of the selection ruled lines LH1 and LH2, and the detection positions of the vertical line candidate LV1C are the same as the positions of the selection ruled line LV1. On the other hand, the detection position of the vertical line candidate LV2C is different from the position of the selected ruled line LV2. Regarding the combination candidate QRC3 (FIG. 17), the detection positions of the reference line candidates LH1C and LH2C are the same as the positions of the selected ruled lines LH1 and LH2, and the detection positions of the vertical line candidate LV2C are the same as the positions of the selected ruled line LV2. On the other hand, the detection position of the vertical line candidate LV1C is different from the position of the selected ruled line LV1. Further, the degree of difference between the detection position of the vertical line candidate LV2C and the position of the selected ruled line LV2 in the combination candidate QRC2 is larger than the degree of difference between the detection position of the vertical line candidate LV1C and the position of the selected ruled line LV1 in the combination candidate QRC3. Therefore, when the scores of the combination candidates QRC1, QRC2, and QRC3 are calculated according to the equation (1), "Score of QRC1> Score of QRC3> Score of QRC2" is obtained. Therefore, when the image processing unit 15 detects the three combination candidates QRC1, QRC2, and QRC3 in step S139, in step S109, the combination candidate having the highest score among the combination candidates QRC1, QRC2, and QRC3. QRC1 is selected as the optimal combination OP.

Returning to FIG. 2, in step S111, the image processing unit 15 determines the quadrilateral region QR based on the optimum combination OP selected in step S109. For example, when the combination candidate QRC1 (FIG. 15) is selected as the optimum combination OP, the image processing unit 15 selects the reference line candidates LH1C, LH2C and the vertical line candidates LV1C, LV2C included in the combination candidate QRC1 as shown in FIG. A quadrilateral region QR having an upper side UPS, a lower side LOS, a left side LES, and a right side RIS formed from each of the above is determined. The upper side UPS is formed from the reference line candidate LH1C, the lower side LOS is formed from the reference line candidate LH2C, the left side LES is formed from the vertical line candidate LV1C, and the right side RIS is formed from the vertical line candidate LV2C. The image processing unit 15 forms the upper side UPS, the lower side LOS, the left side LES, and the right side RIS of the quadrilateral region QR by extending or cutting each of the reference line candidates LH1C and LH2C and the vertical line candidates LV1C and LV2C.

Next, in step S113, the image processing unit 15 determines the target region TR based on the quadrilateral region QR determined in step S111. For example, the image processing unit 15 first corrects the distortion of the shape of the quadrilateral region QR by using the well-known technique “perspective projection conversion” based on the ruled line layout shown in FIG. By this correction, the shape of the quadrilateral area QR is corrected to a rectangle. Next, as shown in FIG. 20, the image processing unit 15 uses the QRA, which is a rectangular area corrected to a rectangular shape (hereinafter, may be referred to as a “rectangular area”), with the four sides of the driver's license DL and the selection ruled line LH1, The target region TR is determined by expanding at a predetermined ratio based on the positional relationship with LH2, LV1, and LV2 (FIG. 8).

Next, in step S115, the image processing unit 15 sets the target area TR determined in step S113 on the captured image PI, as shown in FIG. 21.

Then, in step S117, the image processing unit 15 performs character recognition on the image in the target area TR set in the captured image PI by using, for example, OCR.

As described above, in the first embodiment, the image processing device 10 has a storage unit 13 and an image processing unit 15. The storage unit 13 stores the captured image PI. The image processing unit 15 detects a plurality of line segments in the captured image PI, and based on the mutual relationship of the detected plurality of line segments, a plurality of candidate QRC1, QRC2, and QRC3 for a combination of the four side line segments of the quadrilateral is selected. To detect. Further, the image processing unit 15 calculates a plurality of scores for each of the combination candidates QRC1, QRC2, and QRC3, and selects the optimum combination OP from the combination candidates QRC1, QRC2, and QRC3 based on the calculated plurality of scores. .. Further, the image processing unit 15 determines the target area TR based on the quadrilateral area QR having each of the line segments of the four sides included in the optimum combination OP as each side, and the determined target area TR is placed on the captured image PI. Set.

By doing so, it is possible to accurately set the target area TR in the captured image PI in various shooting situations of the driver's license DL.

Further, in the first embodiment, the image processing unit 15 determines the upper side line segment and the lower side line segment in each of the combination candidates QRC1, QRC2, and QRC3 based on the selected ruled lines LH1 to LH8.

By doing so, in each of the combination candidates QRC1, QRC2, and QRC3, the optimum upper side line segment and the optimum lower side line segment of the quadrilateral region QR can be determined.

Further, in the first embodiment, the selection ruled lines LH1 to LH8 include a reference line and an auxiliary line having a lower priority than the reference line. Further, the plurality of line segments detected in the captured image PI are a line segment corresponding to the reference line (hereinafter sometimes referred to as a "reference line corresponding line segment") and a line segment corresponding to the auxiliary line (hereinafter referred to as "reference line corresponding line segment"). It may be called "auxiliary line segment"). The image processing unit 15 selects a reference line corresponding line segment (LHC1, LHC2) in a plurality of line segments detected in the captured image PI, and then bases the selected reference line corresponding line segment on the auxiliary line corresponding line segment (LHC3). ~ LHC8) is selected. The reference line-corresponding line segment is sometimes called a "first line segment", and the auxiliary line-corresponding line segment is sometimes called a "second line segment".

By doing so, it is possible to select a reference line corresponding line segment and an auxiliary line corresponding line segment according to different criteria according to the high and low priorities of the reference line and the auxiliary line.

Further, in the first embodiment, the image processing unit 15 calculates the score of each combination candidate based on, for example, the edge pixels on the line segments of the four sides of the combination candidates QRC1, QRC2, and QRC3.

Further, for example, the image processing unit 15 calculates the score of each combination candidate in each of the combination candidates QRC1, QRC2, and QRC3 based on the distance between the end point of the line segment and the other line segment.

Further, for example, the image processing unit 15 calculates the score of each combination candidate based on whether or not each of the combination candidate QRC1, QRC2, and QRC3 includes the auxiliary line corresponding line segment.

Further, for example, the image processing unit 15 calculates the score of each combination candidate based on the edge pixels on the auxiliary line corresponding line segment included in each of the combination candidates QRC1, QRC2, and QRC3.

Further, for example, the image processing unit 15 calculates the score of each combination candidate based on the importance set for the auxiliary line corresponding line segment included in each of the combination candidates QRC1, QRC2, and QRC3.

By doing so, a suitable score can be calculated for each combination candidate.

The first embodiment has been described above.

[Example 2]
The setting position of the selection ruled line is not limited to that shown in FIG. For example, in FIG. 8, the horizontal line that first appears above the lower side of the driver's license DL is set as the selection ruled line LH2. However, in FIG. 8, it is also possible to set the second or third horizontal line appearing above the lower side of the driver's license DL as the selection ruled line LH2. The second or third horizontal line appearing above the bottom of the driver's license DL corresponds to the horizontal rule in the "type" table of the driver's license DL.

[Example 3]
When the reference line candidate satisfying the conditions 1 to 3 does not exist, the image processing unit 15 may use any of the auxiliary line candidates satisfying the conditions 4 and 5 as the reference line candidate.

[Example 4]
The storage unit 13 is realized as hardware, for example, by a memory. Examples of the memory include RAM (Random Access Memory) such as SDRAM (Synchronous Dynamic Random Access Memory), ROM (Read Only Memory), and flash memory.

The control unit 11 and the image processing unit 15 are realized as hardware, for example, by a processor. Examples of the processor include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and the like. Further, the control unit 11 and the image processing unit 15 may be realized by an LSI (Large Scale Integrated circuit) including a processor and peripheral circuits. Further, the control unit 11 and the image processing unit 15 may be realized by using a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or the like.

All or part of each process in the above description in the image processing device 10 may be realized by causing a processor included in the image processing device 10 to execute a program corresponding to each process. For example, the program corresponding to each process in the above description may be stored in the memory, and the program may be read from the memory by the processor and executed. Further, the program is stored in a program server connected to the image processing device 10 via an arbitrary network, downloaded from the program server to the image processing device 10 and executed, or a record readable by the image processing device 10. It may be stored in a medium, read from the recording medium, and executed. The recording medium that can be read by the image processing device 10 may be, for example, a memory card, a USB memory, an SD card, a flexible disk, a magneto-optical disk, a CD-ROM, a DVD, a Blu-ray (registered trademark) disk, or the like. Carrying storage media is included. Further, the program is a data processing method described in an arbitrary language or an arbitrary description method, and may be in any format such as source code or binary code. In addition, the program is not necessarily limited to a single program, but is distributed as multiple modules or multiple libraries, or cooperates with a separate program represented by the OS to achieve its function. Including things.

The specific form of distribution / integration of the image processing device 10 is not limited to the one shown in the figure, and all or a part of the image processing device 10 can be used as an arbitrary unit according to various additions or functional loads. It can be functionally or physically distributed and integrated.

[Example 5]
The target medium to which the disclosed technology is applicable is not limited to a driver's license. For example, the disclosed technology can be applied even when the target medium is a form (for example, an invoice, a delivery note, etc.) or a drawing (for example, a drawing of a machine, a house, a land, etc.).

Further, the processing performed on the image in the target area is not limited to character recognition. For example, when the target medium is a drawing, it is possible to perform dimensional recognition on the image in the target area.

10 Image processing device 11 Control unit 13 Storage unit 15 Image processing unit

Claims (7)

A storage unit that stores captured images and
A plurality of line segments are detected in the captured image, and
Based on the interrelationship of the plurality of line segments, a plurality of candidates for the combination of the four side line segments of the quadrilateral are detected.
Multiple scores for each of the plurality of candidates were calculated, and
Based on the plurality of scores, any one combination from the plurality of candidates is selected.
A target area is determined based on a quadrilateral area having each of the line segments of the four sides included in the selected combination as each side.
An image processing unit that sets the target area on the captured image, and
Equipped with
The image processing unit calculates the plurality of scores based on the distance between the end point of the line segment and the other line segment.
Image processing device. The four sides are the upper side, the lower side, the left side, and the right side.
The image processing unit is based on the selection ruled lines selected based on the layout of the plurality of ruled lines drawn on the surface of the medium to be photographed, and the line segment on the upper side and the line segment on the lower side in each of the plurality of candidates. Determine the line segment,
The image processing apparatus according to claim 1. The selected ruled line includes a first ruled line and a second ruled line having a lower priority than the first ruled line.
The plurality of line segments include a first line segment corresponding to the first ruled line and a second line segment corresponding to the second ruled line.
The image processing unit selects the first line segment in the plurality of line segments, and then selects the second line segment based on the selected first line segment.
The image processing apparatus according to claim 2. The image processing unit calculates the plurality of scores based on whether or not the plurality of line segments include the second line segment.
The image processing apparatus according to claim 3. The image processing unit calculates the plurality of scores based on the edge pixels on the second line segment included in the plurality of line segments.
The image processing apparatus according to claim 3. The image processing unit calculates the plurality of scores based on the importance set for the second line segment included in the plurality of line segments.
The image processing apparatus according to claim 3. Detects multiple line segments in the captured image and
Based on the interrelationship of the plurality of line segments, a plurality of candidates for the combination of the four side line segments of the quadrilateral are detected.
Multiple scores for each of the plurality of candidates were calculated, and
Based on the plurality of scores, any one combination from the plurality of candidates is selected.
A quadrilateral region having each of the line segments of the four sides included in the selected combination as each side is determined as a target area.
An image processing method for setting the target area on the captured image.
Calculate the plurality of scores based on the distance between the end points of the line segment and the other line segments.
Image processing method.

Images