JP2021039698A - Image processing device, method, and computer program - Google Patents

Abstract

To accurately determine the area of an object in an image.SOLUTION: An image processing device specifies one or more line segments in an object image, determines multiple area-defining line segments, which are line segments for determining the area of an object based on one or more line segments, determines whether a first line segment of the multiple area-defining line segments satisfies a change condition based on a specific pixel located in a specific area, changes the line segment that should be included in the multiple area-defining line segments from the first line segment to a second line segment, when the change condition is satisfied, and determines the area of the object using the multiple area-defining line segments. The specific pixel is either a pixel that constitutes the object or a pixel that defines the outer edge of the object. The second line segment is located outside a temporary area, which is the area of the object tentatively determined by the multiple area-defining line segments including the first line segment. When the change condition is satisfied, the area of the object is determined using the multiple area-defining line segments including the second line segment.SELECTED DRAWING: Figure 7

Description

The present specification relates to a technique for determining an area of an object in an image.

The image reading device described in Patent Document 1 extracts a rectangular area of a document in an image obtained by scanning the document. At that time, the image reader detects a plurality of straight lines by using the Hough transform or the least squares method, and determines a candidate region composed of four straight lines selected from the plurality of straight lines. The image reading device evaluates a plurality of candidate areas and determines one candidate area as a document area based on the evaluation result.

JP-A-2018-101852 Japanese Unexamined Patent Publication No. 2010-263434

However, in the above technique, a candidate region of the document region is constructed by a straight line detected by using the Hough transform or the least squares method. For this reason, if an appropriate straight line is not detected by using the Hough transform or the least squares method, such as when the straight line to be detected is unclear, the original area may not be determined accurately.

The present specification discloses a technique capable of accurately determining an area of an object in an image.

The techniques disclosed herein can be realized as the following application examples.

[Application Example 1] An image processing device, an image acquisition unit that acquires target image data indicating a target image, a line segment identification unit that specifies one or more line segments in the target image, and the above-mentioned one. Of the line segment determination unit that determines a plurality of area-defined line segments, which are line segments for determining the area of the object in the target image based on the above line segments, and the plurality of area-defined line segments. Is a condition determination unit that determines whether or not the first line segment of the above satisfies the change condition based on the specific pixel located in the specific area, and the specific area is an area determined by the plurality of area defined line segments. The specific pixel is either a pixel constituting the object or a pixel defining the outer edge of the object, which is a region located outside the above and located in the direction in which the first line segment intersects. A determination unit and a first specified line segment changing unit that changes a line segment to be included in the plurality of area specified line segments from the first line segment to the second line segment when the change condition is satisfied. The first line segment is a line segment located at the outer edge of the region determined by the plurality of region-defined line segments, and the second line segment is a plurality of region-defined line segments including the first line segment. A first defined line segment changing portion, which is a line segment located outside the region determined by the above, and a region determining portion for determining the region of the object using the plurality of region defined line segments. When the change condition is satisfied, the area of the object is determined using the plurality of area-defined line segments including the second line segment, and when the change condition is not satisfied, the first line segment is used. An image processing device including the area determination unit that determines an area of the object using the plurality of area defining line segments including the above.

According to the above configuration, when the change condition based on the specific pixel located in the specific area is satisfied, the second line segment located outside the area determined by the area defined line segment before the change including the first line segment. The area of the object is determined using multiple area-defined line segments including two line segments, and if the change conditions are not satisfied, the area of the object is determined using multiple area-defined line segments including the first line segment. Is determined. As a result, the area of the object can be created by using an appropriate area defining line segment, so that the area of the object can be accurately determined in the target image.

The techniques disclosed in the present specification can be realized in various aspects, for example, an image processing method, an image processing device, a computer program for realizing the functions of those methods or devices, and a computer thereof. It can be realized in the form of a recording medium on which the program is recorded (for example, a recording medium that is not temporary).

The figure which shows the structure of the multifunction device as one Example. The flowchart which shows the example of the reading process. The schematic diagram which shows an example of the image used in this Example. Flowchart of object area identification process. Explanatory drawing of the specific process of an object area. Flow chart of area definition line segment determination processing. Flowchart of area definition line segment correction processing. The first explanatory diagram of the process for another scanned image. The second explanatory view of the process for another scanned image.

A. First Example:
A1. Device configuration:
FIG. 1 is a diagram showing a configuration of a multifunction device as an embodiment. The multifunction device 800 includes a display unit 840, an operation unit 850, a reading unit 900, a printing unit 860, and a control unit 810 that controls the multifunction device 800. These elements are connected to each other via a bus.

The display unit 840 is a device for displaying an image, such as a liquid crystal display and an organic EL display. The operation unit 850 is a device that receives an operation by the user, and includes a button, a lever, a touch panel arranged on the display unit 840, and the like.

The reading unit 900 is a device that uses an image sensor 920 to generate image data (also referred to as scan data) indicating a document, and is, for example, a flatbed scanner device. The reading unit 900 will be further described later.

The printing unit 860 is a device that prints an image on paper (an example of a printing medium) by a predetermined method (for example, a laser method or an inkjet method). In this embodiment, the printing unit 860 is an inkjet printing device capable of printing a color image using four types of inks, cyan C, magenta M, yellow Y, and black K.

The control unit 810 includes a processor 811 and a storage device 815. The storage device 815 includes a volatile storage device 812 and a non-volatile storage device 813. The processor 811 is an arithmetic unit that performs data processing, and is, for example, a CPU. The volatile storage device 812 is, for example, a DRAM, and the non-volatile storage device 813 is, for example, a flash memory. These elements are connected to each other via a bus. The non-volatile storage device 813 stores the computer program 814.

The volatile storage device 812 provides a buffer area for temporarily storing various intermediate data generated when the processor 811 performs processing. The computer program 814 is stored in the non-volatile storage device 813. In this embodiment, the computer program 814 is provided in a form pre-stored in the non-volatile storage device 813 at the time of manufacturing the multifunction device 800. Instead, the computer program 814 may be provided in a form downloaded from a server, or may be provided in a form stored in a DVD-ROM or the like. The processor 811 realizes the control of the multifunction device 800 by executing the computer program 814. For example, the processor 811 can execute the reading process described later.

A perspective view of the multifunction device 800 is shown in the lower part of FIG. In the figure, the direction Dp1 and the direction Dp2 indicate a horizontal direction, and the direction Dp3 indicates a vertically upward direction. The direction Dp1 and the direction Dp2 are perpendicular to each other. The direction Dp3 is also referred to as an upward Dp3. The configuration of the reading unit 900 will be further described with reference to this perspective view. In the perspective view, the thin elements related to the reading unit 900 are not shown as appropriate.

The reading unit 900 includes an image sensor 920, a moving device 930, a platen 940, and a cover 950. The platen 940 is provided on the upward Dp3 side of the housing 890. The document table 940 is a substantially rectangular table surrounded by two sides parallel to the direction Dp1 and two sides parallel to the direction Dp2, and is configured by using a transparent plate (for example, a glass plate). A document to be read is arranged on the upper surface Us of the platen 940 on the upward Dp3 side. In FIG. 1, two original documents C1 and C2 are arranged apart from each other on the upper surface Us of the platen 940.

The cover 950 is attached to the housing 890 on the upward Dp3 side so as to be openable and closable. FIG. 1 shows a cover 950 in an open state. When the cover 950 is in the closed state (not shown), the cover 950 covers the platen 940.

The image sensor 920 is arranged on the lower side (opposite side of the upward Dp3) of the platen 940 in the housing 890. The image sensor 920 is a one-dimensional image sensor that optically reads a document, and has a configuration in which a plurality of photoelectric conversion elements (also simply referred to as optical elements) such as a CCD or CMOS are arranged side by side in the direction Dp1. doing.

The mobile device 930 is arranged inside the housing 890. The mobile device 930 includes a power source (eg, an electric motor). The moving device 930 uses a power source to move the image sensor 920 in a direction along the upper surface Us of the platen 940 (specifically, a direction parallel to the direction Dp2).

The moving device 930 moves the image sensor 920 from the position of the end of the platen 940 in the direction opposite to the direction Dp2 toward the direction Dp2 under the control of the processor 811. While moving, the image sensor 920 optically reads the document placed on the platen 940 under the control of the processor 811. The processing circuit (not shown) of the reading unit 900 generates scan data indicating a document based on the electric signal.

A2. Reading process:
FIG. 2 is a flowchart showing an example of the reading process. In the scanning process, the document is scanned by the scanning unit 900, and image data of each document is generated. Hereinafter, it is assumed that the two original documents C1 and C2 of FIG. 1 are read. The manuscripts C1 and C2 are assumed to be, for example, receipts, business cards, books, posters, A4 or A5 size paper, postcards, and the like. In the following, a case where the documents C1 and C2 are receipts will be described as an example.

The processor 811 starts the reading process in response to the user inputting the reading processing start instruction to the operation unit 850. In S110, the processor 811 acquires scan data as target image data. Specifically, the processor 811 controls the reading unit 900 to cause the reading unit 900 to generate scan data, and acquires the scan data from the reading unit 900. For example, the scan data is RGB image data. The RGB image data is bitmap data including the values of a plurality of pixels. Each of the values of the plurality of pixels represents the color of the pixel as an RGB color value (also referred to as an RGB value).

FIG. 3 is a schematic view showing an example of an image used in this embodiment. FIG. 3A shows an example of the scanned image SI represented by the scan data. The scanned image SI includes a plurality of pixels arranged in a matrix along the horizontal direction Dx and the vertical direction Dy perpendicular to the horizontal direction Dx. The horizontal direction Dx is, for example, a direction parallel to the direction Dp1 in FIG. 1, and the vertical direction Dy is a direction parallel to the direction Dp2 in FIG.

The scanned image SI of FIG. 3A includes a manuscript image CO1 showing the manuscript C1 as an object and a manuscript image CO2 showing the manuscript C2 as an object. The original image CO1 includes a base Ba, characters TXa to TXe, and line segments ILb to ILe. The manuscript image CO2 includes a base Bb, characters TXf to TXj, and line segments ILf to ILj. Depending on the manuscripts C1 and C2 to be read, the manuscript images CO1 and CO2 may include other elements such as photographs and charts. The manuscripts C1 and C2 of this embodiment are receipts cut from roll paper using a blade having a jagged shape. For this reason, among the outer edges of the original image CO1, the outer edges OLa and OLb at both ends of the vertical direction Dy corresponding to the longitudinal direction of the roll paper have a jagged shape and correspond to the lateral direction of the roll paper. The outer edges OLc and OLd at both ends of Dx have a straight shape. Similarly, among the outer edges of the original image CO2, the outer edges OLe and OLf at both ends in the vertical direction Dy have a jagged shape, and the outer edges OLg and OLh at both ends in the horizontal direction Dx have a straight shape.

In S120, the processor 811 executes an edge detection process on the scan data to generate edge image data. The edge detection process is a process of detecting edge pixels from a plurality of pixels in the scanned image SI. The edge image data generates edge image data which is binary image data indicating edge pixels and non-edge pixels.

The edge pixel detection method may be various methods. In this embodiment, the processor 811 uses a so-called Laplacian filter to calculate the edge strength of each pixel (for example, the absolute value of the calculation result by the filter). Then, the processor 811 specifies a pixel having an edge strength greater than the threshold value TH1 as an edge pixel, and a pixel having an edge strength equal to or less than the first threshold value as a non-edge pixel. The Laplacian filter may be applied to a specific color component value (for example, the color component value of green G) indicated by the target image data. Instead, the Laplacian filter may be applied to a color component value (eg, luminance value) calculated from the scan data. The threshold value TH1 is experimentally determined in advance so that edge pixels can be appropriately detected. Instead, the processor 811 may adjust the threshold TH1. For example, the processor 811 may adjust the threshold TH1 according to the scan data.

FIG. 3B shows an example of the edge image EI, which is an image represented by the edge image data. In this edge image EI, the edge pixels Eo corresponding to the outer edges OLa to OLh of the original images CO1 and CO2 of the scanned image SI, and the edge pixels corresponding to the characters TXa to TXj and the line segments ILa to ILj in the original images CO1 and CO2. Ei and are specified.

In S130, the processor 811 uses the edge image data to execute a specific process for specifying an object area in the scanned image SI. In this embodiment, a region showing the originals C1 and C2 as objects, that is, a rectangular region corresponding to the original images CO1 and CO2 is specified. Here, the original image CO2 contains the line segments ILf to ILi that form a rectangle (FIG. 3 (A)). In such a case, the rectangular area corresponding to the line segments ILf to ILi constituting the rectangle may be erroneously specified as the object area. In the example of FIG. 3B, three object areas SA1, SA2, and SA3 are specified. Details of the object area identification process will be described later.

In S140, the processor 811 excludes the object area included in the other object area from the processing target among the plurality of object areas. This is because the object area included in the other object area is considered to be an area corresponding to the elements included in the document, not the area corresponding to the documents C1 and C2 to be detected. In the example of FIG. 3B, the processor 811 uses the coordinates of each of the four corner points of the object areas SA1 to SA3 to specify the inclusion relationship of the rectangle composed of the four corner points. The processor 811 excludes the object area SA3 included in the object area SA2 from the processing target.

In S150, the processor 811 selects one object area as the object area of interest from the object areas to be processed. In the example of FIG. 3B, the two object areas SA1 and SA2 are sequentially selected as the object areas of interest.

In S160, the processor 811 executes a crop process for cutting out a portion of the scan data indicating the object area of interest (for example, the object area SA1 (FIG. 3B)). As a result, the object image data corresponding to the object area of interest is generated. In S170, the processor 811 executes a process of correcting the inclination of the object area of interest with respect to the object image data corresponding to the object area of interest. When the document (for example, C1) is arranged at an angle with respect to the direction Dp1 on the upper surface Us (FIG. 1) of the platen 940, the left side or the right side of the object area (for example, SA1) is arranged in the scanned image SI. Tilt with respect to the vertical Dy. Specifically, the processor 811 calculates the angle of one side (for example, the left side) of the object area of interest (for example, the object area SA1 (FIG. 3B)) with respect to the vertical Dy. Then, by rotating the object area of interest by the angle, the left side of the object area of interest is made parallel to the vertical Dy. As a result, the corrected object image data corresponding to the object area of interest is generated. The corrected object image data indicates an object (manuscripts C1 and C2 in this embodiment) in the scanned image SI. For example, the original image TI1 of FIG. 3C is an image indicated by the corrected object image data corresponding to the object area SA1. The original image TI2 of FIG. 3D is an image shown by the corrected object image data corresponding to the object area SA2.

In S180 (FIG. 2), the processor 811 stores the corrected object image data generated in S170 in the non-volatile storage device 813 (FIG. 1). Instead, the processor 811 may store the corrected object image data in a storage device designated by the user (for example, a portable storage device (not shown) such as a USB memory connected to the multifunction device 800). Good.

In S190, the processor 811 determines whether or not the processing of all the object areas to be processed has been completed. If there is an unprocessed object area (S190: No), the processor 811 returns to S150. When all the object areas have been processed (S180: Yes), the processor 811 ends the processing of FIG.

A3. Object Area Identification Process The object area identification process of S130 in FIG. 2 will be described. FIG. 4 is a flowchart of the object area specifying process. In S200, the processor 811 uses the edge image data generated in S120 of FIG. 2 to detect a plurality of line segments in the scanned image SI and label the detected plurality of line segments. Execute. Specifically, the processor 811 identifies a plurality of line segments by a Hough transform using edge pixels. Processor 811 classifies into one or more groups based on the distance between the specified plurality of line segments. For example, a plurality of line segments are classified so that the minimum distance between line segments belonging to the same group is equal to or less than the threshold value and the minimum distance between line segments belonging to different groups is equal to or greater than the threshold value.

For example, in the example of FIG. 3, a plurality of line segments corresponding to the original image CO1 are labeled as one group. Further, in the original image CO2, a plurality of line segments iLf to ILi forming an internal rectangle are labeled as one group, and a plurality of lines corresponding to the portion excluding the internal rectangle (for example, the outer edge portion). Minutes are labeled as a group.

In S205, the processor 811 selects one group as the attention group from the plurality of labeled groups.

The processing of S210 to S270 is executed for the attention group. As a result, a rectangular object area is determined for each labeled group. In the following, the processes of S210 to S270 will be described by taking as an example the case where the group including a plurality of line segments corresponding to the original image CO1 of FIG. 3A is the group of interest.

FIG. 5 is an explanatory diagram of the object area specifying process. FIG. 5A shows a plurality of line segments L1 to L7 corresponding to the original image CO1 as an example of the plurality of line segments belonging to the group of interest.

In S210, the processor 811 determines one reference line segment from the plurality of line segments L1 to L7 belonging to the group of interest. Specifically, the center of gravity CO of a plurality of line segments L1 to L7 is specified. For example, the coordinates of the midpoint of each line segment are calculated as the coordinates of the center of gravity of each line segment. The weighted average coordinates of the coordinates of the centers of gravity of each of the plurality of line segments L1 to L7 are calculated as the coordinates of the center of gravity CO of the line segments L1 to L7. The weight is the length of each line segment. The processor 811 calculates the feature amount CV of each line segment based on the center of gravity CO and the length of each line segment. For example, as illustrated for the line segment L4 in FIG. 5 (A), where D is the distance between the center of gravity CO and the line segment L4 and the length of the line segment L4 is LE, the feature amount CV of the line segment L4. Is the sum of the distance D and the length LE (CV = D + LE). The processor 811 determines the line segment having the largest feature amount CV among the line segments L1 to L7 as the reference line segment. In the example of FIG. 5A, the line segment L4 is determined as the reference line segment. As described above, the selection criterion of the reference line segment is a criterion based on the feature amount CV. Therefore, the selection criteria of the reference line segment are the distance D between the center of gravity CO determined based on one or more line segments and each of the one or more line segments, and the length of each of the one or more line segments. It can be said that it is a standard based on LE. By using such a selection criterion, the reference line segment can be appropriately selected.

In S220, the processor 811 excludes a line segment having a low reliability from the plurality of line segments L1 to L7 belonging to the attention group from the attention group. Specifically, the processor 811 excludes a line segment having a length shorter than a predetermined reference length Lr from the attention group. The reference length Lr is, for example, a value determined based on the length of the reference line segment, and is, for example, a value obtained by multiplying the length of the reference line segment by a predetermined coefficient (for example, 0.1 to 0.3). Is. This is because a line segment shorter than the reference length Lr is considered to be noise or a minute line segment inside the document, and is unlikely to be a line segment corresponding to the outer edge of the document. Further, the processor 811 excludes a line segment whose angle θ1 between the reference line segment and the reference line segment or the angle θ2 between the reference line segment and the line perpendicular to the reference line segment is equal to or greater than the reference angle θr from the attention group. The reference angle θr is, for example, -3 degrees to +3 degrees. In other words, a line segment that is approximately parallel to the reference line segment (a line segment that extends along a direction parallel to the reference line segment) and a line segment that is approximately perpendicular to the reference line segment (along the direction perpendicular to the reference line segment). (Extended line segments) and the remaining line segments except for are excluded from the attention group. This is because the area of the document is considered to be a rectangular area having a reference line segment as one side. In the example of FIG. 5, among the line segments L1 to L7 of FIG. 5A, minute line segments L5 to L7 are excluded. FIG. 5B shows the remaining line segments L1 to L4.

In S230, the processor 811 determines four candidate line segments that are candidates for the four sides of the rectangle of the object area. Specifically, one candidate line segment is the reference line segment determined in S210. The remaining three candidate line segments are one candidate line segment along the first direction D1 (also referred to as a parallel candidate line segment) parallel to the reference line segment and perpendicular to the reference line segment. Two candidate line segments (also referred to as a first vertical candidate line segment and a second vertical candidate line segment) along the second direction D2 (FIG. 5 (B)).

The parallel candidate line segment is determined to be a line segment that satisfies the following two conditions A) and B) among the line segments L1 to L4 belonging to the group of interest.
Condition A) The angle θ1 (angle formed with the first direction D1) with the reference line segment is less than the reference angle θr.
Condition B) The position of the second direction D2 is on the opposite side of the reference line segment with the center of gravity CO in between.
When there are a plurality of line segments satisfying the conditions A) and B), the line segment farthest from the reference line segment is determined as the parallel candidate line segment. In the example of FIG. 5B, since the reference line segment is the line segment L4, the line segment L3 is determined as the parallel candidate line segment.

The first vertical candidate line segment is determined to be a line segment that satisfies the following two conditions C) and D) among the line segments L1 to L4 belonging to the group of interest.
Condition C) The angle θ2 (the angle formed by the second direction D2) between the reference line segment and the vertical line is less than the reference angle θr.
Condition D) Among the line segments satisfying the condition C), the position of the first direction D1 is the most in the first direction D1.
In the example of FIG. 5B, since the reference line segment is the line segment L4, the line segment L1 is determined as the first vertical candidate line segment.

The second vertical candidate line segment is determined to be a line segment that satisfies the following two conditions E) and F) among the line segments L1 to L4 belonging to the group of interest.
Condition E) The angle θ2 (the angle formed by the second direction D2) between the reference line segment and the vertical line is less than the reference angle θr.
Condition F) The position of the first direction D1 is on the opposite side of the first vertical candidate line segment with the center of gravity CO in between.
When there are a plurality of line segments satisfying the conditions E) and F), the line segment farthest from the first vertical candidate line segment is determined as the second vertical candidate line segment. In the example of FIG. 5B, since the reference line segment is the line segment L4, the line segment L2 is determined as the second vertical candidate line segment.

Here, all or part of one parallel candidate line segment and two vertical candidate line segments may not have a line segment satisfying the above conditions. In this case, of the one parallel candidate line segment and the two vertical candidate line segments, the line segment that does not satisfy the condition is not determined.

In S240, the processor 811 determines the extrinsic rectangle SQ that includes the line segments L1 to L4 that belong to the group of interest. The circumscribed rectangle SQ is determined so that the two opposite sides SL1 and SL2 of the four sides are parallel to the first direction D1 and the remaining two sides SL3 and SL4 are parallel to the second direction D2. Further, the circumscribing rectangle SQ is determined so that each of the four sides SL1 to SL4 touches any of the line segments L1 to L4 belonging to the group of interest. In the example of FIG. 5B, the sides SL1 and SL2 of the extrinsic rectangle SQ are in contact with the ends of the line segment L4, respectively. The side SL3 is in contact with the entire line segment L3, and the side SL4 is in contact with the entire line segment L4.

In S250, the processor 811 executes the area-defined line segment determination process. In the region-defined line segment determination process, the processor 811 uses an object based on the reference line segment determined in S210, the candidate line segment determined in S230, and the circumscribing rectangle SQ determined in S240. Four area-defined line segments, which are line segments for determining the area, are determined.

FIG. 6 is a flowchart of the area defined line segment determination process. In S310, the processor 811 selects one side of interest from the four sides SL1 to SL4 of the circumscribed rectangle SQ.

In S320, the processor 811 determines whether or not the processor 811 has determined the candidate line segment corresponding to the side of interest. For example, the sides SL1, SL2, SL3, and SL4 of the circumscribing rectangle SQ in FIG. 5B correspond to the first vertical candidate line segment, the second vertical candidate line segment, the parallel candidate line segment, and the reference line segment, respectively. There is. In the example of FIG. 5B, all of the first vertical candidate line segment, the second vertical candidate line segment, the parallel candidate line segment, and the reference line segment are determined. The candidate line segment corresponding to the attention side is also called the attention candidate line segment.

When the candidate line segment corresponding to the attention side is determined (S320: YES), in S330, the processor 811 calculates the ratio LR of the length of the attention candidate line segment to the length of the attention side. As shown in FIG. 5B, the length of the side SL1 of the circumscribed rectangle SQ is La, and the length of the line segment L1 is Lb. When the side of interest is the side SL1 and the candidate line segment of interest is the line segment L1, the length Lb of the line segment L1 with respect to the length La of the side SL1 of the circumscribing rectangle SQ is calculated as the ratio LR (LR). = (Lb / La)).

In S340, the processor 811 determines whether the ratio LR is greater than the threshold TH2. The threshold TH2 is, for example, a value of 0.5 to 0.7. When the ratio LR is larger than the threshold value TH2 (S340: YES), in S360, the processor 811 determines the candidate line segment of interest as the region-defined line segment. This is because when the ratio LR is larger than the threshold value TH2, it is considered that there is a high possibility that the line segment of interest corresponds to the outer edge of the document. When the ratio LR is equal to or less than the threshold value TH2 (S340: NO), in S350, the processor 811 determines the side of interest as a region-defined line segment. When the ratio LR is equal to or less than the threshold value TH2, the line segment corresponding to the outer edge of the original is not detected, and it is considered unlikely that the candidate line segment of interest is the line segment corresponding to the outer edge of the original. Is. Even when the candidate line segment corresponding to the attention side is not determined (S320: NO), in S350, the processor 811 determines the attention side as the area-defined line segment.

In the example of FIG. 5B, regardless of whether the side of interest is any of the sides SL1 to SL4 of the extrinsic rectangle SQ, the area-defined line segment is not the side of interest (sides SL1 to SL4 of the extrinsic rectangle SQ) but a candidate of interest. A line segment (any of line segments L1 to L4) is determined. An example in which the side of interest is determined as the region defining line segment will be described later.

In S370, the processor 811 determines whether or not all the sides of the circumscribed rectangle SQ are processed as the sides of interest. If there is an unprocessed edge (S370: NO), the processor 811 returns to S310. When all the sides of the circumscribed rectangle SQ have been processed (S370: YES), the processor 811 ends the area-defined line segment determination process.

When the area-defined line segment determination process is completed, in S260 of FIG. 4, the processor 811 executes the area-defined line segment correction process. The area-defined line segment correction process is a process of evaluating the area-defined line segment determined by the area-defined line segment determination process of S250 and appropriately correcting the area-defined line segment.

FIG. 7 is a flowchart of the area defined line segment correction process. In S410, one line segment of interest is selected from the four area-defined line segments determined by the area-defined line segment determination process of S250. In the example of FIG. 5C, since the four region-defined line segments are line segments L1 to L4, the line segment of interest is selected from these line segments L1 to L4.

In S420, the processor 811 determines whether or not the straight line including the line of interest intersects another line segment of interest (two area-defined line segments substantially perpendicular to the line segment of interest). For example, when the line segment L1 in FIG. 5C is the line segment of interest, the straight line including the line segment L1 intersects with any of the other two area-defined line segments L3 and L4. Whether or not it is judged. The straight line including the line segment L1 intersects the line segment L4 at the point P1 and intersects the line segment L3 at the point P2. Therefore, in this case, it is determined that the straight line including the line segment of interest intersects with another area-defined line segment. The straight line including the line segment L2 intersects the line segment L4 at the point P4 and intersects the line segment L3 at the point P3. Therefore, even when the line segment L2 is the line segment of interest, it is determined that the straight line including the line segment of interest intersects with another area-defined line segment. The straight line including the line segment L3 and the straight line including the line segment L4 do not intersect with the line segments L1 and L2. Therefore, when the line segments L3 and L4 are the line segments of interest, it is determined that the straight line including the line segment of interest does not intersect with other area-defined line segments.

When a straight line including a line segment of interest intersects another area-defined line segment (S420: YES), in S430, the processor 811 transfers the line segment of interest to the end of the other area-defined line segment that intersects. shift. For example, when the line segment of interest is the line segment L1, the position of the first direction D1 of the line segment L1 is the end P1t of the first direction D1 of the line segment L4 and the first direction D1 of the line segment L3. It is changed to the outermost position of the end P2t. In the example of FIG. 5C, since the end portion P1t of the line segment L4 is outside the end portion P2t of the line segment L3 (on the side of the first direction D1), the position of the first direction D1 of the line segment L1 is , The position is changed to the position of the end P1t of the line segment L4. The line segment L1m in FIG. 5C is a line segment obtained by changing the position of the line segment L1 in the first direction D1. It can be said that the end portion P1t of the line segment L4 is the end portion of both ends P1t and P4t of the line segment L4 having a long distance in the first direction D1 from the line segment L3. When the line segment of interest is the line segment L2, the position of the line segment L2 in the first direction D1 is changed to the position of the end portion P4t on the opposite side of the line segment L4 from the first direction D1. The line segment L2m in FIG. 5C is a line segment obtained by changing the position of the line segment L2 in the first direction D1.

If the straight line including the line segment of interest does not intersect with another area-defined line segment (S420: NO), the processor 811 skips S430 and proceeds to S440.

In S440, the processor 811 determines whether or not all of the four area-defined line segments have been processed as the line segments of interest. If there is an unprocessed area defined line segment (S440: NO), the processor 811 returns to S410. When all the area-defined line segments have been processed (S440: YES), the processor 811 proceeds to S450.

In S450, as in S410, one line segment of interest is selected from the four area-defined line segments. FIG. 5D shows four region defining line segments L1m, L2m, L3, and L4 at the time when the processing is advanced to S450. In the example of FIG. 5D, the line segment of interest is selected from the four region-defined line segments L1m, L2m, L3, and L4.

In S460, the processor 811 calculates the percentage ER of the edge pixels located on the line segment of interest. Specifically, the processor 811 refers to the edge image data generated in S120 of FIG. 2 and counts the number of edge pixels located on the line segment of interest. The processor 811 calculates the ratio ER by dividing the number of edge pixels by the total number of pixels located on the line segment of interest.

In S470, the processor 811 determines whether or not the ratio ER of the edge pixels is larger than the threshold value TH3. The threshold value TH3 is, for example, a value of 0.2 to 0.4. In FIG. 5D, in addition to the line segments L1m, L2m, L1n, L2n, L3, and L4, edge pixels Eo1 and Eo2 corresponding to the jagged outer edge of the document C1 are shown for explanation. For example, when the area-defined line segments L1m and L2m in FIG. 5D are the line segments of interest, the edge pixels Eo1 and Eo2 corresponding to the jagged outer edges of the document C1 are present on the line segments L1m and L2m. Therefore, the ratio ER becomes larger than the threshold value TH3.

When the ratio ER of the edge pixels is larger than the threshold value TH3 (S470: YES), the processor 811 shifts the line segment of interest to the outside at S480. Here, the direction toward the outside (also referred to as a shift direction) is a direction perpendicular to the line segment of interest and a direction toward the line segment of interest from the center of gravity CO. In other words, the outside means the outside of the rectangular area determined by the current area defining line segments L1m, L2m, L3, and L4. This region is a region surrounded by four straight lines including the current region defining line segments L1m, L2m, L3, and L4, and is a region including the center of gravity CO.

Therefore, when the line segment of interest is the area-defined line segment L1m (FIG. 5 (D)), the shift direction is the first direction D1 and the line segment of interest is the area-defined line segment L2m (FIG. 5 (D)). If, the shift direction is the opposite direction of the first direction D1. In this embodiment, the line segment of interest after the shift is a line segment tangent to the line segment of interest before the shift in the shift direction. Therefore, when the line segment of interest before the shift is the area-defined line segment L1m, the line segment of interest after the shift is a line segment tangent to the first direction D1 with respect to the area-defined line segment L1m (not shown). When the line segment of interest is the area-defined line segment L2m, the line segment of interest after the shift is a line segment tangent to the area-defined line segment L2m in the opposite direction of the first direction D1 (not shown).

When the line segment of interest is shifted in S480, the processor 811 returns to S460 and calculates the ratio ER of the edge pixels for the line segment of interest after the shift.

When the ratio ER of the edge pixels is equal to or less than the threshold value TH3 (S470: NO), in S490, the processor 811 determines the line segment of interest at the current position as the final area-defined line segment. When the line segment of interest selected in S450 is the area-defined line segment L1m, it shifts to a position tangent to the outside of the edge pixel Eo1 corresponding to the jagged outer edge (the position of the line segment L1n in FIG. 5D). Will be done. Then, the line segment L1n located at a position tangent to the outside of the edge pixel Eo1 is determined as the final region-defined line segment. When the line segment of interest selected in S450 is the area-defined line segment L2m, the line segment L2n located at a position tangent to the outside of the edge pixel Eo2 is determined as the final area-defined line segment.

On the other hand, when the line segment of interest selected in S450 is the area-defined line segment L3 or L4, although edge pixels exist on the line segment L3 or L4, the line segment is more than the line segment L3 or L4. There are almost no edge pixels on the outside. Therefore, in this case, the shift of the line segment of interest is performed only once, for example. Therefore, in this case, the line segments L3 and L4 are hardly shifted, and the line segments L3 and L4 are substantially determined as the final region-defined line segments as they are.

In S495, the processor 811 determines whether or not all of the four area-defined line segments have been processed as the line segments of interest. If there is an unprocessed area-defined line segment (S495: NO), the processor 811 returns to S450. When all the area-defined line segments have been processed (S495: YES), the processor 811 ends the area-defined line segment correction process.

When the area-defined line segment correction process is completed, in S270 of FIG. 4, the processor 811 identifies the object area based on the four finally determined area-defined line segments. Specifically, the four intersections of the four straight lines including the four region defining line segments are specified as the four corner points of the object region. For example, when the four region-defined line segments are L1n, L2n, L3, and L4 shown in FIG. 5 (D), as shown in FIG. 5 (E), a straight line and a line segment including the line segment L1n. The intersection Pa of the straight line including L4, the intersection Pb of the straight line including the line L1n and the straight line including the line L3, the intersection Pc of the straight line including the line L3 and the straight line including the line L2n, and the line segment. The intersection Pd of the straight line including L2n and the straight line including the line segment L4 is specified. Then, a rectangular object area SA1 (FIG. 5 (E)) obtained by connecting the four intersections Pa to Pd is specified.

In S280, the processor 811 determines whether or not processing has been completed for all the groups of line segments specified in S200. If there is an unprocessed group (S280: NO), processor 811 returns to S205. When all the groups have been processed (S280: YES), the processor 811 ends the object area identification process. As shown in FIG. 3B, for example, in the scanned image SI, three object areas SA1 to SA3 are specified.

Regarding the object area determination process described above, an example will be described in which scan image data indicating another scan image is acquired. FIG. 8 is a first explanatory view of processing for another scanned image. FIG. 8A shows the original images CO1t and CO2t included in another scanned image. The manuscript image CO1t is an image showing one page of the book in the spread state, and the manuscript image CO2t is an image showing the other page of the book in the spread state. This image further includes a dark portion image SH having a dark color between the original image CO1t and the original image CO2t. A part of the character TX in the original images CO1t and CO2t overlaps with the dark part image SH.

FIG. 8B shows the result of detecting the line segment of S200 of FIG. 4 with respect to the scan data showing the image of FIG. 8A. In FIG. 8B, the line segments L1a to L3a and the line segments L1b to L3b indicate the detected line segments.

In the example of FIG. 8A, the three outer edges OLg to OLi of the original image CO1t and the three outer edges OLj to OLl of the original image CO2t clearly appear in the image. Therefore, in the example of FIG. 8B, the line segments L1a to L3a corresponding to the outer edges OLg to OLi of the original image CO1t and the line segments L1b to L3b corresponding to the outer edges OLj to OLl of the original image CO2t are It has been detected. On the other hand, in the example of FIG. 8A, the outer edge on the right side of the original image CO1t and the fold portion of the book corresponding to the outer edge on the left side of the original image CO2t are not in close contact with the document base 940 at the time of scanning. , Does not clearly appear in the image of FIG. 8 (A). In the image of FIG. 8A, a dark portion image SH appears in the portion corresponding to the crease. In the dark part image SH, the boundary portion with the background is blurred and the color is gradually changed (gradation), so that the edge pixels constituting the clear line segment are not detected. Therefore, in the example of FIG. 8B, the line segment corresponding to the outer edge on the right side (+ Dx side) of the original image CO1t and the outer edge on the left side (−Dx side) of the original image CO2t is not detected.

Therefore, in this example, when determining the candidate line segment in S230 of FIG. 4, as shown in FIG. 8C, three candidates corresponding to the three outer edges OLg to OLi of the original image CO1t. The line segments L1a to L3a are determined, but the candidate line segments corresponding to the outer edge on the right side of the original image CO1t are not determined. Further, the three candidate line segments L1b to L3b corresponding to the three outer edges OLj to OLl of the original image CO2t are determined, but the candidate line segments corresponding to the outer edges on the left side of the original image CO2t are not determined.

FIG. 8C shows the circumscribed rectangles SQa and SQb determined in S240 of FIG. In the example of FIG. 8, in the area-defined line segment determination process (FIG. 6) of S250 of FIG. 4, the right-hand side SLa of the circumscribed rectangle SQa is determined as the area-defined line segment corresponding to the outer edge on the right side of the original image CO1t (FIG. 6). NO in S320 of FIG. 6, S350 of FIG. 6). Similarly, in the area-defined line segment determination process (FIG. 6), the right-hand side SLb of the circumscribed rectangle SQb is determined as the area-defined line segment corresponding to the left outer edge of the original image CO2t (NO in S320 of FIG. 6). S350 in FIG. 6).

As described above, it can be seen that the four region-defined line segments can be determined even when the line segments corresponding to a part of the outer edges of the original images CO1t and CO2t are not detected. FIG. 8D shows four region-defined line segments L1a to L3a and SLa corresponding to the original image CO1t and four region-defined line segments L1b to L3b and SLb corresponding to the original image CO2t. Has been done.

In this example, when the region-defined line segment correction processing of FIG. 7 is performed on the four region-defined line segments L1a to L3a and SLa of the original image CO1t of FIG. 8 (D), the region-defined line segment on the right side is performed. Outside the SLa, there is an edge pixel Ea that constitutes an edge of the character TXa (FIG. 8 (A)). Therefore, the ratio ER of the edge pixels on the region-defined line segment SLa becomes larger than the threshold value TH3 (YES in S470 of FIG. 7). Therefore, the region-defined line segment SLa is shifted to the outside (+ Dx side in FIG. 8D) (S480 in FIG. 7). As a result, the region-defined line segment SLa is finally changed to the line segment SLan shown in FIG. 8 (D). Therefore, the object region corresponding to the original image CO1t is specified by using the final four region defining line segments L1a to L3a and SLan (S270 in FIG. 4). The object area corresponding to the original image CO1t is specified so as to include the character TXa without omission.

Similarly, when the region-defined line segment correction processing of FIG. 7 is performed on the four region-defined line segments L1b to L3b and SLb of the original image CO2t of FIG. 8 (D), the region-defined line segment SLb on the left side is performed. Outside of, there is an edge pixel Eb that constitutes the edge of the character TXb (FIG. 8 (A)). Therefore, the ratio ER of the edge pixels on the region-defined line segment SLb becomes larger than the threshold value TH3 (YES in S470 of FIG. 7). Therefore, the region-defined line segment SLb is shifted to the outside (-Dx side in FIG. 8D) (S480 in FIG. 7). As a result, the region-defined line segment SLb is finally changed to the line segment SLbn shown in FIG. 8 (D). Therefore, the object region corresponding to the original image CO2t is specified by using the final four region defining line segments L1b to L3b and SLbn (S270 in FIG. 4). The object area corresponding to the original image CO2t is specified so as to include the character TXb without omission.

An example will be described in which scan image data indicating another scan image is acquired. FIG. 9 is a second explanatory view of processing for another scanned image. FIG. 9A is a diagram showing a mode of reading when scan data indicating another scan image is generated. In this example, only a part of the document C3 to be read is located on the glass platen 940, and the other part of the document C3 is located outside the platen 940. When reading is performed by the reading unit 900 in such a state, as shown in FIG. 9B, scan data showing an image including a manuscript image CO3 showing only a part of the manuscript C3 is generated. The case where such scan data is used will be described below.

FIG. 9C shows the result of detecting the line segment of S200 of FIG. 4 with respect to the scan data showing the image of FIG. 9A. In FIG. 9C, the line segments L1c to L6c indicate the detected line segments.

In the example of FIG. 9B, the two outer edges OLm and OLn of the original image CO3 clearly appear in the image. Therefore, in the example of FIG. 9C, the line segments L1c and L2c corresponding to the outer edges OLm and OLn of the original image CO3 are detected. On the other hand, in the example of FIG. 9C, the outer edge on the right side of the original image CO3 and the portion corresponding to the outer edge on the upper side are not in close contact with the document base 940 at the time of scanning, so that the image of FIG. 9B is obtained. It does not appear clearly inside. In the image of FIG. 9B, dark portion images SHa and SHb appear in the portions corresponding to these outer edges. In the dark part images SHa and SHb, the boundary portion with the background is blurred and the color is gradually changed (gradation), so that the edge pixels constituting the clear line segment are not detected. Therefore, in the example of FIG. 9C, the line segments corresponding to the outer edges on the right side and the upper side of the original image CO3 are not detected.

Therefore, in this example, when determining the candidate line segment in S230 of FIG. 4, as shown in FIG. 9D, two candidates corresponding to the two outer edges OLm and OLn of the original image CO3. The line segments L1c and L2c are determined, but the candidate line segments corresponding to the upper outer edge of the original image CO3 are not determined. Further, as a candidate line segment corresponding to the outer edge on the right side of the original image CO3, a line segment L3c located inside the original image CO3 is determined.

FIG. 9D shows the circumscribed rectangle SQc determined in S240 of FIG. In the example of FIG. 9, in the area-defined line segment determination process (FIG. 6) of S250 of FIG. 4, the upper side SL1c of the circumscribed rectangle SQc is determined as the area-defined line segment corresponding to the upper outer edge of the original image CO3 (FIG. 6). NO, S350 in S320 of FIG. 6). The right side SL2c of the circumscribed rectangle SQc is determined as a region defining line segment corresponding to the outer edge on the right side of the original image CO3 (NO, S350 in S340 of FIG. 6).

In this way, even if the candidate line segment corresponding to a part of the outer edge of the original image CO3 is not determined, or even if the short line segment inside the original is determined as the candidate line segment, four lines are finally determined. It can be seen that the area-defined line segment of can be determined appropriately. FIG. 9E shows four region-defined line segments L1c, L2c, SL1c, and SL2c corresponding to the original image CO3.

According to the present embodiment described above, the processor 811 as the image acquisition unit acquires the scan image data indicating the scan image SI (S110 in FIG. 2). The processor 811 as a line segment specifying unit identifies one or more line segments (for example, L1 to L7 in FIG. 5A and L1a to L3a in FIG. 8B) in the scanned image SI (FIG. 4). S200). The processor 811 as a line segment determining unit has a plurality of area-defined line segments (for example, L1 to L4 in FIG. 5C, FIG. 8) which are line segments for determining an object area based on these line segments. L1a to L3a, SLa) of (D) are determined (S210 to S250 in FIG. 4, S410 to S440 in FIG. 7).

In the processor 811 as the condition determination unit, the first line segment (for example, L1 or L3 in FIG. 5C and L3a or SLa in FIG. 8D) of a plurality of area-defined line segments L1 to L3 is used. Whether or not the change condition is satisfied is determined based on the specific pixels (edge pixels in this embodiment) located in the specific region (S460 and S470 in FIG. 7). The specific region is a region determined by a plurality of region defining line segments (for example, L1 to L4 in FIG. 5C, L1a to L3a in FIG. 8D, SLa) (rectangular region in this embodiment). It is a region located outside the above and in a direction intersecting the first line segment. For example, in the example of FIG. 5D, when the first line segment is the line segment L1, the specific region is a region separated from the line segment L1 to the outside, and the region along the line segment L1m (edge pixel Eo1 is Area that exists). It can also be said that the first line segment L1 in this case is a line segment located at the outer edge of the region determined by the plurality of region-defined line segments L1 to L4. Further, in the example of FIG. 8D, when the first line segment is the line segment SLa, the specific region is a region along the first line segment SLa (the region where the edge pixel Ea exists). It can also be said that the first line segment SLa in this case is a line segment located at the outer edge of the region determined by the plurality of region defining line segments L1a to L3a and SLa. Further, the change condition is that the ratio ER of the specific pixel to the plurality of pixels located in the specific region is larger than the threshold value TH3 (S460 and S470 in FIG. 7).

When the change condition is satisfied (YES in S470 of FIG. 7), the processor 811 as the first defined line segment changing unit includes the line segments to be included in the four area defined line segments from the first line segment to the second line segment. Change to a line segment (S480 in FIG. 7). For example, in the example of FIG. 5D, the line segment to be included in the four region-defined line segments is changed from the first line segment L1 to the outer second line segment L1n. In the modified example, the second line segment may be L1m outside the first line segment L1. Further, in the example of FIG. 8D, the line segment to be included in the four region-defined line segments is changed from the first line segment SLa to the outer second line segment SLan. The processor 811 as the area determination unit determines an object area (for example, SA1 to SA3 in FIG. 3B) using a plurality of area-defined line segments (S270 in FIG. 4). When the change condition is satisfied, the object area is determined using a plurality of area-defined line segments including the second line segment (for example, L1n or SLan). If the change condition is not satisfied, the object area is determined using a plurality of area-defined line segments including the first line segment (for example, L3 or L3a). As a result, when the change condition is satisfied, the area of the object is determined using the second line segment located outside the area tentatively determined by the area specified line segment before the change including the first line segment. Will be done. If the change condition is not satisfied, the object area is determined using a plurality of area-defined line segments including the first line segment. As a result, since the object area can be determined using an appropriate area defining line segment, the object area (for example, SA1 to SA3) can be accurately determined in the scanned image SI.

In the scanned image SI, the boundary between the object and the background may not be a clear line segment. For example, in the examples of FIGS. 3 and 5, the outer edges OLa and OLb of the original images CO1 and CO2 have a jagged shape, not a straight line segment. In such a case, even if the line segment in the scanned image SI is detected by analyzing the scanned image data (S200 in FIG. 4), the line segment corresponding to the outer edge may not be detected (FIG. 5). (A)). According to the present embodiment, even in such a case, according to the present embodiment, the area-defined line segment is finally set to the line segment L1n located outside the edge pixels Eo1 and EO2 corresponding to the outer edge of the jagged edge. , L2n can be determined. Therefore, the object area SA1 can be accurately determined in the scanned image SI.

Further, in the example of FIG. 8, a part of the outer edges of the original images CO1t and CO2t is a dark part image SH, which is unclear. Even in such a case, the line segment corresponding to the outer edge may not be detected by analyzing the scanned image data (FIG. 8 (B)). Even in such a case, it is possible to specify an appropriate object area that includes the characters TXa and TXb without omission (FIG. 8 (D)).

Further, as described above, in this embodiment, it is determined that the change condition is satisfied when the ratio ER of the specific pixel in the specific area is larger than the threshold value TH3 (S470 in FIG. 7). As a result, it is possible to appropriately determine whether or not the change condition is satisfied based on the ratio of the specific pixels.

Further, as described above, in this embodiment, the specific pixel is an edge pixel constituting an edge. Therefore, it is possible to appropriately determine whether or not the change condition is satisfied based on the edge pixels located in the specific region.

Further, as described above, the processor 811 as the line segment determining unit selects one reference line segment (for example, L4) from one or more line segments L1 to L7 (FIG. 5 (A)) (FIG. 5). 4 S210), two area-defined line segments (L3, L4) along the first direction D1 and two area-defined line segments (L1m, L2m) along the second direction D2, based on the reference line segment. And (S220 to S250 in FIG. 4, S410 to S440 in FIG. 7). As a result, the rectangular object area can be determined accurately.

Further, in the above embodiment, the processor 811 as a line segment determining unit determines a reference line segment and an extrinsic rectangle SQ including at least a part of one or more line segments (S240 in FIG. 4), and the extrinsic rectangle. The first side along the first direction D1 of the SQ (for example, SL4 in FIG. 5B) and the first candidate line segment along the first direction D1 of one or more line segments (for example, FIG. 5 (B)). A region-defined line segment is determined from L4) of B) (S320 to S360 in FIG. 6). Further, the processor 811 has a second side (for example, SL1 in FIG. 5B) along the second direction D2 of the circumscribing rectangle SQ and a second candidate along the second direction D2 of one or more line segments. A region-defined line segment is determined from the line segment (for example, L1 in FIG. 5 (B)) (S320 to S360 in FIG. 6). As a result, an appropriate area-defined line segment can be determined from the sides of the circumscribing rectangle SQ and the line segments L1 to L7 in the scanned image SI. For example, as described with reference to FIGS. 8 and 9, when the first candidate line segment or the second candidate line segment is not detected, or the line segment is excessively short as the first candidate line segment or the second candidate line segment. Even when is detected, the circumscribing rectangle SQ can be used to determine the appropriate four region-defined line segments.

More specifically, the processor 811 is based on a comparison in which the first candidate line segment (for example, L4 in FIG. 5 (B)) is compared with the first side of the circumscribing rectangle SQ (for example, SL4 in FIG. 5 (B)). When the specific condition is satisfied (YES in S340 of FIG. 6), the first candidate line segment is determined as the region defining line segment (S360 of FIG. 6). When the first candidate line segment does not satisfy the specific condition (YES in S340 in FIG. 6), the first side of the circumscribing rectangle SQ is determined as the region defining line segment (S360 in FIG. 6). As a result, a more appropriate region-defined line segment can be determined based on the comparison between the first candidate line segment and the first side of the circumscribed rectangle SQ. Further, the specific condition is that the ratio LR of the length of the first candidate line segment to the length of the first side of the circumscribing rectangle SQ is larger than the threshold value TH2 (S340 in FIG. 6). As a result, when the first candidate line segment has a sufficient length, the first candidate line segment is determined as a region-defined line segment, and when the first candidate line segment does not have a sufficient length, the first candidate line segment is determined. The first side of the extrinsic rectangular SQ is determined as a region-defined line segment. Therefore, a more appropriate region defining line segment can be determined.

Further, according to the above embodiment, as shown in FIG. 5C, the processor 811 as the second defined line segment changing unit has a straight line including the region defined line segment L1 (first line segment L1). When intersecting with the area-defined line segments L3 and L4 (YES in S430 of FIG. 7), the position of the first direction D1 of the area-defined line segment L1 is changed to the position of the end P1t of the area-defined line segment L4. (S430 in FIG. 7). Then, in this case, the determination of the change condition by the processor 811 as the condition determination unit (S470 in FIG. 7) is executed after the position change by S430. As a result, the object area can be determined by using a more appropriate area defining line segment, so that the object area can be determined more accurately. For example, in the example of FIG. 5C, at the stage of the region-defined line segment determination process (FIG. 6) of S250 of FIG. 4, the line segment ILa that happens to exist inside the original image CO1 instead of the outer edge of the original image CO1. (FIG. 3 (A)) is determined as a region-defined line segment L1 (FIG. 5 (C)). Even in such a case, the processor 811 changes the area-defined line segment L1 (first line segment L1) into the area-defined line segment L1m based on the fact that the area-defined line segment L1 intersects the line segments L3 and L4. Can be changed to.

Further, according to the above embodiment, the target image data is image data generated by using the image sensor, more specifically, scan data generated by the reading unit 900. In the image represented by such target image data, the boundary between the document and the background may not be a clear line segment due to, for example, blurring of the image or the type of document used. Even in such a case, the object area corresponding to the original in the image can be accurately specified.

B. Modification example:
(1) In the above embodiment, the change condition for determining whether or not the line segment of interest should be shifted outward in the area-defined line segment correction process of FIG. 7 is that the ratio ER of the edge pixels of the line segment of interest is It is larger than the threshold TH3. The change condition is not limited to this, and various other conditions may be adopted.

For example, the ratio of non-background color pixels may be used instead of the ratio ER of edge pixels. A non-background color pixel is, for example, a pixel having a color value (RGB value) outside the RGB value range (also referred to as a background color range) indicating the background color (for example, the color of the surface covering the platen 940 of the cover 950). Is. The non-background color pixels are likely to be pixels that make up an object (eg, a manuscript image). Further, the edge pixel is considered to include a pixel that defines the outer edge of the object. The specific pixel for determining the change condition is preferably either a pixel constituting the object or a pixel defining the outer edge of the object.

Further, the specific region used for determining the change condition may be, for example, a region of a plurality of pixels in contact with the outside of the line segment of interest, instead of the region of a plurality of pixels on the line segment of interest. Further, the specific region may be a region of a plurality of pixels in contact with the inside of the line segment of interest. In general, the specific region is preferably a band-shaped region extending along the line segment of interest, and may be separated from the line segment of interest by a predetermined distance (for example, several pixels) to the outside or inside. Further, the width of the strip-shaped region in the lateral direction may be several pixels (for example, 2 pixels to 10 pixels) instead of 1 pixel.

Further, the index value used for determining the change condition may be the number of edge pixels or non-background pixels instead of the ratio ER of edge pixels. Further, the index value may be the number of portions in which edge pixels and non-background pixels are continuous over a plurality of pixels.

(2) In the above embodiment, the rectangular object area is specified, but the shape of the object area to be specified is not limited to the rectangle, and may be another shape such as a parallelogram or a triangle. In this case, the number of line segments and the angle of the line segments to be determined may be any number and angles according to the shape of the object area to be specified.

The object area identification process of FIG. 4 is an example and may be changed as appropriate. For example, in S230 of FIG. 4, it is assumed that four candidate line segments can be determined, and if the four candidate line segments cannot be determined, it may be processed as an error. In this case, for example, the determination of the circumscribed rectangle SQ of S240 and the region-defined line segment determination process of S250 may be omitted.

Further, S410 to S440 may be omitted in the area-defined line segment correction process of FIG. 7. In this case, for example, by setting the threshold TH2 to be compared with the ratio LR in S340 of FIG. 6 to a relatively large value, the line segment inside the document (for example, the line segment L1 of FIG. 5 (B)) is set. , L2) is lowered, and the probability that the side of the circumscribing rectangle SQ (for example, the sides SL1 and SL2 in FIG. 5B) is determined as the region-defined line segment is increased. Is also good.

(3) The specific condition for determining the attention candidate line segment determined in S340 of FIG. 6 as the region defining line segment is that the ratio LR of the length of the candidate line segment to the length of the attention side is from the threshold value TH2. It's a big thing. The specific conditions are not limited to this, and various other conditions may be adopted. For example, the specific condition may be that the ratio of the length of the candidate line segment of interest to the length of the reference line segment is larger than a predetermined threshold value, or the length of the candidate line segment of interest is larger than a predetermined threshold value. It may be.

(4) The filter used in the edge detection process of S120 in FIG. 2 may be an arbitrary filter for calculating the edge strength (sobel filter, prewit filter, etc.) instead of the Laplacian filter. When a filter that calculates the difference in a specific direction, such as a Sobel filter or a prewit filter, is used, the edge strength obtained by combining the edge strengths in a plurality of directions is used to detect the edge pixels. Good.

(5) The detection and labeling of the line segment of S200 in FIG. 4 may be various other processes instead of the processes described in the examples. For example, instead of the Hough transform, the line segment may be detected using the least squares method. Further, in the line segment detection process, in order to improve the accuracy of line segment detection, the scan image SI is divided into a plurality of blocks and the pixels constituting the line segment are specified for each block. Processing may be performed.

(6) The specified object area may be an area of any other object instead of the rectangular area which is the area of the original image. For example, when the target image contains a two-dimensional code, the processor 811 can identify the area of the two-dimensional code from the target image by executing the process of the above embodiment. The processor 811 may display the image of the extracted two-dimensional code on the display unit 840, for example. In this case, the target image data does not have to be the scanned image data, and may be, for example, image data generated by a predetermined image generation application.

(7) The method for determining the reference line segment in S210 of FIG. 4 may be various other methods instead of the method described in the examples. For example, the feature amount CV used may be only the distance between each line segment and the center of gravity CO, omitting the length of each line segment. Further, the feature amount CV may be only the length of each line segment, omitting the distance between each line segment and the center of gravity CO.

(8) The object area specified by the scanned image SI may be used for various processes. For example, in S180 of FIG. 2, the processor 811 may display the cropped object image on the display unit 840. Further, the processor 811 may execute a predetermined correction (for example, edge enhancement correction or color skip correction) on the object area in the scanned image SI or the area excluding the object area.

(9) The configuration of the reading unit 900 may be various other configurations instead of the configuration shown in FIG. For example, the reading unit 900 may include a conveying device that conveys a document to the image sensor 920. Then, the image sensor 920 may read the entire document by reading the document conveyed by the conveying device without moving.

(10) Further, the target image data may be captured image data generated by optically imaging the target object (for example, documents C1 and C2) using a digital camera instead of the scan data. .. A digital camera is a device that uses a two-dimensional image sensor to generate image data indicating an object (for example, documents C1 and C2).

(11) The image processing device that identifies the object area from the scan data may be various other devices instead of the control unit 810 of the multifunction device 800. For example, the reading unit 900 may be an external device that can be connected to an image processing device. As described above, the image processing device may be a type of device different from the so-called multifunction device (for example, a personal computer, a tablet computer, a smartphone, etc.). Further, the image processing device may be, for example, a server that receives scan data from a terminal device such as a scanner, a digital camera, or a smartphone and performs image processing. In this case, the server may be a so-called cloud server composed of a plurality of computers capable of communicating with each other.

(12) In each of the above embodiments, a part of the configuration realized by the hardware may be replaced with software, and conversely, a part or all of the configuration realized by the software may be replaced with the hardware. You may do so.

In addition, when a part or all of the functions of the present invention are realized by a computer program, the program is provided in a form stored in a computer-readable recording medium (for example, a non-temporary recording medium). be able to. The program may be used while being stored on the same or different recording medium (computer-readable recording medium) as it was provided. The "computer-readable recording medium" is not limited to a portable recording medium such as a memory card or a CD-ROM, but is connected to an internal storage device in the computer such as various ROMs or a computer such as a hard disk drive. It may also include an external storage device.

Although the present invention has been described above based on Examples and Modifications, the above-described embodiments of the invention are for facilitating the understanding of the present invention and do not limit the present invention. The present invention can be modified and improved without departing from the spirit thereof, and the present invention includes equivalents thereof.

800 ... Composite machine, 810 ... Control unit, 811 ... Processor, 812 ... Volatile storage device, 813 ... Non-volatile storage device, 814 ... Computer program, 815 ... Storage device, 840 ... Display unit, 850 ... Operation unit, 860 ... Printing unit, 890 ... Housing, 900 ... Reading unit, 920 ... Image sensor, 930 ... Moving device, 940 ... Document stand, 950 ... Cover, C1, C2, C3 ... Original, CO1, CO1t, CO2, CO2t, CO3 ... Original image, CV ... feature quantity, D1 ... first direction, D2 ... second direction, EI ... edge image, Ei, Eo, Eo1, Eo2 ... edge pixel, SA1, SA2, SA3 ... object area, SI ... scanned image, SQ, SQa, SQb, SQc ... External rectangle, TH1, TH2, TH3 ... Threshold, TI1, TI2 ... Manuscript image

Claims (12)

It is an image processing device
An image acquisition unit that acquires target image data indicating the target image,
A line segment specifying part that specifies one or more line segments in the target image, and
A line segment determination unit for determining a plurality of area-defined line segments, which are line segments for determining the area of the object in the target image based on the one or more line segments.
A condition determination unit that determines whether or not the first line segment of the plurality of area-defined line segments satisfies the change condition based on the specific pixel located in the specific area, and the specific area is the plurality of line segments. A region outside the region determined by the region defining line segment of the above and located in the direction in which the first line segment intersects, and the specific pixel defines the pixels constituting the object and the outer edge of the object. The condition determination unit, which is one of the pixels to be
A first defined line segment changing unit that changes a line segment to be included in the plurality of area defined line segments from the first line segment to the second line segment when the change condition is satisfied, and is the first line. The line segment is a line segment located at the outer edge of the region determined by the plurality of region-defined line segments, and the second line segment is determined by a plurality of region-defined line segments including the first line segment. The first specified line segment change part, which is a line segment located outside the area,
A region determination unit that determines the area of the object using the plurality of region-defined line segments, and when the change condition is satisfied, the plurality of region-defined line segments including the second line segment. The area of the object is determined using the above, and if the change condition is not satisfied, the area of the object is determined using the plurality of area-defined line segments including the first line segment. Department and
An image processing device comprising. The image processing apparatus according to claim 1.
The condition determination unit is an image processing device that determines that the change condition is satisfied when the ratio of the specific pixel to the specific area is larger than the threshold value. The image processing apparatus according to claim 1 or 2.
An image processing device in which the specific pixel is an edge pixel constituting an edge. The image processing apparatus according to any one of claims 1 to 3.
The line segment determination unit
Select one reference line segment from the one or more line segments,
Based on the reference line segment, two first first region-defined line segments along a first direction parallel to the reference line segment and two second along a second direction perpendicular to the reference line segment. An image processing device that determines the area-defined line segment of the above. The image processing apparatus according to claim 4.
The line segment determination unit
Based on the reference line segment, the reference line segment and a rectangle including at least a part of the one or more line segments are determined.
The first area-defined line segment is determined from the first side of the rectangle along the first direction and the first candidate line segment along the first direction among the one or more line segments. And
The second area-defined line segment is determined from the second side of the rectangle along the second direction and the second candidate line segment along the second direction of the one or more line segments. Image processing equipment. The image processing apparatus according to claim 5.
The line segment determination unit
When the first candidate line segment satisfies a specific condition based on the comparison with the first side of the rectangle, the first candidate line segment is determined as the first region defining line segment.
An image processing device that determines the first side of the rectangle as the first region-defined line segment when the first candidate line segment does not satisfy the specific condition. The image processing apparatus according to claim 6.
The specific condition is that the ratio of the length of the first candidate line segment to the length of the first side is larger than the threshold value. The image processing apparatus according to any one of claims 4 to 7.
The line segment determination unit selects the reference line segment from the one or more line segments based on a predetermined selection criterion.
The predetermined selection criteria include the distance between the reference position determined based on the one or more line segments and each of the one or more line segments, the length of each of the one or more line segments, and the like. An image processing device that is a standard based on. The image processing apparatus according to any one of claims 4 to 8.
The first line segment is one of the two second region-defined line segments.
The image processing device further
When a straight line including the first line segment intersects the first area-defined line segment, the position of the first line segment in the first direction is set to one end of the first area-defined line segment. The second specified line segment changing part for changing to the position of the first direction is provided.
One end of the first region-defined line segment is the first of both ends of the first region-defined line segment with the other line segment of the two second region-defined line segments. The end with a long distance in the direction
When the straight line including the first line segment intersects with the first area defined line segment, the determination of the change condition by the condition determination unit is executed after the change by the second specified line segment change unit. Image processing equipment. The image processing apparatus according to any one of claims 1 to 9.
The target image data is an image processing device that is image data generated by using an image sensor. It is an image processing method
The image acquisition process for acquiring the target image data indicating the target image,
A line segment specifying process for specifying one or more line segments in the target image, and
A line segment determination process for determining a plurality of area-defined line segments, which are line segments for determining an object area in the target image based on the one or more line segments.
This is a condition determination process for determining whether or not the first line segment of the plurality of area-defined line segments satisfies the change condition based on the specific pixel located in the specific area, and the specific area is the plurality of line segments. A region outside the region determined by the region defining line segment of the above and located in the direction in which the first line segment intersects, and the specific pixel defines the pixels constituting the object and the outer edge of the object. The condition determination process, which is one of the pixels to be used,
The first line segment change process for changing the line segment to be included in the plurality of area specified line segments from the first line segment to the second line segment when the change condition is satisfied. The line segment is a line segment located at the outer edge of the region determined by the plurality of region-defined line segments, and the second line segment is determined by a plurality of region-defined line segments including the first line segment. The first specified line segment change process, which is a line segment located outside the area,
In the area determination process for determining the area of the object using the plurality of area-defined line segments, when the change condition is satisfied, the plurality of area-defined line segments including the second line segment are used. The area of the object is determined using the above, and if the change condition is not satisfied, the area of the object is determined using the plurality of area-defined line segments including the first line segment. The process and
An image processing method comprising. It ’s a computer program,
An image acquisition function that acquires target image data indicating the target image, and
A line segment identification function that identifies one or more line segments in the target image, and
A line segment determination function for determining a plurality of area-defined line segments, which are line segments for determining an object area in the target image based on the one or more line segments, and a line segment determination function.
It is a condition determination function that determines whether or not the first line segment of the plurality of area-defined line segments satisfies the change condition based on the specific pixel located in the specific area, and the specific area is the plurality of line segments. A region outside the region determined by the region defining line segment of the above and located in the direction in which the first line segment intersects, and the specific pixel defines the pixels constituting the object and the outer edge of the object. The condition determination function, which is one of the pixels to be
The first line segment changing function for changing the line segment to be included in the plurality of area defined line segments from the first line segment to the second line segment when the changing condition is satisfied. The line segment is a line segment located at the outer edge of the region determined by the plurality of region-defined line segments, and the second line segment is determined by a plurality of region-defined line segments including the first line segment. The first specified line segment change function, which is a line segment located outside the area,
It is an area determination function that determines the area of the object by using the plurality of area-defined line segments, and when the change condition is satisfied, the plurality of area-defined line segments including the second line segment. The area of the object is determined using the above, and if the change condition is not satisfied, the area of the object is determined using the plurality of area-defined line segments including the first line segment. Function and
A computer program that can be realized on a computer.

Images