WO2013094231A1

WO2013094231A1 - Information terminal device, captured image processing system, method, and recording medium recording program

Info

Publication number: WO2013094231A1
Application number: PCT/JP2012/056327
Authority: WO
Inventors: 浩明金田; 和久中林
Original assignee: ナカバヤシ株式会社
Priority date: 2011-12-20
Filing date: 2012-03-12
Publication date: 2013-06-27
Also published as: JP2013131801A

Abstract

The purpose of the present invention is to provide an information terminal device capable of trimming and storing a prescribed region of a captured image even away from home, a captured image processing system, method, and computer-readable recording medium recording a program. This portable information terminal device is provided with an imaging unit and a control unit, and performs trimming processing on a prescribed region of a captured image acquired from the imaging unit. The prescribed region of an imaging subject is surrounded by a line having a prescribed color, and a control unit acquires from the imaging unit a captured image in which the prescribed region of the imaging subject is captured (S1). From the captured image, an extracted image is generated by extracting only a prescribed color which is the trimming subject (S2). In the extracted image, multiple contour lines are extracted which correspond to the prescribed region of the captured image, and the trimming region is set to be the region with the largest area surrounded by said contour lines (S4). The region of the captured image indicated by the trimming region is then trimmed (S12).

Description

Information terminal device, captured image processing system, method, and recording medium recording program

The present invention relates to a captured image processing method, and more specifically, recorded an information terminal device, a captured image processing system, a method, and a program capable of trimming and storing a predetermined area of a captured image even when away from home. The present invention relates to a computer-readable recording medium.

When capturing newspaper articles or magazine clippings as image data, a conventional scanner has been used. However, because scanner installation locations are limited to specific locations such as offices and convenience stores, for example, even if you find interesting articles on the go, you can easily convert them to image data. Could not save.

On the other hand, with recent development of information processing technology, for example, there is a technology for storing image data captured by an imaging device included in a mobile terminal device such as a mobile phone or a smartphone via a network (see, for example, Patent Document 1). . In the technique described in Patent Document 1, digital image data collected by a portable information device such as a PDA with a camera or a portable personal computer is transmitted to a server via a network, and the received digital image is received on the server side. Image processing such as adjustment of brightness, color, and size is performed on the data.

JP 2002-41502 A

Although the technique described in Patent Document 1 has the convenience that image data collected on the go can be saved via a network, the process of trimming only the necessary portions after converting to image data is very troublesome. It was. In addition, even when trying to trim image data, there is a problem that the operability is insufficient with a mobile phone held by a user who is out, and even a compact information device such as a mobile phone can be easily used. Therefore, development of a simple method capable of trimming image data has been desired.

Also, when a newspaper article or magazine clipping is imaged using the camera function of the mobile terminal device, there is a problem that the captured image is distorted (perspective). When a general scanner is used for the purpose of document storage, the plane containing the imaging object (document page) is fixed at the position of the glass surface of the flatbed. Does not occur. However, when the camera function of the mobile terminal device is used, the imaging unit captures an image from a direction that does not coincide with the normal direction of the plane including the imaging target. There is a problem that the captured image is distorted and the content of the article is difficult to read even if the user browses to confirm the content of the captured image.

The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an information terminal device, a captured image processing system, and a method capable of trimming and storing a predetermined region of a captured image even when away from home. Another object of the present invention is to provide a computer-readable recording medium on which a program is recorded.

Furthermore, an object of the present invention is to provide an information terminal device capable of correcting a distortion and storing a picked-up image when picking up an object to be picked up by a portable information terminal device having an image pickup unit, and picked-up image processing It is an object to provide a computer-readable recording medium on which a system, a method, and a program are recorded.

In order to achieve the above object, an information terminal device according to the present invention is an information terminal device that includes an imaging unit and a control unit, and performs a trimming process on a predetermined area of a captured image acquired from the imaging unit. A predetermined region of the object is surrounded by a line having a predetermined color, and the control unit acquires a captured image obtained by capturing the predetermined region of the imaging object from the imaging unit, and the imaging An extracted image obtained by extracting only the predetermined color to be trimmed is generated from the image, and a plurality of contour lines corresponding to the predetermined region of the captured image are extracted from the extracted image and surrounded by the contour lines This is an information terminal device that determines a region having the largest area as a trimming region and trims a region specified by the trimming region of the captured image.

Further, the information terminal device according to the present invention is the information terminal device according to the present invention, wherein the control unit further determines a trimming region, and then extends a plurality of line segments constituting the trimming region. A plurality of intersection candidates formed by two straight lines are obtained from a combination of straight lines, and among the obtained plurality of intersection candidates, the four intersection candidates whose coordinates are located on the outermost side are first trapezoidal correction. A trapezoidal distortion index defined by the four first vertex coordinates is calculated from the four first vertex coordinates, and a trapezoidally corrected second key is calculated based on the calculated index. 2 is determined, and the first vertex coordinate and the second vertex coordinate are used to perform perspective projection transformation of the captured image to generate the captured image in front view, and the first vertex Coordinates and the second Performing perspective projection transformation of the trimming area using point coordinates, generating the trimming area in front view, and trimming the area specified by the trimming area in front view of the captured image in front view preferable.

The captured image processing system according to the present invention includes an information terminal device having an imaging unit and a control unit, and a writing instrument, and the information terminal device performs a trimming process on a predetermined region of the captured image acquired from the imaging unit. In the captured image processing system, a predetermined region of the imaging target is surrounded by a line having a predetermined color using the writing instrument, and the control unit is configured to transmit the imaging target from the imaging unit. A captured image obtained by capturing the predetermined area is acquired, and an extracted image obtained by extracting only the predetermined color to be trimmed is generated from the captured image. In the extracted image, the predetermined area of the captured image is extracted. A plurality of corresponding contour lines are extracted, a region having the maximum area surrounded by the contour lines is determined as a trimming region, and a region specified by the trimming region of the captured image is trimmed. To ring, a captured image processing system.

Further, in the captured image processing system according to the present invention, in the captured image processing system according to the present invention, the control unit further extends a plurality of line segments constituting the trimming area after determining the trimming area. A plurality of intersection candidates formed by two straight lines are obtained from a combination of a plurality of straight lines, and among the obtained plurality of intersection candidates, the four intersection candidates whose coordinates are located on the outermost side are obtained as trapezoidal correction firsts. 1 is determined as a vertex coordinate, a trapezoidal distortion index defined by the four first vertex coordinates is calculated from the four first vertex coordinates, and after trapezoidal correction based on the calculated index The second vertex coordinates are determined, and the first vertex coordinates and the second vertex coordinates are used to perform perspective projection conversion of the captured image to generate the captured image in front view, and the first Vertex coordinates And the perspective projection conversion of the trimming area using the second vertex coordinates to generate the trimming area for front view, and the area specified by the trimming area for front view of the captured image in front view Is preferably trimmed.

A captured image processing method according to the present invention is a method for trimming a predetermined region of a captured image acquired from the image capturing unit in an information terminal device including an image capturing unit and a control unit. A step in which a predetermined region is surrounded by a line having a predetermined color, and the control unit acquires a captured image obtained by capturing the predetermined region of the imaging object from the imaging unit; A step of generating an extracted image obtained by extracting only the predetermined color to be trimmed, and extracting a plurality of contour lines corresponding to the predetermined region of the captured image in the extracted image; A captured image including a step of determining a region having the maximum enclosed area as a trimming region, and a step of trimming a region specified by the trimming region of the captured image It is a management method.

Further, the captured image processing method according to the present invention is the captured image processing method according to the present invention, wherein the control unit determines a trimming region and then extends a plurality of line segments constituting the trimming region. A plurality of intersection candidates formed by two straight lines are obtained from a combination of straight lines, and among the obtained plurality of intersection candidates, the four intersection candidates whose coordinates are located on the outermost side are first trapezoidal correction. A step of determining a vertex coordinate, and calculating a trapezoidal distortion index defined by the four first vertex coordinates from the four first vertex coordinates, and after correcting the trapezoid based on the calculated index Determining the second vertex coordinates, and performing perspective projection transformation of the captured image using the first vertex coordinates and the second vertex coordinates to generate the captured image in front view; Performing a perspective projection transformation of the trimming area using the first vertex coordinates and the second vertex coordinates to generate the trimming area in a front view; and a front view view of the captured image. It is preferable that the method further includes a step of trimming an area designated by the trimming area.

In addition, a computer-readable recording medium that records a captured image processing program according to the present invention performs a trimming process on a predetermined area of a captured image acquired from the imaging unit in an information terminal device including an imaging unit and a control unit. A computer-readable recording medium on which a program is recorded, wherein a predetermined area of the imaging object is surrounded by a line having a predetermined color, and the control unit is configured to transmit the imaging object from the imaging unit. Obtaining a captured image obtained by imaging the predetermined area; generating an extracted image obtained by extracting only the predetermined color to be trimmed from the captured image; and A plurality of contour lines corresponding to a predetermined region are extracted, and a region having the maximum area surrounded by the contour lines is determined as a trimming region. And-up, of the captured image, a computer-readable recording medium which records a captured image processing program for executing the steps of trimming the region specified, to a computer in the trimming area.

The computer-readable recording medium recording the captured image processing program according to the present invention is a computer-readable recording medium recording the captured image processing program according to the present invention, wherein the control unit further includes the trimming area. Is determined, a plurality of intersection candidates formed by the two straight lines are obtained from a combination of a plurality of straight lines obtained by extending a plurality of line segments constituting the trimming region, and coordinates among the obtained intersection candidates are determined. Are determined as four first vertex coordinates from the four first vertex coordinates, and the four intersection candidate candidates that are positioned on the outermost side are determined as first vertex coordinates for trapezoidal correction. Calculating a trapezoidal distortion index, determining a second vertex coordinate after trapezoid correction based on the calculated index; and the first vertex coordinate And using the second vertex coordinates to perform perspective projection transformation of the captured image to generate the captured image in front view, using the first vertex coordinates and the second vertex coordinates, Performing a perspective projection conversion of the trimming area to generate the trimming area in front view, and trimming an area specified by the trimming area in front view of the captured image in front view; It is preferable to execute.

According to the present invention, it is possible to trim and save a predetermined area of a captured image even on the go. In addition, when the imaging target is imaged by a portable information terminal device having an imaging unit, the captured image can be stored after correcting the distortion. There is no distortion in newspaper articles and magazine clippings that have been captured and converted into image information, and the user can accurately read the contents of the articles.

It is a schematic block diagram of the portable terminal device which concerns on one Embodiment of this invention, (a) is a front view, (b) is a back view. It is a block diagram of a portable terminal device. It is a flowchart which shows the process order of the image processing method which the portable terminal device which concerns on one Embodiment of this invention performs. It is a schematic diagram for demonstrating the perspective of an imaging target object. It is a schematic diagram for demonstrating the determination method of the vertex coordinate for trapezoid correction | amendment. It is a schematic diagram for demonstrating the determination method of the vertex coordinate after trapezoid correction | amendment. It is an example of a marker pen used for specifying a region R to be captured.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and drawings, the same reference numerals indicate the same or similar components, and thus the description of the same or similar components is omitted.

FIG. 1 is a schematic configuration diagram of a mobile terminal device according to an embodiment of the present invention. (A) is a front view, (b) is a back view. FIG. 2 is a block diagram of the mobile terminal device.

As shown in FIGS. 1 and 2, the mobile terminal device 1 includes a device main body 2, an imaging unit 3, a display unit 4, a touch panel 5, a storage unit 6, and a control unit 7. In addition, the mobile terminal device 1 includes an antenna for wireless communication, a microphone and a speaker for voice calls, and the like (none of which are shown). Such a portable terminal device 1 is not particularly limited, and examples thereof include a smartphone, a mobile phone, and a PDA.

As the device body 2, a known smartphone body can be used, which includes a power switch and various buttons. The imaging unit 3 is a known configuration that images a subject image incident through a lens, and can capture a newspaper article or magazine clipping to be captured. Such an imaging unit 3 includes, for example, an imaging element such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor) that outputs an analog electrical signal, photoelectrically converts a subject image incident through a lens, and performs imaging. A device that converts an analog electric signal from the element into a digital electric signal and outputs image data can be used.

As the display unit 4, a known display device such as a liquid crystal display or an organic EL display can be used, and image information or the like can be displayed on the display screen. The touch panel 5 is a known touch panel that can recognize the touch position when the user touches the surface, and is disposed on the display unit 4. As the touch panel 5, a known type such as a resistive film type, a surface acoustic wave type, an electromagnetic induction type, a capacitance type, or the like can be used. Further, various inputs can be made by the user touching the touch panel 5. For example, by touching the touch panel 5, an instruction for imaging by the imaging unit 3 and display on the display unit 4 can be input.

The storage unit 6 includes a storage medium such as a known hard disk or semiconductor memory that stores programs and data for information processing. The storage unit 6 can store image information captured by the imaging unit 3 (hereinafter also simply referred to as “captured image”). The captured image is stored in various image formats such as jpeg and gif, for example.

The control unit 7 includes a processor such as a known CPU that performs information processing based on programs and data. The control unit 7 can control each of the above constituent elements by executing a program stored in the storage unit 6.

In the following description, it will be described as an image processing method performed by the mobile terminal device 1 unless otherwise specified. The processing performed by the mobile terminal device 1 actually means processing performed by the control unit 7 of the mobile terminal device 1. The control unit 7 temporarily stores necessary data (such as intermediate data during processing) using the storage unit 6 as a work area, and appropriately records data to be stored for a long period of time such as a calculation result in the storage unit 6. In addition, the mobile terminal device 1 generates, for example, a program used for performing the processing of steps S1 to S15 described below in an execution format (for example, generated by being converted from a programming language such as C language by a compiler). The mobile terminal device 1 is recorded in advance in the storage unit 6 and performs processing using the program recorded in the storage unit 6.

FIG. 3 is a flowchart showing the processing order of the image processing method performed by the mobile terminal device according to the embodiment of the present invention. Hereinafter, the processing order of the image processing method according to the embodiment of the present invention will be described in detail based on the flowchart shown in FIG.

In the present embodiment, as shown in FIG. 4, a case will be described in which a captured image has a perspective (perspective, hereinafter simply referred to as a perspective) in the Y-axis direction in the figure. In the following description, trapezoidal correction refers to a captured image in which the normal direction of the plane including the object to be imaged and the image capturing direction do not coincide with each other, and the image capturing direction coincides with the normal direction. This means that the image is converted to a front-view image without any.

In step S1, a predetermined area of a newspaper article to be captured is imaged. The user marks a predetermined region Rn of the newspaper article N with a predetermined color (for example, red) using the marker pen M, and the user uses the imaging unit 3 to mark the predetermined region surrounded by the marker. An area including Rn is imaged. The portable terminal device 1 records a captured image. In the present embodiment, as shown in FIG. 4, the captured image is parsed in the Y-axis direction in the figure. The capture target region Rn is also parsed in the Y-axis direction in the figure. As the marker pen M, a marker pen M having a V-shaped or U-shaped shape, for example, in which a colored portion at the tip is divided into two protrusions is used. When marking is performed using such a marker pen M, the capture target region Rn is a region surrounded by a double line pattern.

After capturing the captured image with the perspective, in step S2, an image (extracted image) is generated by extracting only the color to be trimmed from the captured image. The extracted image is image data used for recognizing a trimming region Rt described later. Since the vertical and horizontal image sizes of the extracted image match the vertical and horizontal image sizes of the captured image, the extracted image functions as a so-called “layer” for the captured image. The generation of the extracted image is performed by, for example, a known processing method that extracts using a threshold value. Since the region Rn to be captured by the newspaper article N is marked by being surrounded by a predetermined color, when only a predetermined color in the captured image is extracted, the largest region surrounded by the extracted color in the extracted image is This becomes a trimming region Rt described later. Here, since the captured image is parsed, the extracted image is also parsed. Further, since the region Rn to be captured is surrounded by a double line pattern, the region surrounded by the extracted color in the extracted image is also surrounded by the double line pattern. In the subsequent processing, the layer of the extracted image is converted into coordinates, and the extracted image is specified using coordinates instead of the layer.

In step S3, the extracted double line image is converted into a single line. That is, the width of the extracted double line is expanded and contracted to be converted into a single line (single line). The double line is detected by a known processing method that detects a boundary using a threshold value, for example.

In step S4, coordinates for trimming are detected. Since the extracted image is image data of only a predetermined color (red), it is first binarized (monochrome data). Next, all contour lines are extracted from the binarized image data, and the extracted contour lines are linearly approximated. Further, a region where the area of the region surrounded by the contour line is maximized is determined by combining the linearized contour lines, and the region where the area is maximized is set as a trimming region Rt. The method of contour extraction and straight line approximation is performed by a known processing method. Since the coordinates of the extracted image are parsed, the shape of the trimming region Rt is also parsed.

In step S5, it is determined whether or not the area of the trimming region Rt is larger than a predetermined area. If it is larger, the process of step S6 is performed, and if smaller, the process is terminated.

In step S6, the vertex coordinates for keystone correction are calculated. The vertex coordinates for trapezoid correction are obtained from the vertex coordinates constituting the trimming region Rt obtained in step S4. FIG. 5 is a schematic diagram for explaining a method of determining vertex coordinates for trapezoid correction. As shown in FIG. 5, the trimming region Rt is composed of a plurality of line segments that connect the vertices r1 to r10 in order. An angle formed by two line segments among the plurality of line segments is obtained. If the obtained angle is equal to or greater than a predetermined angle (for example, 70 degrees), the coordinates of the intersection formed by two straight lines obtained by extending the two line segments are obtained and set as a candidate for the intersection. If the obtained coordinates of the intersection candidate protrude from the area of the captured image, the intersection candidate is excluded. Processing for obtaining such intersection candidate is performed for all combinations (brute force) in which the combinations of the two line segments are changed. Then, of the obtained plurality of intersection candidates, the four intersections whose coordinates are located on the outermost side are designated as vertex coordinates P1 to P4 for trapezoid correction.

In step S7, the user finely adjusts the coordinate positions of the vertex coordinates P1 to P4 on the display screen. The mobile terminal device 1 displays the trapezoidal correction vertex coordinates P1 to P4 obtained in step S6, the coordinates indicating the trimming area Rt, and the captured image indicating the capture target area Rn on the display unit 4 in an overlapping manner. The coordinate positions of the vertex coordinates P1 to P4 finely adjusted based on the user input from the touch panel 5 are recorded.

In step S8, the vertex coordinates after the keystone correction are calculated. The vertex coordinates after the trapezoid correction are obtained from the trapezoid correction vertex coordinates P1 to P4 determined in step S6 or S7. FIG. 6 is a schematic diagram for explaining a method of determining vertex coordinates after trapezoid correction. First, the angles woven by two opposite sides (two straight lines) of the trapezoid defined by the vertex coordinates P1 to P4 are obtained, and the two sides having the smaller inclination are selected. This is a process for discriminating the direction of the perspective. In the present embodiment, since the perspective is attached in the Y-axis direction shown in FIG. Is selected. Next, with respect to the longer side (side P2-P3) of these two sides, the inclinations α1 and α2 with the adjacent sides are obtained. Next, an average value of the two obtained inclinations α1 and α2 is obtained, and this is set as a trapezoidal inclination α _av . The obtained trapezoidal inclination α _av becomes an index representing the degree of parsing of the captured image.

Next, the aspect ratio of the image after the trapezoid correction is acquired from the obtained trapezoidal inclination α _av and the table for correspondence of the aspect ratio for trapezoid correction prepared in advance. The aspect ratio correspondence table is a correspondence table between the trapezoid inclination α _av and the aspect ratio of the image after the trapezoid correction, and is a table prepared in advance by actual measurement using the imaging unit 3 of the mobile terminal device 1. . Table 1 shows an example of the aspect ratio correspondence table. In Table 1, the constant k is the lateral magnification of the short side (P1-P4) when the long side of the trapezoid (side P2-P3 in this embodiment) is 1.

From the coordinate positions of the vertex coordinates P1 to P4 and the value of the lateral magnification k obtained from the correspondence table, the coordinate positions of the four vertices P1x to P4x after trapezoid correction are calculated and recorded. In the present embodiment, the sides P2-P3 are the long sides of the trapezoid and the sides P1-P4 are the short sides, so the coordinate positions of the points P1, P4 of the sides P1-P4, which are the short sides, and the horizontal side magnification Based on k, coordinate positions P1x and P4x after trapezoidal correction are calculated.

In step S9, the trapezoidal correction of the captured image is performed, and the captured image of the front view without distortion is created from the captured image distorted with the perspective. Since the vertex coordinates P1 to P4 before the trapezoid correction and the vertex coordinates P1x to P4x after the trapezoid correction are already obtained on the extracted image, the vertex coordinates P1 to P4 and P1x to P4x before and after the correction are corrected. The perspective projection conversion is performed using the information, and the trapezoidal correction of the captured image with the perspective is performed. The captured image after the keystone correction is a captured image of front view without perspective. Since perspective projection conversion is known, detailed description thereof is omitted in this specification.

In step S10, the trapezoidal correction of the trimming area is performed, and a front-view trimming area without distortion is created from the trimming area Rt that is distorted with perspective. Perspective projection conversion is performed using the information of the vertex coordinates P1 to P4 and P1x to P4x before and after correction, and the keystone correction of the coordinates of the trimmed region Rt with the perspective is performed. The trimming area after the keystone correction is a trimming area in front view without perspective.

In step S11, the user finely adjusts the coordinate position of the vertex coordinates of the trimming area on the display screen. The mobile terminal device 1 displays on the display unit 4 the coordinates indicating the front-view trimming area obtained in step S10 and the captured image obtained in step S9 on the display unit 4, and the user from the touch panel 5 Based on the input, the coordinate position of the finely adjusted vertex coordinates of the trimming area in front view is recorded.

In step S12, the captured image is trimmed. First, a mask image for trimming processing is created from the coordinates of the front-view trimming region created in step S10 or step S11. Next, trimming processing of the captured image of the front view created in step S9 is performed using the created mask image. Since the coordinate position of the vertex coordinates and the captured image are both front-view images, the captured image after trimming is a front-view image without perspective. A known method is used for image trimming using a mask image. By the trimming process, a region outside the trimming region in the captured image is filled with, for example, a predetermined color (for example, white).

In step S13, the image quality of the trimmed captured image is adjusted. For example, the white balance (color temperature) of the trimmed image is adjusted according to the shooting environment of the captured image, the histogram is made uniform, and gamma correction is performed.

In step S14, it is determined whether or not there is a double line in the captured image. A double line detection process is performed on the captured image with the perspective captured in step S1, and if a double line exists, the process of step S15 is performed. If a double line does not exist, the double line detection process is performed. The process ends. The double line is detected by a known processing method that detects a boundary using a threshold value, for example.

In step S15, the trimmed image whose image quality has been adjusted is stored in the storage unit 6, and the process is terminated.

In this way, if the user marks the area of the article to be captured with a predetermined color using the marker pen M, the mobile terminal device 1 trims the area marked with the image and images the image. It can be saved as information. Since it is a simple method of enclosing and marking a region to be trimmed of an article using the marker pen M, it is possible to easily trim image data even when away from home. Further, since the stored trimmed image is a front-view image without perspective, there is no distortion in a newspaper article or magazine clipped as image information, and the user can accurately read the contents of the article.

As mentioned above, although this invention was demonstrated by specific embodiment, this invention is not limited to above-described embodiment.

In the above embodiment, the mobile terminal device 1 applies both the trimming process of the captured image and the trapezoidal correction. However, when the user determines that the captured image is not distorted and it is not necessary to correct the trapezoidal image. The mobile terminal device 1 may apply only the trimming process of the captured image. That is, the trapezoidal correction of the captured image is an arbitrary process according to the user's desire.

In the above embodiment, all the contour lines are extracted and the trimming area is determined in step S4. However, the method for obtaining the trimming area is not limited to this. For example, the user uses the marker pen M. A region surrounded by freehand may be used as a trimming region as it is.

In the above-described embodiment, the region R to be captured is marked with a double line pattern using the marker pen M in which the colored portion at the tip is divided into two protrusions. The pattern of the line surrounding is not limited to this. For example, as shown in FIG. 7, a marker pen in which the colored portion at the tip is divided into three protrusions may be used to mark the region R to be captured in a triple line pattern. Furthermore, the region R to be captured may be surrounded by a pattern in which the line widths of the three lines are different using marker pens having different widths of the three protrusions. In this way, by using the line pattern surrounding the region R to be captured as a specific pattern and detecting the presence of the specific pattern in the captured image, it is possible to eliminate the use other than the specific marker pen. Note that the process of converting a double line image into a single line in step S3 and the process of determining the presence of a double line in step S14 are arbitrary processes. For example, when the line surrounding the region R to be captured is a single line These processes can be omitted. In this case, the captured image can be trimmed using the region surrounded by the single line as a trimming region.

In the above embodiment, a normal dye-based or pigment-based ink is used for the marker pen, but instead, an infrared ink that reflects or absorbs infrared light may be used. If the area R to be captured is marked by using infrared reflective ink, the infrared light is reflected or absorbed at the mark portion, and this is detected on the mobile terminal device 1 side to identify the area Rn to be captured. May be.

In the above embodiment, in step S7, the mobile terminal device 1 records the coordinate positions of the finely adjusted vertex coordinates P1 to P4, and in step S11, the mobile terminal device 1 detects the trimming area of the front view. The coordinate positions of the finely adjusted vertex coordinates are recorded, but the process of finely adjusting these coordinate positions or the trimming area is an arbitrary process, and the process may be omitted.

Moreover, in the said embodiment, although it determines whether the double line existed in the captured image in step S14, the result of having detected the double line with respect to the captured image in step S3, for example, It may be stored as a Boolean flag, and in step S14, it may be determined whether or not a double line exists in the captured image based on the stored information of the flag.

In the above embodiment, white balance adjustment, histogram equalization, and gamma correction are illustrated as examples of adjusting the image quality of a trimmed captured image. However, the present invention is not limited to these. Any method of adjusting the image quality suitable for browsing the trimmed image on the mobile terminal device 1 can be applied as appropriate.

In the above embodiment, after the trimming process, the area outside the trimming area in the captured image is filled with a predetermined color. However, this area may be made transparent in the form of a transparent GIF, for example.

DESCRIPTION OF SYMBOLS 1 Mobile terminal device 2 Apparatus main body 3 Image pick-up part 4 Display part 5 Touch panel 6 Storage part 7 Control part N Newspaper article M Marker pen Rn The area | region to capture Rt Trimming area | region

Claims

An information terminal device comprising an imaging unit and a control unit, and trimming a predetermined area of a captured image acquired from the imaging unit,
A predetermined area of the imaging object is surrounded by a line having a predetermined color,
The control unit is
From the imaging unit, obtain a captured image obtained by imaging the predetermined area of the imaging object,
Generating an extracted image obtained by extracting only the predetermined color to be trimmed from the captured image;
In the extracted image, a plurality of contour lines corresponding to the predetermined region of the captured image are extracted, and a region having the largest area surrounded by the contour line is determined as a trimming region,
An information terminal device for trimming an area specified by the trimming area of the captured image.
The control unit further includes:
After determining the trimming area, a plurality of intersection candidates formed by the two straight lines are obtained from a combination of a plurality of straight lines obtained by extending a plurality of line segments constituting the trimming area. Among these, the four candidate intersections whose coordinates are located on the outermost side are determined as first vertex coordinates for trapezoid correction,
A trapezoidal distortion index defined by the four first vertex coordinates is calculated from the four first vertex coordinates, and a second vertex coordinate after the trapezoid correction is determined based on the calculated index. And
Using the first vertex coordinates and the second vertex coordinates, perspective projection conversion of the captured image, to generate the captured image of the front view,
Using the first vertex coordinates and the second vertex coordinates to perform perspective projection transformation of the trimming region, to generate the trimming region in front view,
The information terminal device according to claim 1, wherein an area specified by the trimming area in front view of the captured image in front view is trimmed.
The control unit further includes:
The image quality adjustment selected from the group consisting of color temperature adjustment, histogram equalization, and gamma correction is applied to the cropped captured image in front view. Information terminal device.
The predetermined area of the captured image is surrounded by a predetermined pattern;
The control unit further includes:
Detecting the presence of the predetermined pattern in the captured image;
The information terminal device according to any one of claims 1 to 3, wherein when the presence of the predetermined pattern is detected in the captured image, the cropped captured image in front view is recorded.
The information terminal device according to claim 4, wherein the predetermined pattern is created using a writing instrument in which a colored portion at a tip is divided into a plurality of protrusions.
An imaged image processing system comprising an information terminal device having an imaging unit and a control unit, and a writing instrument, wherein the information terminal device performs a trimming process on a predetermined area of a captured image acquired from the imaging unit,
A predetermined region of the imaging object is surrounded by a line having a predetermined color using the writing instrument,
The control unit is
From the imaging unit, obtain a captured image obtained by imaging the predetermined area of the imaging object,
Generating an extracted image obtained by extracting only the predetermined color to be trimmed from the captured image;
In the extracted image, a plurality of contour lines corresponding to the predetermined region of the captured image are extracted, and a region having the largest area surrounded by the contour line is determined as a trimming region,
A captured image processing system for trimming an area specified by the trimming area of the captured image.
The control unit further includes:
After determining the trimming area, a plurality of intersection candidates formed by the two straight lines are obtained from a combination of a plurality of straight lines obtained by extending a plurality of line segments constituting the trimming area. Among these, the four candidate intersections whose coordinates are located on the outermost side are determined as first vertex coordinates for trapezoid correction,
A trapezoidal distortion index defined by the four first vertex coordinates is calculated from the four first vertex coordinates, and a second vertex coordinate after the trapezoid correction is determined based on the calculated index. And
Using the first vertex coordinates and the second vertex coordinates, perspective projection conversion of the captured image, to generate the captured image of the front view,
Using the first vertex coordinates and the second vertex coordinates to perform perspective projection transformation of the trimming region, to generate the trimming region in front view,
The captured image processing system according to claim 6, wherein an area specified by the trimming area in the front view of the captured image in the front view is trimmed.
In an information terminal device including an imaging unit and a control unit, a method for trimming a predetermined region of a captured image acquired from the imaging unit,
A predetermined area of the imaging object is surrounded by a line having a predetermined color,
The control unit is
Obtaining a captured image obtained by imaging the predetermined area of the imaging object from the imaging unit;
Generating an extracted image obtained by extracting only the predetermined color to be trimmed from the captured image;
Extracting a plurality of contour lines corresponding to the predetermined region of the captured image in the extracted image, and determining a region having a maximum area surrounded by the contour line as a trimming region;
Trimming a region specified by the trimming region of the captured image.
The control unit is
After determining the trimming area, a plurality of intersection candidates formed by the two straight lines are obtained from a combination of a plurality of straight lines obtained by extending a plurality of line segments constituting the trimming area. Of these, determining the four intersection candidate candidates whose coordinates are located on the outermost side as first trapezoidal coordinates for trapezoid correction;
A trapezoidal distortion index defined by the four first vertex coordinates is calculated from the four first vertex coordinates, and a second vertex coordinate after the trapezoid correction is determined based on the calculated index. And steps to
Performing perspective projection transformation of the captured image using the first vertex coordinates and the second vertex coordinates to generate the captured image in front view;
Performing perspective projection transformation of the trimming area using the first vertex coordinates and the second vertex coordinates, and generating the trimming area in front view;
The captured image processing method according to claim 8, further comprising: trimming a region specified by the trimming region in the front view of the captured image in the front view.
In an information terminal device including an imaging unit and a control unit, a computer-readable recording medium storing a program for trimming a predetermined area of a captured image acquired from the imaging unit,
A predetermined area of the imaging object is surrounded by a line having a predetermined color,
The control unit is
Obtaining a captured image obtained by imaging the predetermined area of the imaging object from the imaging unit;
Generating an extracted image obtained by extracting only the predetermined color to be trimmed from the captured image;
Extracting a plurality of contour lines corresponding to the predetermined region of the captured image in the extracted image, and determining a region having a maximum area surrounded by the contour line as a trimming region;
A computer-readable recording medium recording a captured image processing program for causing a computer to perform a step of trimming an area specified by the trimming area of the captured image.
The control unit further includes:
After determining the trimming area, a plurality of intersection candidates formed by the two straight lines are obtained from a combination of a plurality of straight lines obtained by extending a plurality of line segments constituting the trimming area. Of these, determining the four intersection candidate candidates whose coordinates are located on the outermost side as first trapezoidal coordinates for trapezoid correction;
A trapezoidal distortion index defined by the four first vertex coordinates is calculated from the four first vertex coordinates, and a second vertex coordinate after the trapezoid correction is determined based on the calculated index. And steps to
Performing perspective projection transformation of the captured image using the first vertex coordinates and the second vertex coordinates to generate the captured image in front view;
Performing perspective projection transformation of the trimming area using the first vertex coordinates and the second vertex coordinates, and generating the trimming area in front view;
The computer-readable computer program product according to claim 10, wherein a captured image processing program for causing a computer to perform the step of trimming an area specified by the trimming area of the front view of the captured image of the front view is recorded. recoding media.