JP4363152B2 - Captured image projection device, image processing method and program for captured image projection device - Google Patents

Description

  The present invention relates to a captured image projection apparatus, method, and program.

  In OHP, which reads and projects paper documents and OHP paper, with the development of digital cameras, there is a photographic image projection device called a document camera using a digital camera (document camera) instead of a conventional line scanner. ) Is being developed.

  In this photographed image projection device, a user photographs a document placed on a document table with a camera, stores image data of the document photographed with the camera in a storage device of a computer, performs image processing, and uses a projector to perform screen processing. A document image is enlarged and projected on the image (for example, see Patent Document 1).

Such a photographed image projection apparatus is changing for a conventional OHP because a three-dimensional object to be photographed that could not be obtained by the conventional OHP can be imaged and projected.
JP 2002-354331 A (page 2-3, FIG. 1)

  However, in a conventional photographed image projection apparatus, when a document is photographed from an oblique direction with a camera, the side of the document that is close to the camera deviates from the angle of view (shooting range) of the camera, so the zoom magnification of the camera must be reduced. I don't get it.

  As a result, the area occupied by the document decreases, so even if keystone correction is performed, the ratio of the document to the image decreases, the projected document image becomes smaller, and the image projected on the screen becomes difficult to see. End up. This tendency becomes prominent especially when the tilt angle is increased.

  In addition, since it is difficult to accurately extract the shape of the document image, extra image portions other than the document are included even if the keystone correction is performed, and it becomes difficult to see the image projected on the screen.

  The present invention has been made in view of such a conventional problem, and an object thereof is to provide a photographic image projection apparatus capable of acquiring an easy-to-see image, an image processing method of the photographic image projection apparatus, and a program. To do.

In order to achieve this object, a captured image projection apparatus according to the first aspect of the present invention provides:
In a photographic image projection device that projects an image of a photographed document on a screen,
An image processing unit that performs image processing on any one of a plurality of shapes of the document image so as to correct distortion of the document image obtained by shooting;
A projection unit that projects the image processed by the image processing unit onto the screen;
The image processing unit
A shape acquisition unit that acquires the outline of the original image candidate from the original image and acquires the shape surrounded by the acquired outline and the vertex position thereof;
An area acquisition unit for obtaining an area of the shape acquired by the shape acquisition unit;
The area obtained by the area acquisition unit is compared with the area of the document size designated in advance, and the shape having the area closest to the area of the designated document size is obtained among the areas obtained by the area acquisition unit. A shape selection unit for selecting the shape of the original image;
The shape of the original image selected by the shape selection unit is associated with the actual original shape, and the relationship between the original image and the actual original is determined from the vertex position of the original image acquired by the shape acquisition unit. A projection parameter acquisition unit for obtaining a projection parameter to be shown;
An image conversion unit that performs image conversion of the document image using the projection parameter obtained by the projection parameter acquisition unit.

The shape acquisition unit
An edge image detector for detecting an edge image of the document image;
A straight line detection unit that detects straight lines that are candidates for the outline of the document from the edge image detected by the edge image detection unit;
The outline of the document may be acquired by combining straight lines detected by the straight line detection unit.

  The area acquisition unit may determine the area of the shape surrounded by the outline acquired by the shape acquisition unit based on the size of the document image and the vertex position acquired by the shape acquisition unit.

A lens for condensing light, an image forming unit for forming an image by the light collected by the lens, and a lens control unit for controlling the position of the lens so that the original image is formed on the image forming unit And a camera having
The area acquisition unit
Based on the distance between the lens and the image forming unit when the document image is formed on the image forming unit by the control of the lens control unit, and the distance between the lens and the document. The size of the original image acquired by the shape acquisition unit may be obtained.

The lens control unit
A light output unit that outputs light for distance measurement;
A distance sensor that measures a distance between the lens and the document based on light reflected from the document by the light output from the light output unit;
The area acquisition unit is configured to control a distance between the lens and the document measured by the distance sensor when the document image is formed on the image forming unit by the control of the lens control unit. The distance between the lens and the imaging unit may be obtained based on the focal length.

The lens control unit controls the position of the lens so that the distance between the lens and the image forming unit is preset and the contrast of the document image formed on the image forming unit is maximized. Passive type
The area acquisition unit is based on the distance between the lens and the imaging unit output from the lens control unit when the contrast of the document image is maximized, and the focal length of the lens. A distance between the lens and the document may be obtained.

An image processing method of a captured image projection apparatus according to a second aspect of the present invention includes:
An image processing method of a photographic image projection apparatus for projecting a photographed document image on a screen,
Obtaining a contour of the document image candidate from the document image having a plurality of shapes, obtaining the shape surrounded by the obtained contour and its vertex position;
Obtaining an area of the acquired shape;
By comparing the area of pre-specified document size and the obtained area of the obtained area, selecting the shape having the closest area to the area of the designated document size as the shape of the original image Steps,
Associating the shape of the selected document image with the shape of the actual document, and obtaining a projection parameter indicating the relationship between the document image and the actual document from the vertex position of the acquired document image;
Performing the image conversion of the document image using the obtained projection parameter.

The program according to the third aspect of the present invention is:
On the computer,
A procedure for acquiring the contour of the document image candidate from a document image having a plurality of shapes obtained by photographing, and acquiring the shape surrounded by the acquired contour and its vertex position;
A procedure for obtaining an area of the acquired shape;
By comparing the area of pre-specified document size and the obtained area of the obtained area, selecting the shape having the closest area to the area of the designated document size as the shape of the original image procedure,
A procedure for associating the shape of the selected original image with the actual original shape and obtaining a projection parameter indicating the relationship between the original image and the actual original from the vertex position of the acquired original image;
A procedure for performing image conversion of the document image using the obtained projection parameter;
Is to execute.

  According to the present invention, an easy-to-see image can be obtained.

Hereinafter, a captured image projection apparatus according to an embodiment of the present invention will be described with reference to the drawings.
(Embodiment 1)

FIG. 1 shows the configuration of the captured image projection apparatus according to the first embodiment.
The captured image projection apparatus includes a document camera 1 and a projector 2.

  The document camera 1 is a camera system for photographing a document 4 to be projected, and includes a camera unit 11, a support column 12, a pedestal 13, and an operation table 14. The camera unit 11 is for photographing the document 4. A digital camera is used for the camera unit 11. The column 12 is for attaching the camera unit 11. The pedestal 13 is for supporting the column 12.

  Further, the document camera 1 includes an image data generation unit 21 and a data processing unit 22 as shown in FIG. The image data generation unit 21 captures the document 4 and captures the image data of the document 4.

  The data processing unit 22 acquires image data of the captured original 4 from the image data generation unit 21, performs image processing, and outputs the image data subjected to the image processing to the projector 2.

  The image data generation unit 21 and the data processing unit 22 may be provided in the camera unit 11 shown in FIG. 1, or the image data generation unit 21 and the data processing unit 22 are respectively connected to the camera unit 11 and the operation console. 14 may be provided.

  The image data generation unit 21 includes an optical lens device 101 and an image sensor 102.

  The optical lens device 101 is composed of a lens that collects light in order to photograph the document 4.

  The image sensor 102 is for capturing an image formed by the optical lens device 101 condensing light as digitized image data, and is configured by a CCD or the like.

  The data processing unit 22 includes a memory 201, a display device 202, an image processing device 203, an operation unit 204, a program code storage device 205, and a CPU 206.

The memory 201 is for temporarily storing an image from the image sensor 102, image data to be sent to the display device 202, and the like.
The display device 202 is for displaying an image sent out to the projector 2.

  The image processing device 203 is for performing image processing such as image distortion correction and image effect on the image data temporarily stored in the memory 201.

  The image processing apparatus 203 performs distortion correction (trapezoid correction) from the image of the document 4 taken in an oblique state or rotated state with respect to the photographing surface of the camera unit 11 as shown in FIG. Thus, an image of the document 4 taken from the front as shown in FIG. 3B is generated.

  In order to perform distortion correction, the image processing apparatus 203 cuts a square from the image of the distorted original 4, performs projective conversion on the image of the original 4 to the cut out square, and performs inverse conversion.

More specifically, the image processing device 203 is controlled by the CPU 206 to perform the following processing.
(1) Extraction of affine parameters from image of document 4 (2) Image conversion using extracted affine parameters (3) Extraction of image effect correction parameters relating to luminance or color difference and image effect processing (4) Adjustment of image conversion Details of these processes will be described later.

  The operation unit 204 includes switches and keys for controlling the document projection function. The operation unit 204 outputs operation information to the CPU 206 when the user operates these keys and switches.

  Further, as shown in FIG. 4, the operation unit 204 includes an upper enlargement key 211, a lower enlargement key 212, a left enlargement key 213, a right enlargement key 214, a right rotation key 215, and a left as image adjustment keys. A rotation key 216, an enlargement key 217, a reduction key 218, and a shooting / release key 219 are provided.

  An upper enlargement key 211, a lower enlargement key 212, a left enlargement key 213, and a right enlargement key 214 are projective conversion keys operated when performing projective conversion.

  The upper enlargement key 211 is used to compare the upper part of the screen above the horizontal line and the lower part of the screen below the horizontal line with the horizontal line passing through the center of the image as an axis, and rotate the upper part of the screen to the viewing side. The lower enlarged key 212 is a key operated when the lower part of the screen is rotated to the viewing side.

  The left enlargement key 213 and the right enlargement key 214 are keys for adjusting the left and right distortions by rotating the image around a vertical line passing through the center of the image. A key that is operated when the image is small, and a right enlargement key 214 are keys that are operated when the right is small.

  The right rotation key 215 and the left rotation key 216 are rotation correction keys for adjusting the rotation of the image, respectively. The right rotation key 215 is a key operated when the image is rotated to the right, and the left rotation key. Reference numeral 216 denotes a key operated when the image is rotated to the left.

  The enlargement key 217 and the reduction key 218 are keys operated when performing image conversion such as image enlargement and reduction, respectively.

  If 2 is set in the Ratio indicating the rate of enlargement or reduction by one press, the image processing apparatus 203 enlarges the image twice by pressing the enlargement key 217 once. Further, when the enlargement key 217 is pressed again, the operation unit 204 and the CPU 206 respond to control the image processing device 203, and the image processing device 203 further enlarges the image twice, and as a result, the original image is Enlarged to 4x image. To return the enlarged image to ½, the reduction key 218 is pressed. The operation unit 204 and the CPU 206 respond, and the image processing apparatus 203 reduces the enlarged image.

  Further, the upper enlargement key 211, the lower enlargement key 212, the left enlargement key 213, and the right enlargement key 214 also function as cursor keys when the enlargement key 217 is operated. That is, the up enlargement key 211 serves as an up movement key for moving the image of the document 4 displayed on the display device 202 upward when the enlargement key 217 is operated. The lower enlargement key 212 is a downward movement key for moving the image of the document 4 downward when the enlargement key 217 is operated. The left enlargement key 213 serves as a left movement key for moving the image of the document 4 to the left when the enlargement key 217 is operated. The right enlargement key 214 serves as a right movement key for moving the image of the document 4 to the right when the enlargement key 217 is operated.

  Further, the enlargement key 217 and the reduction key 218 function as keys for entering the paper setting mode when pressed simultaneously. In this paper setting mode, the right rotation key 215 and the left rotation key 216 serve as cursor keys for selecting a desired paper size. The description of the paper setting mode will be described later.

  The shooting / cancel key 219 is a key operated when shooting the document 4. In addition, it becomes a determination key for determining the selected paper size in a paper setting mode to be described later.

  The program code storage device 205 is for storing a program executed by the CPU 206, and is configured by a ROM or the like.

  The CPU 206 controls each unit in accordance with a program stored in the program code storage device 205. The memory 201 is also used as a work memory for the CPU 206.

  Further, the CPU 206 determines whether or not there is a motion in the image of the document 4 that is the shooting target, according to a flowchart described later, and switches the shooting mode according to the determination result.

  There are two shooting modes, a moving image mode and a still image mode. The moving image mode is a mode that is set when the user intends to place the document 4 or the like that the user wants to project on the pedestal 13, and is a mode that projects the image projected by the camera unit 11 onto the projector 2 as it is.

  In the moving image mode, for example, the CPU 206 controls each unit so that an image with an image resolution of about VGA (600 × 480 dots) is projected at a speed of 30 fps (frames / second). As described above, the moving image mode is a mode in which real-time characteristics are emphasized although the resolution is low.

  The still image mode is a mode that is set after the user places the document 4, and is a mode in which the camera unit 11 captures a high resolution image and projects a high resolution still image. When the camera unit 11 is a camera with an imaging resolution of 3 million pixels, the cut-out projected image is an XGA (1024 × 768) still image.

  The CPU 206 determines whether or not there is a motion in the image of the document 4 in order to switch the shooting mode. In order to make such a determination, the CPU 206 obtains an image change amount MD from the previously captured image. The image change amount MD is an amount representing how much the image being taken has changed compared to the previous image. There are several ways to calculate this quantity. As an example, the CPU 206 obtains the sum of absolute values of differences between pixels as the image change amount MD from the data of the previously captured image.

That is, if the previous pixel data is Pn-1 (x, y), the current pixel data is Pn (x, y), and 1 ≦ x ≦ 640 1 ≦ y ≦ 480, the image change amount MD is the following number: Represented by 1.

However, since the amount of calculation is large in order to obtain the total sum of all the pixels, as a method for obtaining the image change amount MD, there is a method in which several pixels are extracted and realized.

  In addition, a threshold value Thresh1 and a threshold value Thresh2 are set in advance as threshold values for determining the presence or absence of motion in comparison with the image change amount MD. The threshold value Thresh1 is a threshold value for determining whether or not there is a motion, and the CPU 206 determines that there is no motion if the image change amount MD is less than the threshold value Thresh1.

  The threshold value Thresh2 is a slight movement that does not require switching to the video mode even if there is a movement, for example, when a shadow moves or a pen is placed in the still image mode. It is a threshold value for determining whether or not.

  If the image change amount MD is less than the threshold value Thresh2, the CPU 206 determines that the movement is slight enough not to switch to the moving image mode. The threshold value Thresh2 is set higher than the threshold value Thresh1 (Thresh1 <Thresh2). The memory 201 stores the threshold value Thresh1 and the threshold value Thresh2 in advance.

  Further, when switching from the moving image mode to the still image mode, the CPU 206 determines that there is no movement in the image, and switches the shooting mode to the still image mode after a predetermined time has elapsed. For this reason, the CPU 206 determines that there is no movement in the image and then measures the still time Ptime, and the memory 201 stores a predetermined time HoldTime set in advance for comparison with the still time Ptime.

  The distance sensor 103 is a sensor provided in the vicinity of the lens device in order to measure the distance to the photographing target, and can measure the distance to the subject target on which the center of the camera image is reflected.

  The projector 2 irradiates the screen 3 with projection light and forms an image of the document 4. The document camera 1 and the projector 2 are connected via a cable 15 such as RGB. The document camera 1 outputs the image data of the document 4 photographed by the camera unit 11 to the projector 2 via the cable 15.

Next, the operation of the captured image projection apparatus according to the first embodiment will be described.
When the user turns on the power of the captured image projection apparatus, the CPU 206 of the document camera 1 reads out the program code from the program code storage device 205 and executes basic projection processing.

The contents of this basic projection process will be described with reference to the flowchart shown in FIG.
First, the CPU 206 initializes camera setting parameters such as focus, exposure, and white balance of the camera unit 11 of the document camera 1 (step S11).

  The CPU 206 initializes the shooting mode to the moving image mode (step S12). The CPU 206 sets the moving image mode (Motion) in the designated area on the memory 201 in order to initialize the shooting mode to the moving image mode. Further, the CPU 206 sets the image data read from the image sensor 102 of the camera unit 11 to be image data by VGA.

  As a result, the scene captured by the camera unit 11 is condensed on the image sensor 102 via the optical lens device 101, and the image sensor 102 creates a low-resolution digital image for moving image shooting from the collected image. To do. Then, the image sensor 102 sends the created digital image to the memory 201 periodically, for example, at 30 sheets per second.

Next, the CPU 206 resets the stationary time Ptime (step S13).
Next, the CPU 206 controls the image sensor 102 and the memory 201 so as to transfer low-resolution image data from the image sensor 102 to the memory 201 (step S14). Here, only the image data of the image sensor 102 is transferred to the memory 201, and the display device 202 does not display an image. The reason why the display device 202 does not display an image is that image data to be displayed on the display device 202 is stored in areas indicated by different addresses on the memory 201.

  Next, the CPU 206 obtains the image change amount MD with respect to the previously captured image according to Equation 2 (step S15).

The CPU 206 determines whether the shooting mode is the moving image mode or the still image mode (step S16).
In the initial state, the movie mode is set as the shooting mode. For this reason, the CPU 206 determines that the moving image mode is set in the initial state, and copies the moving image data in the memory 201 to a predetermined area of the memory 201 in order to project the captured moving image (step S17). As a result, the display device 202 reads the captured image data from the memory 201 and outputs RGB signals to the projector 2. The projector 2 projects an image based on this signal.

  The CPU 206 compares the aforementioned threshold value Thresh1 set in advance with the image change amount MD obtained in step S15, and determines whether or not there is a motion based on the comparison result (step S18).

  At this time, if the user still continues the operation such as placing paper, the image change amount MD is equal to or greater than the threshold value Thresh1. In this case, the CPU 206 determines that there is a motion in the image (Yes in step S18), resets the still time Ptime, reads low-resolution image data, obtains the image change amount MD, and obtains the image memory 201. Writing to the projected image area is performed (steps S13 to S17). As a result, while the user performs an operation, the captured image projection device continues to maintain the moving image mode state, and a low resolution moving image is projected onto the screen 3.

  After that, when the user finishes setting the paper and the image stops moving, the image change amount MD becomes less than the threshold value. In this case, the CPU 206 determines that there is no motion in the image (No in step S18), and adds 1 to the still time Ptime (step S19).

  Then, the CPU 206 determines whether or not the stationary time Ptime has reached a predetermined time HoldTime (step S20).

  When it is determined that the still time Ptime has not reached the predetermined time HoldTime (No in step S20), the CPU 206 reads low-resolution image data again and determines the image change amount MD until it determines that there is no motion in the image. If there is no movement, 1 is added to the stationary time Ptime (steps S14 to S19). In this case, since the stationary time Ptime is not reset, the stationary time Ptime is counted up every time the moving image is read by the CPU 206.

  If it is determined that the still time Ptime has reached the predetermined time HoldTime (Yes in step S20), the CPU 206 determines that the user has stopped the image and sets the shooting mode to the still image mode (step S21).

  The CPU 206 controls the image sensor 102 so as to capture a high-resolution still image (step S22). Then, the CPU 206 writes the image data acquired by the image sensor 102 in the memory 201. The CPU 206 writes the image data in the memory 201 and then returns the resolution of the camera unit 11 to the low resolution reading state again.

  The CPU 206 controls the image processing apparatus 203, and the image processing apparatus 203 extracts projection parameters for performing oblique correction on the read high-resolution still image (step S23). This projection parameter indicates the relationship between the shape of the image of the document 4 and the actual shape of the document 4.

Further, the image processing apparatus 203 performs clipping of the object to be imaged and projection conversion to the front image in accordance with the extracted projection parameters (step S24).
The image processing apparatus 203 extracts image effect correction parameters in order to convert the created corrected image into a clear and easy-to-identify image such as contrast correction (step S25).
The image processing apparatus 203 performs image effect processing using the extracted image effect correction parameters (step S26).

  The CPU 206 writes the corrected image data in a predetermined area in the memory 201 as in the case of the moving image in order to project the image data subjected to the image effect processing. Then, the CPU 206 causes the projector 2 to output image data as RGB signals from the display device 202 (step S27).

  Once in the still image mode, the CPU 206 resets the still time Ptime again, reads a low resolution image, and obtains the image change amount MD (steps S13 to S15).

  Here, when the shooting mode is determined to be the still image mode (step S16), the CPU 206 compares the obtained image change amount MD with another predetermined threshold value Thresh2 (Thresh1 <Thresh2), and the image has a motion. Whether or not (step S28).

  If the image change amount is less than the threshold value Thresh2, the CPU 206 determines that there is no motion in the image (No in step S28). In this case, the CPU 206 continues to compare the image change amount MD with the threshold value Thresh1, and determines whether or not there is a motion in the image (step S30).

  If the image change amount MD is greater than or equal to the threshold value Thresh2, the CPU 206 determines that the image is moving (Yes in step S28), and sets the shooting mode to the moving image mode (step S29). Then, the CPU 206 resets the stationary time Ptime (step S13).

  On the other hand, even if the image change amount MD is less than the threshold value Thresh2, if it is equal to or greater than the threshold value Thresh1, the CPU 206 determines that the image has a motion (Yes in step S30). In this case, since the image change amount MD is between the threshold Thresh1 and the threshold Thresh2, the CPU 206 captures a still image at a high resolution again without switching the capturing mode to the moving image mode. Image processing of the acquired high-resolution still image data is performed (steps S22 to S27).

  If the image change amount MD is less than the threshold value Thresh2 and less than the threshold value Thresh1, the CPU 206 determines that the still state is continuing (No in step S30). In this case, the CPU 206 determines whether or not the image adjustment key is operated based on the operation information output from the operation unit 204 as the instruction information for adjusting the image effect (step S31).

  If it is determined that the image adjustment key has not been operated (No in step S31), the CPU 206 returns to step S13 and resets the stationary time Ptime.

  When it is determined that the image adjustment key has been operated (Yes in step S31), the CPU 206 performs image conversion adjustment such as image enlargement or rotation (step S32).

  The CPU 206 determines whether the image conversion is projective conversion, rotation, or the like, or enlargement / reduction or movement of the image (step S33).

  When the CPU 206 determines that the image conversion is projective conversion, rotation, or the like, the image processing apparatus 203 is controlled by the CPU 206 to extract the image effect correction parameters again (step S25) and perform the image effect processing. This is performed (step S26). On the other hand, when the CPU 206 determines that the image conversion is enlargement / reduction or movement of the image, the image processing device 203 performs image effect processing using the image effect correction parameter extracted from the previous time from the image (step S26). ). Then, the CPU 206 causes the projector 2 to output image data as RGB signals from the display device 202 (step S27).

  In this way, the CPU 206 performs projection control while switching between the moving image mode and the still image mode. Thereby, while the user is performing the operation, the frame frequency priority image is projected, and when the image is stationary, the document 4 to be imaged is cut out and the image effect processing is performed to project the high resolution image. Done.

Next, image processing performed by the image processing apparatus 203 will be described.
First, a basic concept (implementation method) of affine transformation used by the image processing apparatus 203 for image processing will be described.

Affine transformation is widely applied to image spatial transformation. In this embodiment, projective transformation is performed using two-dimensional affine transformation without using three-dimensional camera parameters. This is because the coordinates (x, y) after conversion are related by the following equation (2) by conversion such as movement, enlargement / reduction, and rotation of the point of the coordinates (u, v) before conversion. Projective transformation can also be performed by this affine transformation.

Final coordinates (x, y) are calculated by the following equation (3).

Expression 3 is an expression for projective transformation, and the coordinates x and y are reduced toward 0 according to the value of z ′. That is, the parameter included in z ′ affects the projection. This parameter is a13, a23, a33. Since other parameters can be normalized by the parameter a33, a33 may be 1.

  FIG. 6 shows the coordinates of the vertices of the quadrangular document 4 shown in FIG. The relationship between the quadrangle photographed by the camera unit 11 and the actual image of the document 4 will be described with reference to FIG.

In FIG. 7, the UVW coordinate system is a three-dimensional coordinate system of an image obtained by photographing with the camera unit 11. The A (Au, Av, Aw) vector and the B (Bu, Bv, Bw) vector represent the paper of the document 4 as vectors in the three-dimensional coordinate system UV-W.
The S (Su, Sv, Sw) vector indicates the distance between the origin of the three-dimensional coordinate system UV-W and the document 4.

The projection screen shown in FIG. 7 is for projecting the image of the document 4 (corresponding to the screen 3).
If the coordinate system on the projection screen is x, y, the image projected on the projection screen may be considered as the image photographed by the camera unit 11. It is assumed that the projection screen is positioned vertically at a distance f from the W axis. Assume that an arbitrary point P (u, v, w) on the paper of the document 4 and the origin are connected by a straight line, and that there is a point where the straight line intersects the projection screen, the XY coordinate of the intersection is p (x , y). At this time, the coordinate p is expressed by the following equation 4 from the projective transformation.

From Equation 4, the relationship shown in the following Equation 5 is obtained from the relationship between the points P0, P1, P2, P3 and the projection points p0, p1, p2, p3 on the projection screen as shown in FIG.

At this time, the projection coefficients α and β are expressed by the following equation (6).

Next, projective transformation will be described.
An arbitrary point P on the document 4 is expressed by the following equation 7 using S, A, and B vectors.

When the relational expression of Expression 5 is substituted into Expression 7, the coordinates x and y are expressed by Expression 8 below.

When this relationship is applied to the affine transformation formula, the coordinates (x ′, y ′, z ′) are expressed by the following equation (9).

  By substituting m and n into Equation 9, the corresponding point (x, y) of the captured image is obtained. Since the corresponding point (x, y) is not necessarily an integer value, the pixel value may be obtained using an image interpolation method or the like.

In order to perform such affine transformation, the image processing apparatus 203 first extracts projection parameters from the captured image of the document 4 (step S23 in FIG. 5).
Projection parameter extraction processing executed by the image processing apparatus 203 will be described with reference to the flowchart shown in FIG.

  The image processing device 203 extracts coordinate points (rectangular contours) at the four corners of the image of the document 4 from the captured image of the document 4 (step S41). The image processing apparatus 203 extracts a rectangular outline shown in the flowchart of FIG.

That is, the image processing apparatus 203 generates a reduced luminance image from the input image in order to reduce the number of operations for image processing (step S51).
The image processing device 203 generates an edge image of the document 4 from the generated reduced luminance image (step S52).

The image processing device 203 detects a straight line parameter included in the edge image from the edge image of the document 4 (step S53).
The image processing apparatus 203 generates a quadrangle that is a candidate for forming the contour of the document 4 from the detected straight line parameter (step S54).

The image processing apparatus 203 thus generates candidate rectangles and assigns priorities to the generated rectangles (step S42 in FIG. 8).
The image processing apparatus 203 selects a rectangle according to the priority order and determines whether or not the selected rectangle has been extracted (step S43).

  If it is determined that the quadrangle has been extracted (Yes in step S43), the image processing apparatus 203 acquires a quadrangle as the shape of the image of the document 4, and calculates an affine parameter from the acquired quadrangle (step S44).

  On the other hand, if it is determined that the image could not be extracted (No in step S43), the image processing apparatus 203 acquires a quadrangular image having the maximum area as an image of the document 4 (step S45), and the affine is obtained from the acquired quadrangular vertex. A parameter is calculated (step S44).

Next, the projection parameter extraction process will be described more specifically.
An example of the reduced luminance image generated by the image processing apparatus 203 in step S51 is shown in FIG. The image processing apparatus 203 generates an edge image as shown in FIG. 10B from such a reduced luminance image using an edge detection filter called a Roberts filter (step S52). The Roberts filter is a filter that detects an edge of an image by weighting two 4-neighboring pixels, obtaining two filters Δ1 and Δ2, and averaging them.

11A shows the coefficient of the filter Δ1, and FIG. 11B shows the coefficient of the filter Δ2. When the coefficients of the two filters Δ1, Δ2 are applied to the pixel value f (x, y) of a certain coordinate (x, y), the converted pixel value g (x, y) Represented by

  The edge image shown in FIG. 10B includes straight line parameters. The image processing device 203 performs radon transformation from the edge image to detect straight line parameters (processing in step S53).

The Radon transform is an integral transform that associates n-dimensional data with (n-1) -dimensional projection data. Specifically, as shown in FIGS. 12A and 12B, an r-θ coordinate system rotated by an angle θ from the xy coordinate system with image data f (x, y) is considered. Image projection data p (r, θ) in the θ direction is defined by the following equation 11.

This conversion by Equation 11 is called Radon conversion.
Image data f (x, y) as shown in FIG. 12A is converted into image projection data p (r, θ) as shown in FIG. 12B by Radon transformation.

  According to such a Radon transform, a straight line L in the xy coordinate system as shown in FIG. 13A is represented by ρ = x cos α + y sin α in the polar coordinate system. Since this straight line L is entirely projected on the point p (ρ, α), it is possible to detect the straight line by detecting the peak of p (r, θ). Using this principle, the image processing apparatus 203 generates data p (r, θ) from the edge image by Radon transform.

  Next, the image processing apparatus 203 extracts the peak point from the generated data p (r, θ). Therefore, the image processing apparatus 203 executes this peak point extraction process according to the flowchart shown in FIG.

The image processing apparatus 203 searches for the maximum value pmax at p (r, θ) (step S61).
The image processing apparatus 203 calculates a threshold value pth = pmax * k (k is a constant value between 0 and 1) (step S62).

The image processing apparatus 203 generates a binary image B (r, θ) from p (r, θ) using pth as a threshold (step S63).
The image processing device 203 performs B (r, θ) image labeling. The number of labels obtained at this time is N1 (step S64).

  The image processing apparatus 203 examines r and θ at the maximum value of p (r, θ) in each label area. Then, the image processing apparatus 203 acquires these values as ri, θi (i = 1 to Nl), respectively (step S65). This is a straight line parameter.

  Next, since the image processing apparatus 203 generates a quadrilateral candidate using the detected straight line parameter (processing in step S54), the remarkably distorted quadrilateral is excluded from the candidates. This is because the document camera 1 does not shoot the document 4 that is far away from the pedestal 13, and therefore it is assumed that a significantly distorted square does not represent the outline of the document 4.

  That is, as shown in FIG. 15A, four straight lines a1 to a4 are displayed in the display area of the LCD of the display device 202 when the camera unit 11 takes an image, and the intersection points p1 to p4 of the four straight lines a11 to a14 are displayed. When p4 is displayed, the straight lines a1 to a4 are candidates for straight lines forming a quadrangle.

  On the other hand, when there are less than four straight lines displayed in the display area, or even when four straight lines are displayed in the display area as shown in FIGS. Otherwise, this straight line set is not a candidate for a straight line set forming a quadrangle.

  As shown in FIG. 15A, if the angles of the straight lines a1 to a4 with respect to the lower side of the display area are θ11, θ12, θ21, and θ22, respectively, the angle closest to the angle θ11 of the straight line a1 is the angle of the straight line a3. Since θ12, the straight line a1 and the straight line a3 are determined as straight lines facing each other.

  Further, as shown in FIG. 15D, if the angle difference between the straight lines a5 and a6 facing each other is large, the document 4 is not photographed by the document camera 1 at a position away from the pedestal 13, and therefore the straight line a5. The group of .about.a8 is not a candidate for a group of straight lines forming a quadrangle. If the original 4 is placed on the pedestal 13 and the angle of the straight line is measured, it can be determined whether or not the straight line is based on the sides of the original 4. Therefore, the angle difference dθ1 is defined as dθ1 and dθ2. , Dθ2 are provided with threshold values dθ1th and dθ2th in advance, and the threshold values dθ1th and dθ2th are stored in the memory 201.

  Based on such a viewpoint, the image processing apparatus 203 acquires a quadrangle candidate from a plurality of straight lines. Specifically, the image processing apparatus 203 executes this quadrangle detection process according to the flowchart shown in FIG.

The image processing apparatus 203 initializes the number of rectangle candidates Nr to 0 (step S71).
The image processing apparatus 203 acquires the number N1 of straight lines displayed in the display area (step S72).

The image processing apparatus 203 determines whether or not the acquired number of straight lines N1 is four or more (step S73).
When it is determined that the number N1 of the acquired straight lines is less than 4 (No in step S73), the image processing apparatus 203 ends this process. In this case, the number of rectangle candidates Nr is zero.

  When it is determined that the number N1 of the acquired straight lines is four or more (Yes in Step S73), the image processing apparatus 203 determines whether there is a set of straight lines that form a quadrangle (Step S74).

  Even if there is a set of straight lines, if there are not four intersections due to the set straight lines, the image processing apparatus 203 determines that there is no set of straight lines forming a quadrangle (No in step S74), and performs this processing. Terminate.

  If it is determined that there is a pair of straight lines that form a quadrangle (Yes in step S74), the image processing apparatus 203 acquires the angle differences dθ1 and dθ2 of the corresponding pair of straight lines (step S75).

  The image processing apparatus 203 determines whether or not the angle difference dθ1 is less than a preset threshold value dθ1th (step S76).

  When it is determined that the angle difference dθ1 is greater than or equal to the threshold value dθ1th (No in step S76), the image processing apparatus 203 repeats the process for the next candidate for the straight line group that forms the quadrangle (steps S74 to 75).

  On the other hand, when it is determined that the angle difference dθ1 is less than the threshold value dθ1th (Yes in step S76), the image processing apparatus 203 determines whether the angle difference dθ2 is less than the threshold value dθ2th (step S77).

  When it is determined that the angle difference dθ2 is greater than or equal to the threshold value dθ2th (No in step S77), the image processing apparatus 203 repeats the process for the next candidate for the straight line group forming the quadrangle (steps S74 to S76).

  On the other hand, when it is determined that the angle difference dθ2 is less than the threshold value dθ2th (Yes in step S77), the image processing apparatus 203 uses the obtained quadrangle having four intersections as candidates and uses only one quadrangle candidate number Nr. Increment (step S78), and the number of candidate rectangles Nr is stored in the memory 201.

  The image processing apparatus 203 stores a set of straight lines forming a quadrangle in the memory 201 (step S79).

  If the image processing apparatus 203 sequentially executes the processes in steps S74 to S79 and determines that there is no straight line group candidate that forms the next square (No in step S74), the image processing apparatus 203 performs the square detection process. Terminate.

  Next, the image processing apparatus 203 selects a most suitable rectangle as representing the side of the document 4 from the rectangle candidates. First, a method for obtaining the size of each quadrangle candidate being photographed will be described.

From Equation 5, the vertical length | A | and the horizontal length | B | of the quadrangle to be imaged are expressed by the following Equation 12.

Here, since k1 = Sw / f, if the distance Sw to the photographing target and the camera parameter f are known, the size of the rectangle can be obtained.

  There are several methods for obtaining the distance Sw to the object to be photographed and the camera parameter f, and differ depending on what kind of lens the optical lens device 101 includes. Specifically, there are two types: 1) a fixed focus lens and 2) an auto focus lens. Hereinafter, in each case, a method for obtaining the distance Sw to the imaging target and the camera parameter f will be described.

1) Fixed focus lens When a fixed focus lens is used in the optical lens device 101, the focal length f of the camera is set in advance to a predetermined value. On the other hand, the distance Sw to the photographing target (original 4) can be approximated to the distance D from the camera unit 11 to the pedestal 13. The distance D from the camera unit 11 to the pedestal 13 is easily obtained from the mechanical configuration of the optical lens device 101 when the user performs focusing (forms the image of the document 4 on the image sensor 102). Possible value. Therefore, | A | and | B | can be obtained by substituting f = lens focal point and Sw = D into Equation 12.

  However, in order to obtain the size of the quadrangle with higher accuracy, it is desirable that the distance Sw to the object to be photographed can be directly measured, instead of performing an approximation of Sw = D. Therefore, it is necessary to measure the distance from the lens of the optical lens device 101 to the document 4 as a subject. For this reason, as shown in FIG. 17, the image data generation unit 21 of the document camera 1 includes a distance sensor 103. The distance sensor 103 is composed of an infrared sensor or the like, is provided adjacent to the optical lens device 101, and measures the distance to the document 4 displayed at the center of the image of the camera unit 11.

  When the document camera 1 has a configuration as shown in FIG. 17, the distance Sw to the photographing target can be represented by a value measured by the distance sensor 103. On the other hand, the focal length f of the camera unit 11 is a known value set in advance as described above. Therefore, | A | and | B | can be obtained by substituting f = lens focal point and Sw = measured value by the distance sensor 103 into Equation 12.

2) Autofocus lens When an autofocus lens is used as the optical lens device 101, the optical lens device 101 changes the focus position of the lens and automatically positions the in-focus position. In such an autofocus lens device, the camera parameter f used in Equation 12 is not strictly the same as the focal length of the lens. For convenience of calculation, when the focal length of the lens is f ′, when a thin lens is used, f ′ uses the distance La from the lens to the image sensor 102 and the distance Lb from the lens to the subject, and It is expressed by Equation 13.

Since the relationship between the distance La and the distance Lb is defined by this equation 13, if either La or Lb is known, f = La and Sw = Lb are substituted into the equation 12 to calculate | A | And | B | can be determined.

  There are two main types of autofocus lenses: a) active system and b) passive system, and the methods for obtaining the distances La and Lb differ in each case.

2-a) Active Autofocus Lens The active autofocus lens is an autofocus method that determines the focus position using infrared rays and PSD (distance sensor). When the optical lens device 101 is an active autofocus lens, the document camera 1 includes an infrared light emitting unit (not shown) that emits a predetermined level of infrared light in addition to the distance sensor 103 described above, and a lens position. And a stepping motor (not shown) for focusing (focusing).

A method for obtaining the distance La and the distance Lb by the document camera 1 having the active autofocus lens will be described with reference to the flowchart of FIG.
First, the CPU 206 controls the infrared light emitting unit to emit infrared light (step S81). Further, the CPU controls the distance sensor 103 to measure the distance Lb to the photographing target (step S82).

On the other hand, the distance La from the lens to the image sensor 102 is expressed by the relationship shown in the following equation 14 using the focal length f ′ of the lens and the distance Lb to the subject.

Therefore, the CPU 206 obtains the distance La by substituting the value of the distance Lb and the focal length f ′ obtained in Step S82 into Equation 14 (Step S83).
The CPU 206 controls and drives the stepping motor, measures the number of steps, and controls the distance from the lens to the image sensor 102 to be a predetermined distance La.

2-b) Passive Method Autofocus Lens The passive method is a method of autofocus (autofocus) in which the lens position is moved to find the position with the highest contrast (phase difference detection) and the focus position is determined.

A method for obtaining the distance La and the distance Lb by the document camera 1 having a passive autofocus lens will be described with reference to the flowchart of FIG.
First, the CPU 206 controls the optical lens device 101 to change the lens position and adjust the focus (step S91). Then, the CPU 206 acquires information on the position of the focused lens, and obtains the value of the distance La between the lens and the image sensor 102 (step S92).

On the other hand, the distance Lb from the lens to the subject is expressed by the relationship expressed by the following equation 15 using the focal length f ′ of the lens and the distance La from the lens to the image sensor 102.

Therefore, the CPU 206 calculates the distance Lb by substituting the distance La and the value of the focal length f ′ obtained in step S92 into Equation 15 (step S93).
In order to calculate the distance Lb to the photographing object with higher accuracy, it is desirable to perform focusing in a state where the aperture is opened in advance (a state where the depth of field is shallow).

  As described above, the CPU 206 obtains the size (| A | and | B |) of each square in the object to be photographed using the lens position, focal length, and the like. Further, the image processing apparatus 203 selects the most suitable rectangle as the side of the document 4 from these rectangles whose sizes are known. Specifically, the image processing apparatus 203 selects a rectangle having a size closest to the paper size of the document as a rectangle representing the document from these rectangles.

  In order for the image processing apparatus 203 to select the rectangle closest to the paper size, it is necessary that the paper size of the document is set and input in advance by the user. The contents of the paper size setting process will be described with reference to the flowchart of FIG.

When the user presses the enlarge key 217 and the reduce key 218 at the same time, the CPU 206 of the document camera 1 reads the program code from the program code storage device 205 and executes a paper size setting process.
First, the CPU 206 copies the image data on the paper setting screen to a predetermined area of the memory 201. As a result, the image data on the paper setting screen is read out to the display device 202 and a predetermined signal is output to the projector 2. Based on this signal, the projector 2 projects a paper size setting screen as shown in FIG. 21 (step S101).

  The CPU 206 initializes the designated sheet number P of the selected sheet to 1 assuming that the sheet size of A4 positioned at the top in the screen of FIG. 21 is in the selected state (step S102). Then, the CPU 206 writes in the memory 201 and controls the display device 202 to turn on (highlight) the selected area of the currently selected paper (in this case, A4 size) on the paper size setting screen (step). S103). Thereby, it is possible to inform the user that the A4 size is selected. Thereafter, the CPU 206 waits for a paper size selection instruction from the user to be input via the operation unit 204.

  The CPU 206 determines which paper size is selected by the instruction from the user input via the operation unit 204 (step S104). When a signal indicating that the right cursor key (left rotation key 214) has been pressed by the user is received from the operation unit 204 (step S104: Sw215), the CPU 206 adds 1 to the designated paper number P (step S105).

  The CPU 206 returns to step S103, writes data in the memory 201, controls the display device 202, and turns on the paper size selection area of the next line on the screen of FIG.

  On the other hand, when a signal indicating that the left cursor key (right rotation key 215) has been pressed is received from the operation unit 204 (step S104: Sw214), the CPU 206 adds −1 to the designated sheet number P (step S106).

  The CPU 206 returns to step S103 to write in the memory 201, and controls the display device 202 to light up the paper size selection area in the upper row on the screen of FIG.

  When a signal indicating that the enter key (shooting / cancellation key 219) has been pressed is received from the operation unit 204 (step S104: Sw219), the CPU 206 is previously called as an array S in the memory 201 from the program code. From the area of each paper size, the area S (P) of the paper size (paper P) currently selected is read as the paper area Sr (step S107). Then, the value of the area Sr is stored in the memory 201 (step S108).

  Next, the contents of the rectangle selection process in which the image processing apparatus 203 selects a rectangle based on the document size set by the above-described paper size setting process will be described with reference to the flowchart of FIG.

The image processing apparatus 203 selects one of the rectangles Rt from the candidate number Nr of rectangles Ri.
First, the image processing apparatus 203 acquires La and Lb information obtained by the focus mechanism described above (step S111).

  The image processing device 203 initializes the variables i, t, and dSmin. Here, i is a variable indicating a rectangle candidate, t is a variable indicating a selected rectangle, and dSmin is the minimum value of the difference | S−Sr | between the area S of the rectangle and the area Sr of the designated sheet. is there. Specifically, the image processing apparatus 203 performs initialization with i = 1, t = 1, and dSmin = MAX (maximum value that can be taken numerically) (step S112).

  Next, the image processing apparatus 203 substitutes the four points of the quadrilateral Ri and the values of Sw = Lb and f = La obtained by the above-described processing into Equation 12, and calculates | A | and | B | Step S113). Then, a square area S (= | A | * | B |) is calculated from the obtained | A | and | B |, and the difference from the designated sheet area Sr, the absolute value dS of (S−Sr). Is obtained (step S114).

The image processing apparatus 203 searches for i that minimizes the absolute value dS of the difference between the sheet area Sr and the square area S.
Specifically, first, the image processing apparatus 203 compares the absolute value dS of the area difference with the value of dSmin (step S115). When dS takes a value greater than or equal to dSmin (step S115: dS ≧ dSmin), the process jumps to step S117. On the other hand, when dS is smaller than dSmin (step S115: dS <dSmin), the image processing apparatus 203 newly sets dSmin = dS and t = i and overwrites the memory 201 (step S116).

  Next, the image processing apparatus 203 adds +1 to the variable i (step S117), and compares i with the number of candidate squares Nr (step S118). When i is smaller than Nr, that is, when processing is not performed for all i (step S118: i ≦ Nr), the image processing apparatus 203 returns to step S113 and continues a series of processing. On the other hand, if i has reached Nr (step S118: i> Nr), i having the smallest dS value is extracted. The candidate number i finally obtained is stored as t in the memory 201, and the corresponding square Rt is selected (step S119).

  In this way, the image processing apparatus 203 always selects, as a document image, a rectangle closest to the paper size designated by the user from among the squares in the image. Thus, even when the document base is square and larger than the document size, it is possible to accurately extract the document image without including an excessive portion.

  In addition, since the rectangle is determined using the absolute value of the predetermined paper area, the shape of the original image is extracted with higher accuracy than in other methods, for example, when comparing using the ratio of each side of the rectangle. can do.

(1) Extraction of Affine Parameter from Image of Document 4 Next, a method for obtaining a projection parameter (affine parameter) from a selected square will be described.
Matrix elements shown in Equation 9 according to Equations 6 and 8 using the vertex coordinates (x0, y0), (x1, y1), (x2, y2), and (x3, y3) of the four points of the selected rectangle An affine parameter can be obtained.

At this time, since the scale of m and n is in a range of 0 ≦ m ≦ 1, 0 ≦ n ≦ 1, the coordinate unit is corrected. Assume that the U-axis side and V-axis side image sizes are scaled by usc and vsc, respectively, without changing the center of the image. At this time, assuming that the center of the image is uc, vc, the scale conversion of the image is expressed by the following equation (16).

When the equation 16 is rewritten on the u and v scales, the coordinates (x ′, y ′, z ′) of the converted rectangle are represented by the following equation 17.

Here, Af is an affine transformation matrix, and the affine transformation matrix Af is expressed by the following equation (18).

Each element of the affine transformation matrix Af is an affine parameter.
From Equations 17 and 18, the aspect ratio k of the quadrangle indicating the relationship between the u-axis and the v-axis is expressed by the following Equation 19.

  In the case of the document camera 1, the focal length of the camera unit 11 is usually designed to be the distance between the lens and the base 13 on which the document 4 is placed. For this reason, the focal length of the camera unit 11 may be the distance between the camera unit 11 and the document 4. In any case, in the case of the document camera 1, the focal length or the distance between the camera unit 11 and the document 4 is a known value.

  However, when the lens of the optical lens device 101 is a zoom lens, the focal length varies depending on the zoom position. Therefore, by storing the camera parameter f corresponding to the zoom position in a table or the like in advance, the aspect ratio k = B / A (absolute value) can be calculated.

When the maximum image (screen) size that can be output to the screen 3 is given by (umax, vmax), uc and vc are expressed by the following equation (20).

Further, when vmax / umax> k and vmax / umax ≦ k, the image is scaled according to the following equations 21 and 22, respectively, and an image having a desired aspect ratio k is obtained.

Based on such a concept, the image processing apparatus 203 acquires affine parameters from quadrangular vertices. This process will be described based on the flowchart shown in FIG.
The image processing apparatus 203 calculates projection coefficients α and β according to Equation 6 from the vertex coordinates (x0, y0), (x1, y1), (x2, y2), and (x3, y3) of the four points of the rectangle. (Step S121).

The image processing apparatus 203 calculates the aspect ratio k of the document 4 according to Equation 19 (step S122).
The image processing apparatus 203 designates the center point (uc, vc) of the image according to Equation 20 (step S123).
The image processing apparatus 203 compares the maximum image size vmax / umax with the aspect ratio k expressed by Equation 16 (step S124).

  When vmax / umax> k (Yes in step S124), assuming that the aspect ratio k is not changed, the image processing apparatus 203 determines that the maximum image size vmax on the V-axis side (vertical) is larger than the image size of the document 4. Judged to be large. Then, the image processing apparatus 203 obtains usc and vsc according to Equation 21 so that the maximum image size on the U-axis side matches the image size of the document 4, and determines the scale of the image of the document 4 on the V-axis side. (Step S125).

  When vmax / umax ≦ k (No in step S124), assuming that the aspect ratio k is not changed, the image processing apparatus 203 determines that the maximum image size umax on the U-axis side (lateral) is larger than the image size of the document 4. Judged to be large. Then, the image processing apparatus 203 obtains usc and vsc according to Equation 22 so that the maximum image size on the V-axis side matches the image size of the document 4 and determines the scale of the image of the document 4 on the U-axis side. (Step S126).

The image processing apparatus 203 calculates Equation 18 from the calculated usc, vsc, uc, vc and the vertex coordinates (x0, y0), (x1, y1), (x2, y2), (x3, y3) of the four points of the rectangle. Thus, an affine transformation matrix Af is obtained (step S127).
The image processing apparatus 203 acquires the affine parameter A using each element of the affine transformation matrix Af as the affine parameter A (step S128).

(2) Image Conversion Using Extracted Affine Parameters Next, an image processing method for creating a corrected image using the obtained affine parameters will be described.
First, when projective transformation or other affine transformation is performed using affine parameters, as shown in FIG. 24, the point p (x, y) of the original image is transformed by projective transformation using the transformation matrix Ap (after). Assume that it corresponds to the point P (u, v) of the image. In this case, it is better to obtain the point p (x, y) of the original image corresponding to the point P (u, v) of the converted image than to obtain the point P of the converted image corresponding to the point p of the original image. preferable.

Note that when obtaining the coordinates of the point P of the converted image, an interpolation method based on the bilinear method is used. In the bilinear interpolation method, the coordinate point of the other image (transformed image) corresponding to the coordinate point of one image (original image) is searched, and the (pixel) values of four points around the coordinate point of the one image are found. Is a method for obtaining a (pixel) value of a point P (u, v) of the converted image. According to this method, the pixel value P of the point P of the converted image is calculated according to the following Expression 23.

In order to obtain the point p (x, y) of the original image corresponding to the point P (u, v) of the converted image, the image processing apparatus 203 executes the processing of the flowchart shown in FIG.
The image processing device 203 initializes the pixel position u of the converted image to 0 (step S131).

The image processing device 203 initializes the pixel position v of the converted image to 0 (step S132).
The image processing apparatus 203 substitutes the pixel position (u, v) of the converted image for the affine parameter A, and obtains the pixel position (x, y) of the original image according to Equation 3 (step S133).

The image processing apparatus 203 obtains a pixel value P (u, v) from the obtained pixel position (x, y) by the bilinear method according to Equation 23 (step S134).
The image processing apparatus 203 increments the corrected image coordinate v by one (step S135).

  The image processing device 203 compares the coordinate v of the corrected image with the maximum value vmax of the coordinate v, and determines whether or not the coordinate v of the corrected image is greater than or equal to the maximum value vmax (step S136).

  When it is determined that the coordinate v is less than the maximum value vmax (No in step S136), the image processing apparatus 203 executes steps S133 to S135 again.

  When it is determined that the coordinate v has reached the maximum value vmax by repeating the processes in steps S103 to S105 (Yes in step S136), the image processing apparatus 203 increments the coordinate u of the corrected image by one ( Step S137).

The image processing apparatus 203 compares the coordinate u with the maximum value umax of the coordinate u, and determines whether or not the coordinate u is equal to or greater than the maximum value umax (step S138).
When it is determined that the coordinate u is less than the maximum value umax (No in step S138), the image processing apparatus 203 executes the processes in steps S132 to S137 again.

  When it is determined that the coordinate u has reached the maximum value umax by repeating the processes of steps 132 to S137 (Yes in step 138), the image processing apparatus 203 ends the image conversion process.

(3) Extraction of image effect correction parameters relating to luminance or color difference and image effect processing Next, processing for extracting image effect correction parameters from the image thus obtained, and image effects performed using these parameters Processing will be described. The image effect process is a process for obtaining a clearer image.

  The image effect correction parameter is a variable necessary for image effect processing, such as a maximum value, a minimum value, a peak value of a luminance histogram, a peak value of a color difference histogram, and an average value.

  In order to extract the image effect correction parameters, it is further necessary to (3-a) cut out an image of the actual original part excluding the contour from the entire image, and (3-b) to generate a histogram of luminance and color difference. First, processing necessary for extracting image effect correction parameters will be described.

(3-a) Image Cutout As shown in FIG. 26, the image of the document 4 includes not only a substantial image such as a photograph, a figure, and a character, but also a pedestal 13 and a shadow on the paper surface of the document 4. In addition, an image irrelevant to the contents of the document 4 may be shown. An inner frame and an outer frame are set for the entire image, and the inner frame and the outer frame in which the actual document portion, the pedestal 13 and the shadow of the paper surface of the document 4 are reflected in the inner frame where the actual image of the document 4 exists. The area between the frame is the periphery. In many cases, the outline of the document 4 in the peripheral portion is generally black. In some cases, it is preferable to have such peripheral data in the image data.

However, if a histogram is generated including an image of the peripheral portion of the document 4, stretching of the histogram may not function effectively.
For this reason, only when the image effect correction parameters are extracted, the image of the actual original portion is cut out to generate a histogram of luminance and color difference, and the image effect correction parameters extracted from the histogram are used to generate an image for the entire image. It is assumed that effect processing is performed. This is a more effective parameter.

  Specifically, if the image data including the peripheral portion is M rows and N columns and the number of dots between the outer frame and the inner frame is K dots in both the X and Y directions, on the X axis side, Pixel data from K to M-K rows that are pixel data of the actual document portion is acquired, and pixel data from K to NK columns that are pixel data of the actual document portion is acquired on the Y-axis side. The pixel data of the actual original part is data of (M-2 * K) rows and (N-2 * K) columns.

(3-b) Generation of luminance and color difference histograms A luminance histogram and a color difference histogram are generated for the image of the actual document portion cut out in this way.

  The luminance histogram shows the distribution of luminance values (Y) existing in the actual document portion, and is generated by counting the number of pixels in the actual document portion for each luminance value. FIG. 27A shows an example of a luminance histogram. In FIG. 27A, the horizontal axis represents the luminance value (Y), and the vertical axis represents the number of pixels. In order to correct the image effect, it is necessary to obtain a maximum value (Ymax), a minimum value (Ymin), and a peak value (Ypeak) as image effect correction parameters.

  The maximum value counts the number of pixels for each luminance value, and is a value indicating the maximum luminance among luminance values having a predetermined number or more set in advance, and the minimum value is a predetermined number or more This is a value indicating the minimum luminance among the luminance values having the counted value. The peak value is a luminance value that maximizes the count value. The peak value is considered to indicate the luminance value of the background color of the document 4 to be photographed.

  The color difference histogram indicates the distribution of the color differences (U, V) existing in the actual document portion, and is generated by counting the number of pixels in the actual document portion for each color difference. FIG. 27B shows an example of the color difference histogram. In FIG. 27B, the horizontal axis indicates the color difference, and the vertical axis indicates the number of pixels. Also in the case of the color difference histogram, a peak value (Upeak, Vpeak) of the color difference where the count value of the pixel becomes maximum appears. This peak value is also considered to indicate the color difference of the background color of the document 4. In order to correct the image effect, it is necessary to obtain the peak value and average value (Umean, Vmean) of the color difference histogram as the image effect correction parameters. The average value is a color difference value having an average count value.

  Note that in order to obtain an image with excellent visibility by correcting the image effect, the correction effect differs depending on the background color of the document 4, so the image effect correction method needs to be changed depending on the background color of the document 4. For this reason, it is necessary to determine the background color of the document 4. The background color of the document 4 is determined from the peak values of the luminance histogram and the color difference histogram.

  Here, the background color of the document 4 is classified into three. The first is a case where the background color is white, such as a whiteboard or notebook. The second is a case where the background color is black, such as a blackboard. The third is a case where the background color is other than white or black, such as a magazine or a pamphlet.

Specifically, the background color of the document 4 is determined according to the following determination formula.
(2-a) White Determination Condition The white determination condition is represented by the following Expression 24, and when the condition shown in Expression 24 is satisfied, the background color of the document 4 is determined to be white (W).

(2-b) Black Determination Condition The black determination condition is expressed by the following Expression 25, and when the condition shown in Expression 25 is satisfied, the background color of the document 4 is determined to be black (b).

  If the conditions shown in equations 24 and 25 are not satisfied, the background color of the document 4 is determined to be color (C). For example, the color threshold is set to 50, the white determination threshold is set to 128, and the black determination threshold is set to 50, for example.

Based on such an idea, the image processing apparatus 203 executes an image effect correction parameter extraction process according to the flowchart shown in FIG.
The image processing apparatus 203 counts the number of pixels having each luminance (Y) value in the actual document portion, and generates a luminance histogram as shown in FIG. 27A (step S141).

The image processing apparatus 203 acquires the maximum value (Ymax), the minimum value (Ymin), and the peak value (Ypeak) of the luminance from the generated luminance histogram (step S142).
The image processing apparatus 203 creates a histogram of color differences (U, V) as shown in FIG. 27B (step S143).

The image processing apparatus 203 obtains the peak value (Upeak, Vpeak) of the color difference (U, V) as the image effect correction parameter (step S144).
The image processing apparatus 203 obtains an average value (Umean, Vmean) of color differences as an image effect correction parameter (step S145).

The image processing apparatus 203 discriminates the background color of the document 4 from these peak values (Ypeak, Upeak, Vpeak) of the image histogram according to the judgment condition formulas shown in Expressions 24 and 25 (step S146).
The image processing apparatus 203 stores the image effect correction parameter and the background color data of the document 4 in the memory 201 (step S147).

Next, the image processing apparatus 203 performs image effect processing (step S25) using the image effect correction parameters extracted in this way.
As described above, in order to effectively perform the image effect processing, it is necessary to change the processing contents depending on the background color.

  When the background color is white, such as a white board or notebook, luminance conversion as shown in FIG. When the background color is black, such as a blackboard, luminance conversion as shown in FIG. 29B is performed. When the background color is other than white or black, such as a magazine or pamphlet, the conversion as shown in FIG. 29C is performed. In FIGS. 29A, 29B, and 29C, the horizontal axis indicates the input value of the pixel value, and the vertical axis indicates the output value of the pixel value.

  When the background color is white, as shown in FIG. 29A, the inclination angle of the luminance conversion line is changed with the peak value as a boundary. The predetermined luminance value is set to 230, for example, and the input luminance peak value is raised to the luminance value 230. Then, the maximum value is raised to the maximum brightness. Therefore, the luminance conversion line is represented by two line segments as shown in FIG.

  When the background color is black, luminance conversion is performed so that the peak value becomes a certain luminance value (20) as shown in FIG. Also in this case, as shown in FIG. 29B, the luminance conversion line is represented by two line segments.

  When the background color is a color other than white or black, as shown in FIG. 29C, the minimum value or less and the maximum value or more are cut and expressed as one line segment as in the normal enlargement process. The luminance conversion line is set as follows.

  Note that a conversion table of background luminance (Y) and output (Y ′) may be set in advance and stored in the memory 201 as described above. According to the created conversion table, each output value is obtained from the input pixel value, and image effect processing is performed. In the image thus converted, bright pixels are brighter and dark pixels are darker, so that the luminance distribution is widened and the image is excellent in visibility.

  In addition, depending on the photographed image, the white balance of the digital camera may not be adjusted properly, and for example, the color may change to yellow or the like as a whole. This color change cannot be corrected only by performing image effect processing of the luminance histogram.

In this case, color adjustment is performed to obtain a preferable image. FIG. 30 shows an example of a luminance conversion graph for performing color adjustment.
In FIG. 30, the horizontal axis indicates the input value of the color difference, and the vertical axis indicates the output value of the color difference. In the color adjustment, luminance conversion is performed according to the luminance conversion graph shown in FIG. 30 so that the average values (Umean, Vmean) of the color differences (U, V) shown in FIG.

When the values of the color differences U and V are both 0, the color is an achromatic color, and color adjustment is performed so that the peak values (Upeak, Vpeak) are 0. That is, the color conversion line is represented by two line segments. The output value U ′ with respect to the input value U is given by the following equation (26).

The same applies to the color difference V.

  Next, as the image effect processing, when the background is not sufficiently whitened only by the above-described luminance extension, the background is whitened, and the color of the portion considered to be the background in the image is white (Y: 255, U: 0, V: 0) or a color close to that.

  FIG. 31A is a diagram illustrating a luminance value increase rate with respect to a luminance value of a certain pixel with a luminance value of a certain pixel as a reference (0). The horizontal axis represents the luminance value, and the vertical axis represents the luminance value raising rate. FIG. 31B is a diagram showing the color difference and the change rate of the color difference. The horizontal axis indicates the color difference, and the vertical axis indicates the change rate of the color difference.

  In the figure, C0 indicates a range from 0 in which the luminance and the color difference are increased by 100%, and C1 indicates a range in which the increase rate is varied according to the luminance value. As shown in FIGS. 31A and 31B, the color of a pixel whose luminance (Y) value is equal to or greater than a certain value (Yw) and whose color difference (U, V) is within a certain range (C0) is defined as a whitening range. The brightness value is raised and the color difference is set to zero. If the background is whitened as described above, the image effect processing becomes very effective when the background is not sufficiently whitened only by extending the luminance.

Based on such an idea, the image processing apparatus 203 executes image effect processing according to the flowchart shown in FIG.
The image processing apparatus 203 reads the stored image effect correction parameters from the memory 201 (step S151).

The image processing apparatus 203 determines whether the background is white (step S152).
When the background is determined to be white (Yes in step S152), the image processing apparatus 203 performs luminance conversion as illustrated in FIG. 29A so that the background is whiter and the visibility is improved. The brightness histogram is adjusted (step S153).

  If it is determined that the background is not white (No in step S152), the image processing apparatus 203 determines whether the background is black (step S154).

  When it is determined that the background is black (Yes in step S154), when the background is black, the image processing apparatus 203 performs luminance conversion as illustrated in FIG. S155).

If it is determined that the background is not black (No in step S154), the image processing apparatus 203 performs luminance conversion as shown in FIG. 29C and performs histogram adjustment according to the background color of the document 4 ( Step S156).
The image processing apparatus 203 performs color adjustment by performing conversion as shown in FIG. 30 on the image adjusted in this way (S157).

  The image processing apparatus 203 performs background whitening processing (S158) according to the flowchart shown in FIG.

That is, the image processing apparatus 203 initializes the count value j to 0 (step S161).
The image processing apparatus 203 initializes the count value i to 0 (step S162).
The image processing apparatus 203 determines whether or not the luminance (Y) of the pixel (i, j) of the input image is equal to or greater than a certain value (Yw) (step S163).

  When it is determined that the luminance (Y) is equal to or greater than a certain value (Yw) (Yes in step S163), the image processing apparatus 203 determines that the absolute values of the color differences U and V of the pixel (i, j) of the input image are predetermined values. It is determined whether it is less than C0 (step S164).

  When it is determined that the absolute values of the color differences U and V are less than the predetermined value C0 (Yes in step S164), the image processing apparatus 203 sets the luminance value (Y) to 255 and sets the color difference (u, v) to 0. In step S165, the value of the pixel (i, j) is rewritten, and the count value i is incremented (step S166).

  On the other hand, when it is determined that the absolute values of the color differences U and V are not less than the predetermined value C0 (No in step S164), the image processing apparatus 203 determines whether or not the absolute values of the color differences U and V are less than the predetermined value C1. Is determined (step S167).

  When it is determined that the absolute values of the color differences U and V are less than the predetermined value C1 (Yes in Step S167), the image processing apparatus 203 sets the luminance value Y = Y + a * (255−Y) (Step S168), The value i, j) is rewritten, and the count value i is incremented (step S166).

  If it is determined that the absolute values of the color differences U and V are not less than the predetermined value C1 (No in step S167), the image processing apparatus 203 increments the count value i without changing the luminance value (step S166).

The image processing apparatus 203 compares the count value i with the maximum value imax and determines whether or not the count value i has reached the maximum value imax (step S169).
When it is determined that the count value i has not reached the maximum value imax (No in step S169), the image processing apparatus 203 executes the processes in steps S163 to S168 again.

  When it is determined that the count value i has reached the maximum value imax by repeatedly executing the processes in steps S163 to S168 (Yes in step S169), the image processing apparatus 203 increments the count value j (step S170). .

  The image processing apparatus 203 compares the count value j with the maximum value jmax and determines whether the count value j has reached the maximum value jmax (step S171).

  When it is determined that the count value j has not reached the maximum value jmax (No in step S171), the image processing apparatus 203 executes the processes in steps S162 to S170 again.

  When it is determined that the count value j has reached the maximum value jmax by repeatedly executing the processes in steps S162 to S170 (Yes in step S171), the image processing apparatus 203 ends the background whitening process.

(4) Adjustment of Image Conversion Next, adjustment (step S33 in FIG. 5) performed on an image that has been once converted will be described.
In the case where some errors are included in the coordinates of the extracted quadrangular vertices, as shown in FIGS. 34 (a) and 34 (b), the result of projection with the obtained affine parameters may not be preferable. is there. For this reason, the captured image projection apparatus of the present embodiment is configured so that the user can adjust the projection transformation.

  When the user operates each key of the operation unit 204, the operation unit 204 sends this operation information to the CPU 206 as instruction information in response to the user's operation. The CPU 206 determines this operation information and controls the image processing device 203 according to the determination result.

  As shown in FIG. 35, when the interpolation pixel Q (u ′, v ′) of the corrected image is obtained, the inverse transformation Ai is performed on the interpolation pixel Q (u ′, v ′) to obtain the interpolation pixel Q. A corrected image P (u, v) corresponding to (u ′, v ′) is obtained, and further the inverse transformation is performed on the corrected image P (u, v) to obtain p (x, y) of the original image. Usually, pixel interpolation is performed on the image p (x, y). When performing two-stage image conversion, such as projection conversion and enlargement conversion, a conversion matrix obtained by combining the two conversions is obtained, and the original image is converted at a time. In this case, the image can be obtained at a higher speed than when the conversion is performed twice, and image degradation is less than that in the method of creating the image by performing the inverse conversion twice.

The rotation inverse transformation matrix Ar in the case of acquiring the image before conversion from the image after conversion obtained by rotating the image before conversion by the angle θ about the X axis and Y axis is expressed by the following equation (27). expressed.

An enlargement matrix Asc in the case of obtaining an image before conversion from an image after conversion obtained by enlarging the image before conversion by Sc magnification around the X and Y axes is expressed by the following equation (28). The

  Note that once an image is enlarged, a rounding error may be processed by adjusting or calculating an affine parameter. For this reason, before enlarging the image, it is necessary to restore the original affine parameters of the same magnification.

The movement matrix As for obtaining the image before conversion from the image after conversion obtained by moving the image before conversion in the X and Y directions by Tx and Ty, respectively, is expressed by the following equation (29). Is done.

The projection effect matrix Ap in the case of acquiring the image before conversion from the image after conversion obtained by inclining the image before correction in the X and Y directions by α and β, respectively, is given by expressed.

When performing two-stage inverse transformation, the inverse transformation matrix A is expressed by the following equation (31).

Image conversion adjustment processing executed based on this concept will be described based on the flowcharts shown in FIGS.
The CPU 206 determines whether or not the enlargement factor Zoom is 1 (step 181). Since the enlargement factor Zoom is initialized to 1 in advance, the CPU 206 first determines that the enlargement factor Zoom is 1 (Yes in step S181). If it is determined that the zoom factor Zoom is 1, the CPU 206 determines whether any one of the upper enlargement key 211, the lower enlargement key 212, the left enlargement key 213, and the right enlargement key 214 is pressed as a projection conversion key. (Step 182).

  If it is determined that the projection conversion key has been pressed (Yes in step S182), the CPU 206 determines the type of the operated projection conversion key.

  If it is determined that the pressed projective transformation key is the right enlargement key 214, the CPU 206 substitutes α = 0.1 and β = 0 into the projection effect matrix Ap shown in Equation 26, respectively, and the inverse transformation matrix Ai = Ap. Is acquired (step S183).

  When determining that the pressed projective transformation key is the left enlargement key 213, the CPU 206 assigns α = −0.1 and β = 0 to the projective effect matrix Ap shown in Equation 30, respectively, and the inverse transformation matrix Ai = Ap is acquired (step S184).

  If the CPU 206 determines that the pressed projective transformation key is the upper enlargement key 211, the CPU 206 substitutes α = 0 and β = 0.1 for the projection effect matrix Ap shown in Equation 30, respectively, and the inverse transformation matrix Ai = Ap. Is acquired (step S185).

  If the CPU 206 determines that the pressed projective transformation key is the lower magnification key 212, the CPU 206 substitutes α = 0 and β = −0.1 for the projection effect matrix Ap shown in Equation 30, respectively, and the inverse transformation matrix Ai = Ap is acquired (step S186).

When it is determined that the projection conversion key is not pressed (No in Step 182), the CPU 206 determines whether or not the rotation key is pressed (Step S187).
When it is determined that the rotation key has been pressed (Yes in step S187), the CPU 206 determines the type of the pressed rotation key.

  If it is determined that the pressed rotation key is the right rotation key 215, the CPU 206 substitutes θ = −1 for the rotation inverse transformation matrix Ar shown in Equation 27 to obtain the inverse transformation matrix Ai = Ar ( Step 188).

  If it is determined that the pressed rotation key is the left rotation key 216, the CPU 206 substitutes θ = 1 for the rotation inverse transformation matrix Ar shown in Equation 27 to obtain the inverse transformation matrix Ai = Ar (step). 189).

  If it is determined that the rotation correction key has not been pressed (No in step S187), the CPU 206 can restore the current affine parameters so that the original affine parameters can be restored before the enlargement process is performed. Are saved in the matrix Af (step S190).

  On the other hand, when it is determined that the zoom factor Zoom is not 1 (No in step S181), the CPU 206 determines whether or not the cursor key is pressed (step S191).

  When it is determined that the cursor key has been pressed (Yes in step S191), the CPU 206 determines the type of the pressed cursor key.

  When determining that the cursor key is the right movement key (the enlargement key 217 and the right enlargement key 214), the CPU 206 adds the respective movement amounts Tx = 64, Ty of the X axis and the Y axis to the movement matrix As shown in Equation 29. = 0 is substituted to obtain the inverse transformation matrix Ai = As (step S192).

  When determining that the cursor key is the left movement key (the enlargement key 217 and the left enlargement key 213), the CPU 206 adds the respective movement amounts Tx = −64, X axis and Y axis to the movement matrix As shown in Equation 29. Substituting Ty = 0 to obtain the inverse transformation matrix Ai = As (step S193).

  When determining that the cursor key is the upward movement key (enlargement key 217 and upward enlargement key 211), the CPU 206 adds the movement amounts Tx = 0, Ty of the X axis and the Y axis to the movement matrix As shown in Equation 29, respectively. = 64 is substituted to obtain the inverse transformation matrix Ai = As (step S194).

  If it is determined that the cursor key is the downward movement key (enlargement key 217 and downward enlargement key 212), the CPU 206 adds the movement amounts Tx = 0, Ty of the X axis and the Y axis to the movement matrix As shown in Equation 29. = -64 is substituted to obtain an inverse transformation matrix Ai = As (step S195).

  On the other hand, when it is determined that the cursor key is not pressed (No in step S191), or when it is determined that the rotation correction key is not pressed and the current affine parameters are saved in the matrix Af (step S160). The CPU 206 determines whether or not the enlargement key 217 or the reduction key 218 has been pressed (step S196).

If it is determined that the enlargement key 217 or the reduction key 218 is not pressed (No in step S196), the CPU 206 ends the projective conversion adjustment processing.
On the other hand, when it is determined that the enlargement key 217 or the reduction key 218 is pressed (Yes in step S196), the CPU 206 determines the type of the pressed key.

  If it is determined that the pressed key is the enlargement key 217, the CPU 206 obtains a new enlargement ratio Zoom as an enlargement ratio Zoom = Zoom * Ratio (Ratio is, for example, 2) (step S197), and the number By substituting Sc = Ratio into Sc of the expanded matrix Asc shown in FIG. 28, an inverse transformation matrix Ai = Asc is obtained (step S198).

If it is determined that the pressed key is the reduction key 218, the CPU 206 acquires a new enlargement ratio Zoom as an enlargement ratio Zoom = Zoom / Ratio (step S199).
In the case of the reduction key 218, the CPU 206 determines whether or not the enlargement factor Zoom exceeds 1 (step S200).

  If it is determined that the zoom rate Zoom exceeds 1 (Zoom> 1) (Yes in step S200), the CPU 206 substitutes Sc = 1 / Ratio for Sc of the zoom matrix Asc shown in Equation 28, and vice versa. A transformation matrix Ai = Asc is acquired (step S201).

  Next, when the inverse transformation matrix Ai is set (steps S183 to S186, S188, S189, S192 to 195, S198, S201), the CPU 206 obtains the inverse transformation matrix A according to Equation 31 (step S202).

  The CPU 206 supplies the obtained conversion matrix A to the image processing apparatus 203 and controls the image processing apparatus 203 to perform image conversion based on the inverse conversion matrix A. The image processing apparatus 203 performs image conversion by affine transformation based on the supplied inverse transformation matrix A (step S203).

On the other hand, when it is determined that the zoom factor Zoom does not exceed 1 (Zoom ≦ 1) (No in step S200), the CPU 206 sets the zoom factor Zoom = 1 (step S204).
The CPU 206 returns the original inverse transformation matrix A to A = Af (the affine parameter before enlargement saved in step S190) (step S205).

  Similarly, the CPU 206 supplies the obtained inverse transformation matrix A to the image processing device 203 to control the image processing device 203 so as to perform image transformation, and the image processing device 203 supplies the supplied inverse transformation matrix. Based on A, image conversion by affine transformation is performed (step S203).

  The image created by enlarging or the like is a part of the document 4. The document 4 described in an enlarged manner is often a figure, a graph, a photograph, and the like. When image effect processing such as contrast enlargement is performed using only the enlarged portion, the expected image quality is obtained. It may not be obtained. This is because when the document 4 is a figure or a photograph, the background color ratio is small.

  Therefore, when enlarging, reducing, or moving an image, it is preferable to use the image effect correction parameters before performing such image conversion as they are. For this reason, in step S33 (image conversion) in FIG. 5, when the image conversion is enlargement, reduction, movement, or the like of the image, the CPU 206 skips the extraction of the image effect correction parameters, and The processing device 203 performs image effect processing using the image effect correction parameters before image conversion.

  On the other hand, when the image conversion is projective conversion or rotation, the operation is performed even if the object to be photographed (original 4) is not cut out correctly, so that a new image adjustment parameter is newly extracted using the newly adjusted image. Is preferably performed. For this reason, when the image conversion is projective conversion or rotation, the CPU 206 shifts to image effect correction parameter extraction processing (step S25 in FIG. 5).

  As described above, according to the first embodiment, the image processing apparatus 203 acquires the outline from the image of the document 4, acquires the quadrangle shape of the document 4, and sets the projection parameters from the vertex positions of the rectangle. In search, the image of the document 4 is projectively transformed.

  Therefore, if the document 4 that is the subject is a quadrangle, an image of only the document 4 is automatically cut out and converted into an image viewed from the front. In addition, the image of the document 4 can be projected onto the screen 3 in the correct orientation regardless of the orientation when the document 4 is placed.

  In addition, since the optimum image effect processing is performed on the cut out image, an image with sufficiently high legibility can be obtained without illuminating the document 4.

  In addition, when image rotation and projection effect processing correction is performed on an image that has been cut out once, it can be performed using the image effect correction parameters obtained in the first cut-out, making it easier to view images that are easier to view. Even if the document 4 is warped or bent, the image processing can be adjusted by a simple operation.

  In addition, it is possible to perform enlargement, reduction, etc. on the cut-out image, and not to create the image from the cut-out image to obtain the image, but to perform image conversion for cut-out and enlargement at a time. Fewer images can be created.

  When image effect processing is performed on the enlarged image of the clipped image, the image effect correction parameters before enlargement are stored, and the image effect processing is performed using the stored parameters. An image can be obtained.

(Embodiment 1)
The captured image projection apparatus according to the second embodiment includes a computer, and the computer performs image processing.

FIG. 38 shows the configuration of the captured image projection apparatus according to the second embodiment.
A captured image projection apparatus according to the second embodiment includes a computer 5 in addition to the document camera 1 and the projector 2.

  The document camera 1 and the computer 5 are connected via a communication cable 31 such as a USB (Universal Serial Bus), and the computer 5 and the projector 2 are connected via a video image cable 32 such as an RGB cable.

  As shown in FIG. 39, the document camera 1 according to the second embodiment includes an image compression device 207 and an interface device 208 in the data processing unit 22 instead of the image processing device 203 according to the first embodiment.

  The image compression device 207 compresses image data in order to reduce the amount of data transmitted to the computer 5. The image compression device 207 compresses a still image using a technique such as JPEG (Joint Photographic Expert Group) standard.

  The interface device 208 is for transmitting the compressed image data to the computer 5 and receiving a shooting command from the computer 5.

  The CPU 206 has a function of initializing camera setting parameters such as the focus, exposure, and white balance of the optical lens device 101 to the moving image mode. As a result, the scene captured by the camera unit 11 is condensed on the image sensor 102 via the optical lens device 101, and the image sensor 102 uses a digital image with a low resolution for moving image shooting from the collected image. Image data is created and sent to the memory 201 at, for example, about 30 sheets per second.

The computer 5 sends a shooting command to the document camera 1 to control the document camera 1, receives image data, and transmits image data subjected to image processing to the projector 2.
The computer 5 includes an interface device 231, a display device 232, an image processing device 233, an operation unit 234, an HDD (Hard Disk Drive) 235, a CPU 236, a ROM (Read Only Memory) 237, and a RAM (Random Access). Memory) 238.

The interface device 231 is for receiving compressed image data and transmitting a shooting command and the like.
Similar to the display device 202, the display device 232 is for displaying an image sent to the projector 2.

  The image processing device 233 is for performing image processing such as image distortion correction and image effect on the received image data, and has the same function as the image processing device 203 of the first embodiment. In addition, the image processing device 233 performs compression decoding on the compressed image to generate uncompressed data.

  The image processing apparatus 233 may be configured by hardware or software. Since the function can be updated by upgrading the software, the image processing device 233 is preferably configured by software.

  Since the computer 5 has the function of the image processing device 203 of the document camera 1, it is not necessary to install image processing hardware in the camera unit 11, and a commercially available standard digital camera can be used. Good.

The operation unit 234 includes switches and keys for the user to input data and commands.
The HDD 235 is for storing data and the like. The HDD 235 also stores data of software for document processing installed in advance.

The CPU 236 controls each part of the computer 5 and controls the document camera 1 by transmitting to the document camera 1 a photographing command for instructing photographing of a high-resolution still image.
The ROM 237 stores basic program codes executed by the CPU 236.
The RAM 238 stores data necessary for the CPU 236 to execute.

Next, the operation of the captured image projection apparatus according to the second embodiment will be described based on the flowchart shown in FIG.
The CPU 206 of the document camera 1 initializes the interface memory 208, the image compression device 207, and the work memory in the memory 201 (step S211).
The CPU 206 initializes the focus, exposure, white balance and camera setting parameters of the optical lens device 101 to the moving image mode (step 212).

The CPU 206 checks the interface device 208 and determines whether or not a shooting command from the computer 5 has been received (step S213).
If it is determined that the shooting command has not been received (No in step S213), the CPU 206 determines whether the image adjustment key of the operation unit 204 has been pressed based on the operation information from the operation unit 204 (step S214). ).

  If it is determined that the image adjustment key has been pressed (Yes in step S214), the CPU 206 transmits the type information of the pressed image adjustment key to the computer 5 via the interface device 208 (step S215), and the image sensor 102. A low-resolution image is read from (step S216).

  On the other hand, if it is determined that the image adjustment key has not been pressed (No in step S214), the CPU 206 reads a low-resolution image from the image sensor 102 without transmitting type information (step S216).

The image compression device 207 compresses the image data read by the CPU 206 under the control of the CPU 206 (step S217).
The CPU 206 transmits the low-resolution image data compressed by the image compression device 207 to the computer 5 through the interface device 208 at a speed of, for example, about 30 frames per second (step S218).

  Then, the CPU 206 and the image compression device 207 execute the processes of steps S213 to S218. In this manner, the document camera 1 periodically transmits a low resolution image to the computer 5 unless it receives a shooting command.

  If it is determined that a shooting command has been received (Yes in step S213), the CPU 206 sets the shooting mode of the image sensor 102 and the optical lens device 101 to a high-resolution still image mode (step S219).

The CPU 206 controls the camera unit 11 so as to capture a high-resolution still image (step S220).
The CPU 206 sends the image data captured by the camera unit 11 to the image compression device 207, and the image compression device 207 compresses the received image data (step S221).

The image compression device 207 transmits the compressed high-resolution still image data to the computer 5 via the interface device 208 (step S222).
The CPU 206 sets the low-resolution moving image mode again (step S223), executes the processing of steps S213 to S218 until receiving a shooting command, and transmits the acquired low-resolution image to the computer 5 via the interface device 208. Send to.

When the document processing software installed in the HDD 235 is activated, the computer 5 executes processing of the document camera function according to the flowcharts shown in FIGS.
The CPU 236 initializes communication and the like (step S231).
The CPU 236 determines whether or not data has been received and determines the content of the received data (step S232).

If it is determined that the received data is a low-resolution image, the image processing device 233 converts the compressed data into uncompressed data by performing compression decoding on the compressed image (step S233).
As in the first embodiment, the CPU 236 calculates the image change amount MD according to Equation 1 (step S234).

The CPU 236 determines whether the shooting mode is the moving image mode or the still image mode (step S235).
In the initial state, since the shooting mode is set to the moving image mode, the CPU 236 determines that the shooting mode is the moving image mode (Yes in step S235). In this case, the CPU 236 performs drawing of the received low resolution image (step S236). The image processing device 233 outputs the rendered low-resolution image data to the projector 2, and the projector 2 projects this image onto the screen 3.

  The CPU 236 obtains the image change amount MD with respect to the previously captured image according to the equation 1, compares the obtained image change amount MD with a preset threshold Thresh1, and determines whether the image has a motion based on the comparison result. Is determined (step S237).

If it is determined that there is a motion in the image (MD ≧ Thresh1) (Yes in step S237), the CPU 236 clears the stationary time Ptime (step S245) and continues the moving image mode.
On the other hand, when it is determined that there is no motion in the image (Thresh1> MD) (No in step S237), the CPU 236 adds 1 to the stationary time Ptime (step S238).
The CPU 236 determines whether or not the movement has stopped and a predetermined time “HoldTime” has elapsed (step S239).
If it is determined that the predetermined time HoldTime has not elapsed (Ptime <HoldTime) (No in step S239), the CPU 236 maintains the moving image mode and waits until the next data is received (step S232).

  When it is determined that the predetermined time HoldTime has elapsed (Ptime ≧ HoldTime) (Yes in Step S239), the CPU 236 sets the shooting mode to the still image mode (Step S240).

  The CPU 236 transmits a shooting command for instructing shooting of a high-resolution still image (step S241), and waits until still image data is received from the document camera 1 (step S232).

  When data is received and the received data is determined to be a low-resolution image (step S232), the CPU 236 decodes the low-resolution image (step S233) and calculates an image change amount MD (step S234). It is determined whether or not the shooting mode is the moving image mode (step S235).

  When it is determined that the still image mode is selected (No in step S235), the CPU 236 compares the image change amount MD with the two threshold values Thresh2 to determine whether or not there is a motion as in the first embodiment. (Steps S242 and S243).

  If MD <Thresh1 (No in step S242, Yes in step S243), the CPU 236 transmits a shooting command (step S241).

  If Thresh1 ≦ MD ≦ Thresh2 (No in step S242, No in step S243), CPU 236 leaves the still image currently displayed as it is and the shooting mode remains in the moving image mode (step S244). Then, the CPU 236 sets 0 as the stationary time Ptime (step S245). Then, CPU 236 waits until the next data is received (step S232).

  If Thresh2 <MD (Yes in step S242), the CPU 236 waits until the next data is received in the still image mode (step S232).

When it is determined that the received data is a high-resolution image (step S232), the CPU 236 decodes the high-resolution still image (step S246 in FIG. 42).
As in the first embodiment, the image processing device 233 extracts projection parameters and performs image effect processing (steps S247 to S250).

  The image processing device 233 draws an image and outputs it to the projector 2 (step S251), and the CPU 236 waits until the next data is received (step S232).

  If it is determined that the received data is related to key operation information, the CPU 236 determines whether or not the shooting mode is the still image mode. If the received data is not the still image mode (No in step S252), the CPU 236 determines the next data. Wait until it is received (step S232).

  If it is determined that the still image mode is selected (Yes in step S252), the CPU 236 performs adjustment and image conversion in the same manner as in the first embodiment (step S253), and performs image effect processing according to the content of the image conversion (step S254, step S254). S249, S250). Then, the image processing device 233 draws an image and outputs it to the projector 2 (step S251), and the CPU 236 waits until the next data is received (step S232).

  As described above, according to the second embodiment, the computer 5 is provided, and the computer 5 performs image processing. Accordingly, since it is not necessary to mount image processing hardware in the camera unit 11, a commercially available standard digital camera can be used, and a captured image projection apparatus can be configured. For this reason, the system can be configured at low cost.

In carrying out the present invention, various forms are conceivable and the present invention is not limited to the above embodiment.
For example, in the first embodiment, when extracting the image effect correction parameters, the image clipping process is performed on the image obtained by converting the image of the document 4 using the projection parameters. However, even if the image of the document 4 is distorted, the image effect correction parameters can be extracted from the actual document portion without performing projective conversion.

  In the above-described embodiment, the shape of the document 4 is a quadrangle, and each process is performed by obtaining a rectangle from the outline of the document 4. However, although not realistic, the shape of the document 4 is not limited to a quadrangle, and may be a pentagon or the like.

In the above-described embodiment, the program has been described as being stored in advance in a memory or the like. However, a program for causing a computer to operate as the whole or a part of the apparatus or to execute the above-described processing is a flexible disk, a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk), The information may be stored in a computer-readable recording medium such as an MO (Magneto Optical disk) and distributed, installed in another computer, operated as the above-described means, or the above-described steps may be executed.
Furthermore, the program may be stored in a disk device or the like included in a server device on the Internet, and may be downloaded onto a computer by being superimposed on a carrier wave, for example.

It is a block diagram which shows the structure of the picked-up image projection apparatus which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the document camera shown in FIG. It is explanatory drawing of the function of the image processing apparatus shown in FIG. It is explanatory drawing of each key with which the operation part shown in FIG. 2 is provided. It is a flowchart which shows the content of the basic projection process which the picked-up image projection apparatus shown in FIG. 1 performs. It is explanatory drawing of the content of the clipping process of the image processing apparatus shown in FIG. It is explanatory drawing of the basic view of extraction of a projection parameter, and an affine transformation. It is a flowchart which shows the content of the projection parameter extraction process which the image processing apparatus shown in FIG. 2 performs. It is a flowchart which shows the content of the square outline extraction process which the image processing apparatus shown in FIG. 2 performs. It is explanatory drawing of a reduced luminance image and an edge image. It is explanatory drawing of the function of Roberts filter. It is an image figure for demonstrating the principle of radon conversion. It is explanatory drawing of the operation | movement which acquires the data of a polar coordinate system by radon-transforming the straight line of a X, Y coordinate system. It is a flowchart which shows the content of the peak point detection process from the data of the polar coordinate system which the image processing apparatus shown in FIG. 2 performs. It is explanatory drawing which shows the view which detects a square from the straight line which detected and extracted the peak point. It is a flowchart which shows the content of the square detection process which the image processing apparatus shown in FIG. 2 performs. It is a block diagram which shows a structure in case the document camera shown in FIG. 1 is provided with a distance sensor. It is a flowchart which shows the content of the distance measurement process performed when the optical lens apparatus shown in FIG. 2 or FIG. 17 has an active autofocus lens. It is a flowchart which shows the content of the distance measurement process performed when the optical lens apparatus shown in FIG. 2 or FIG. 17 has a passive autofocus lens. It is a flowchart which shows the content of the setting process of the paper size which the image processing apparatus shown in FIG. 2 or FIG. 17 receives the input from a user. FIG. 6 is a diagram illustrating an example of a screen presented to a user during a paper size setting process. It is a flowchart which shows the content of the selection process of the detected square which the image processing apparatus shown in FIG. 2 performs. It is a flowchart which shows the content of the process which calculates | requires an affine parameter from the square vertex performed by the image processing apparatus shown in FIG. It is explanatory drawing of the inverse transformation for obtaining the original image from the image after projective transformation. It is a flowchart which shows the content of the image conversion process by the affine transformation which the image processing apparatus shown in FIG. 2 performs. It is explanatory drawing which shows the example of an image cut out by the image processing apparatus shown in FIG. FIG. 27A is an explanatory diagram showing an example of a histogram, FIG. 27A shows a luminance histogram, and FIG. 27B shows a color difference histogram. 3 is a flowchart showing the contents of image effect correction parameter extraction processing executed by the image processing apparatus shown in FIG. 2. 29A and 29B are explanatory diagrams showing image effect processing. FIG. 29A shows image effect processing when the background color is white, and FIG. 29B shows image effect processing when the background is black. 29 (c) shows image effect processing when the background is other than white or black. It is explanatory drawing which shows an example of the luminance conversion graph for performing color adjustment. It is a figure for demonstrating whitening of a background. It is a flowchart which shows the content of the image effect process which the image processing apparatus shown in FIG. 2 performs. It is a flowchart which shows the content of the background whitening process which the image processing apparatus shown in FIG. 2 performs. It is explanatory drawing which shows the example which could not correct | amend the distortion of an image by projective transformation. It is a figure for demonstrating the correspondence with the original image, a projection conversion image, and the enlarged projection conversion image. 3 is a flowchart (part 1) showing details of correction adjustment and image conversion processing executed by the image processing apparatus shown in FIG. 2; FIG. 3 is a flowchart (part 2) showing the contents of correction adjustment and image conversion processing executed by the image processing apparatus shown in FIG. 2. FIG. It is a block diagram which shows the structure of the picked-up image projection apparatus which concerns on Embodiment 2 of this invention. It is a block diagram which shows the structure of the document camera shown in FIG. It is a flowchart which shows the content of the basic process which the document camera shown in FIG. 39 performs. FIG. 39 is a flowchart (No. 1) showing the contents of a document basic process executed by the computer shown in FIG. 38. FIG. FIG. 39 is a flowchart (No. 2) showing the contents of a document basic process executed by the computer shown in FIG. 38. FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Document camera, 2 ... Projector, 3 ... Screen, 4 ... Original, 11 ... Camera part, 201 ... Memory, 202, 232 ... Display apparatus, 203, 233 ... Image processing device, 204, 234 ... Operation unit, 206,236 ... CPU

Claims (8)

In a photographic image projection device that projects an image of a photographed document on a screen,
An image processing unit that performs image processing on any one of a plurality of shapes of the document image so as to correct distortion of the document image obtained by shooting;
A projection unit that projects the image processed by the image processing unit onto the screen;
The image processing unit
A shape acquisition unit that acquires the outline of the original image candidate from the original image and acquires the shape surrounded by the acquired outline and the vertex position thereof;
An area acquisition unit for obtaining an area of the shape acquired by the shape acquisition unit;
The area obtained by the area acquisition unit is compared with the area of the document size designated in advance, and the shape having the area closest to the area of the designated document size is obtained among the areas obtained by the area acquisition unit. A shape selection unit for selecting the shape of the original image;
The shape of the original image selected by the shape selection unit is associated with the actual original shape, and the relationship between the original image and the actual original is determined from the vertex position of the original image acquired by the shape acquisition unit. A projection parameter acquisition unit for obtaining a projection parameter to be shown;
An image conversion unit that performs image conversion of the document image using the projection parameter obtained by the projection parameter acquisition unit,
A photographed image projection apparatus characterized by the above. The shape acquisition unit
An edge image detector for detecting an edge image of the document image;
A straight line detection unit that detects straight lines that are candidates for the outline of the document from the edge image detected by the edge image detection unit;
Combining the straight lines detected by the straight line detection unit to obtain the contour of the document;
The photographed image projection apparatus according to claim 1. The area acquisition unit obtains the area of the shape surrounded by the outline acquired by the shape acquisition unit based on the size of the document image and the vertex position acquired by the shape acquisition unit.
The photographed image projecting apparatus according to claim 1, wherein: A lens for condensing light, an image forming unit for forming an image by the light collected by the lens, and a lens control unit for controlling the position of the lens so that the original image is formed on the image forming unit And a camera having
The area acquisition unit
Based on the distance between the lens and the image forming unit when the document image is formed on the image forming unit by the control of the lens control unit, and the distance between the lens and the document. Obtaining the size of the original image acquired by the shape acquisition unit.
The photographed image projector according to claim 3. The lens control unit
A light output unit that outputs light for distance measurement;
A distance sensor that measures a distance between the lens and the document based on light reflected from the document by the light output from the light output unit;
The area acquisition unit is configured to control a distance between the lens and the document measured by the distance sensor when the document image is formed on the image forming unit under the control of the lens control unit. A distance between the lens and the imaging unit is determined based on a focal length;
The photographed image projection apparatus according to claim 4, wherein: The lens control unit controls the position of the lens so that the distance between the lens and the image forming unit is preset and the contrast of the document image formed on the image forming unit is maximized. Passive type
The area acquisition unit is based on the distance between the lens and the imaging unit output from the lens control unit when the contrast of the document image is maximized, and the focal length of the lens. Obtaining a distance between the lens and the document;
The photographed image projection apparatus according to claim 4, wherein: An image processing method of a photographic image projection apparatus for projecting a photographed document image on a screen,
Obtaining a contour of the document image candidate from the document image having a plurality of shapes, obtaining the shape surrounded by the obtained contour and its vertex position;
Obtaining an area of the acquired shape;
By comparing the area of pre-specified document size and the obtained area of the obtained area, selecting the shape having the closest area to the area of the designated document size as the shape of the original image Steps,
Associating the shape of the selected document image with the shape of the actual document, and obtaining a projection parameter indicating the relationship between the document image and the actual document from the vertex position of the acquired document image;
Performing image conversion of the document image using the obtained projection parameter,
An image processing method of a photographed image projecting apparatus characterized by the above. On the computer,
A procedure for acquiring the contour of the document image candidate from a document image having a plurality of shapes obtained by photographing, and acquiring the shape surrounded by the acquired contour and its vertex position;
A procedure for obtaining an area of the acquired shape;
By comparing the area of pre-specified document size and the obtained area of the obtained area, selecting the shape having the closest area to the area of the designated document size as the shape of the original image procedure,
A procedure for associating the shape of the selected original image with the actual original shape and obtaining a projection parameter indicating the relationship between the original image and the actual original from the vertex position of the acquired original image;
A procedure for performing image conversion of the document image using the obtained projection parameter;
A program for running