JP2013187764A - Image processing device, image processing method and projector - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique that more accurately detects a screen frame and performs trapezoidal distortion correction.SOLUTION: An image processing device performs the steps of: detecting a pattern on the basis of a pattern image in which a measurement pattern PT1 having a predetermined shape projected on a screen 30 from a liquid crystal panel 130 is imaged; detecting respective two screen frame straight lines in a crossing relation at each corner of the screen 30 on the basis of a frame image in which at least a part of a screen frame SCF is imaged; subsequently calculating a coordinate conversion coefficient for converting camera coordinates in the image imaged to panel coordinates in the liquid crystal panel 130 on the basis of a panel coordinate value of a pattern in the liquid crystal panel 130 and a camera coordinate value of the pattern detected from the pattern image; and obtaining vertices of the screen frame SCF in the panel coordinates on the basis of the screen frame straight lines detected and the coordinate conversion coefficient and calculating a correction value for performing trapezoidal distortion correction. A projector performs the trapezoidal distortion correction.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a projector.

  When an image is displayed on a screen using a projector, an image displayed on the screen (hereinafter referred to as a “display image”) may be distorted in a trapezoidal shape depending on the relative positional relationship between the projector and the screen. Regarding such a problem, for example, Patent Document 1 discloses a technique in which a screen is imaged by an imaging device provided in a projector, a screen frame is detected from the captured image, and a trapezoidal distortion is corrected based on the detected shape of the screen frame. Have been described.

  However, in the technique of Patent Document 1, the screen frame is detected as a total of four straight lines, one for each of the upper and lower and left and right frame sides. In general, since there is distortion in an imaging camera lens provided in a projector, it is difficult to accurately detect a screen frame as four straight lines.

JP 2010-050542 A JP 2010-050540 A JP 2008-212355 A JP 2006-060447 A

  In view of the above-described problems, the problem to be solved by the present invention is to provide a technique for more accurately detecting a screen frame and correcting keystone distortion.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms or application examples.

Application Example 1 An image processing apparatus that detects a pattern based on a pattern image in which a pattern having a predetermined shape projected on a projection surface is projected from a projection unit having a light modulation device. And a straight line detection unit that detects two straight lines that intersect each corner of the projection surface based on an image of a frame in which at least a part of the frame of the projection surface is captured, Based on the coordinates of the pattern in the light modulation device and the coordinates of the pattern detected from the pattern image, a coordinate conversion coefficient for converting the coordinates in the captured image into the coordinates in the light modulation device is calculated. A vertex of the frame is obtained based on the coordinate transformation coefficient calculation unit, the two detected straight lines for each corner and the coordinate transformation coefficient, and trapezoidal distortion correction is performed based on the vertex. And a correction value calculation unit for calculating a correction value of the fit, the image processing apparatus.

  With such a configuration, two straight lines that intersect each other at each corner of the projection surface are detected, so that in the captured image, the projection surface that is susceptible to the lens distortion of the camera is detected. The straight line at the corner can be detected more accurately. Therefore, the correction value can be calculated based on such a straight line and the coordinate conversion coefficient, and the trapezoidal distortion correction can be performed more accurately.

Application Example 2 In the image processing apparatus according to Application Example 1, the pattern is positioned at each corner of the light modulation device, and the coordinate conversion coefficient calculation unit is configured to output the coordinates at each corner of the projection surface. The conversion value is calculated, and the correction value calculation unit is based on the detected two straight lines that intersect each corner of the projection surface and the coordinate conversion coefficient corresponding to the corner. An image processing apparatus for obtaining a vertex.

  In such a configuration, since the coordinate conversion coefficient is calculated for each corner of the projection surface, the influence of the lens distortion of the camera and the distortion of the projection lens is minimized, and the distortion of the lens of the camera is reduced. It is possible to accurately detect the straight line at the corner of the projection surface that is easily affected, and calculate the intersection of the straight line in the light modulation device to obtain the vertex. Therefore, the trapezoidal distortion correction can be performed more accurately.

[Application Example 3] In the image processing apparatus according to Application Example 1 or Application Example 2, the straight line detection unit is configured so that one of the two straight lines on one frame side of the projection surface has one straight line. When not detected, the image processing apparatus estimates the undetected straight line based on another straight line.

  With such a configuration, two straight lines are detected for each corner of the projection surface, so that two straight lines are detected for one side of the frame of the projection surface. For this reason, a straight line that is not detected can be supplemented with another straight line that is estimated to be on one frame side of the projection surface. Therefore, trapezoidal distortion correction can be performed even if a straight line that is not detected exists.

[Application Example 4] The image processing apparatus according to any one of Application Examples 1 to 3, wherein the straight line detection unit is estimated to be on one frame side of the projection surface. If at least one of the relationship between the positions of two detected straight lines and the relationship of the inclination does not satisfy a predetermined relationship, one of the two detected straight lines is discarded and the other straight line is discarded. An image processing apparatus that estimates one straight line based on the above.

  With such a configuration, an erroneously detected straight line can be determined based on the relationship between the positions of two straight lines estimated to be on the frame side of one projection surface and the relationship between the inclinations. . Therefore, even if an erroneously detected straight line exists, the straight line can be discarded and complemented with another straight line, so that two straight lines are calculated for each frame side of the projection surface. , Trapezoidal distortion correction can be performed.

Application Example 5 In the image processing apparatus according to Application Example 4, in the case where the positional relationship does not satisfy the predetermined relationship, the straight line detection unit includes the detected straight line out of the two detected straight lines. An image processing apparatus that discards a straight line that is far from a pattern and estimates the one straight line based on a straight line that is close to the pattern.

  With such a configuration, when the positional relationship does not satisfy the predetermined relationship, a straight line that is far from the pattern is determined to be a falsely detected straight line and discarded, and a straight line that is closer to the pattern It can be complemented with. Therefore, the trapezoidal distortion correction can be performed more accurately.

[Application Example 6] In the image processing apparatus according to Application Example 4 or Application Example 5, the straight line detection unit may detect the two detected lines when the relationship between the tilts does not satisfy a predetermined relationship. Of the straight lines, the straight line having a different inclination from the frame side estimated to have the two detected straight lines is discarded, and the one straight line is estimated based on the straight line having a closer inclination to the frame side. , Image processing device.

  With such a configuration, when the relationship between the inclinations does not satisfy the predetermined relationship, a straight line with a different inclination from the frame side is determined to be a falsely detected straight line and discarded. It can be complemented with a closer straight line. Therefore, the trapezoidal distortion correction can be performed more accurately.

[Application Example 7] The image processing apparatus according to any one of Application Examples 1 to 6, wherein the correction value calculation unit is configured such that the straight line detection unit is detected for each corner 2 An image processing apparatus for obtaining a vertex of the frame after converting a straight line of a book into coordinates in the light modulation device.

  With such a configuration, the two straight lines detected at each corner are converted into coordinates in the light modulation device, and then the intersection of the two straight lines is calculated to obtain the vertex of the frame. Can be detected with high accuracy. Therefore, the trapezoidal distortion correction can be performed with higher accuracy.

  In addition to the configuration as the image processing apparatus described above, the present invention can also be configured as an image processing method, a projector including the image processing apparatus, and a computer program. Such a computer program may be recorded on a computer-readable recording medium. As the recording medium, various media such as a flexible disk, a CD-ROM, a DVD-ROM, a magneto-optical disk, a memory card, and a hard disk can be used.

It is explanatory drawing which shows the hardware constitutions of the projector as embodiment of the image processing apparatus of this invention. FIG. 2 is a block diagram illustrating a specific configuration of the image processing circuit illustrated in FIG. 1. It is a flowchart which shows the flow of the initial screen frame correction process by a projector. It is a figure which shows an example of the pattern for a measurement formed in a liquid crystal panel. It is a figure which shows an example of the display image of the pattern for a measurement projected with respect to the screen. It is a figure which shows an example of the captured image of the pattern for a measurement. It is a figure for demonstrating an example of the method of detecting a straight line from a frame image. It is a figure for demonstrating the method to convert a screen frame straight line from a camera coordinate to a panel coordinate. It is a figure for demonstrating the method of calculating a correction value. It is a flowchart for demonstrating an automatic screen frame correction process. It is a figure which shows an example of the pattern for a measurement formed in a liquid crystal panel in step S50. It is a figure which shows the example of the black image projected in order to detect a screen frame straight line in step S140. It is a figure for demonstrating the process when a part of eight screen frame straight lines are not detected by edge detection. It is a figure for demonstrating the process when a straight line detection part misdetects a screen frame straight line. It is another figure for demonstrating the process when a straight line detection part misdetects a screen frame straight line.

A. Projector configuration:
FIG. 1 is an explanatory diagram showing a hardware configuration of a projector 10 as an embodiment of an image processing apparatus of the present invention. The projector 10 projects image light representing an image and displays the image on a projection surface such as the screen 30.

  The projector 10 includes an A / D (analog / digital) converter 110, an image processing circuit 120, a liquid crystal panel 130, a liquid crystal panel driver 132, an illumination optical system 140, and a projection optical system 150 having a projection lens 152. A lens driving unit 154, a CPU 160, a RAM 175, a ROM 170 having a measurement pattern storage unit 171, a remote control unit 180, a remote control 181, an imaging unit 190, and a vibration detection unit 129. . Each component of the projector 10 is connected to each other via a bus 102.

  The A / D converter 110 performs A / D conversion on an input image signal input via an image supply device such as a DVD player or a PC (personal computer) (not shown) via a cable 200, and converts the digital image signal into a digital image signal. Output.

  The image processing circuit 120 includes various functional units that execute image processing. The image processing circuit 120 performs screen frame correction processing and image display status (for example, brightness, contrast, synchronization, etc.) on a digital image signal (hereinafter referred to as “input image”) output from the A / D conversion unit 110. Tracking, color density, hue, etc.) are adjusted and output to the liquid crystal panel drive unit 132. The screen frame correction processing and the functional units of the image processing circuit 120 will be described later.

  The liquid crystal panel driving unit 132 drives the liquid crystal panel 130 based on the digital image signal input via the image processing circuit 120.

  The illumination optical system 140 includes a light source that emits light, a uniformizing optical system that uniformizes the illuminance distribution of the light emitted from the light source, and the like. The light emitted from the illumination optical system 140 enters the liquid crystal panel 130.

  The liquid crystal panel 130 is a light modulation device that modulates light emitted from the illumination optical system 140 based on image data. The liquid crystal panel 130 forms an image (hereinafter also referred to as a “panel image”) for modulating the illumination light emitted from the illumination optical system 140 into effective image light representing an image on the liquid crystal panel 130.

  The projection optical system 150 is attached to the front surface of the housing of the projector 10 and enlarges and projects the light modulated by the liquid crystal panel 130 into image light. The lens driving unit 154 drives the projection lens 152 included in the projection optical system 150 to change the size of the display image displayed on the screen 30. The liquid crystal panel 130, the illumination optical system 140, and the projection optical system 150 constitute a projection unit that projects an image.

  The remote controller control unit 180 receives an instruction from the user via the remote controller 181 and transmits the instruction to the CPU 160 via the bus 102. In this embodiment, the projector 10 receives an instruction from the user through the remote controller 181 and the remote controller control unit 180, but may receive an instruction from the user through another configuration such as an operation panel. .

  The imaging unit 190 includes a CCD camera and captures and acquires various images. Hereinafter, an image captured by the imaging unit 190 is also referred to as a “captured image”. Among the captured images, an image obtained by capturing at least a part of a frame of the screen 30 (hereinafter also referred to as “screen frame SCF”) and a measurement pattern projected on the screen 30 (details will be described later) What is captured is also referred to as a “pattern image”. The captured image acquired by the imaging unit 190 is stored in the RAM 175. Note that the imaging unit 190 may include another device capable of imaging instead of the CCD camera.

  The vibration detection unit 129 monitors the movement and stillness of the projector 10. In the present embodiment, the vibration detection unit 129 includes a gyro sensor (not shown), and detects the movement of the projector 10 by the gyro sensor.

  The CPU 160 causes the image processing circuit 120 to execute a screen frame correction process in accordance with an instruction from the remote controller control unit 180 and a detection result by the vibration detection unit 129. This screen frame correction process will be described later.

B. Image processing circuit configuration:
FIG. 2 is a block diagram showing a specific configuration of the image processing circuit 120 shown in FIG. As illustrated, the image processing circuit 120 includes a pattern output unit 121, a pattern detection unit 122, a straight line detection unit 123, a coordinate conversion unit 124, a correction value calculation unit 125, and a distortion correction unit 126. ing.

  The pattern detection unit 122 detects a measurement point for performing a screen frame correction process, which will be described later, based on the pattern image captured by the imaging unit 190. The measurement points will be described later.

  The straight line detection unit 123 detects two straight lines having a crossing relationship for each corner of the screen 30 based on the frame image captured by the imaging unit 190. Therefore, the straight line detection unit 123 detects a total of eight straight lines with respect to the screen 30 having four corners.

  The coordinate conversion unit 124 calculates a coordinate conversion coefficient based on the coordinate value of the measurement point of the measurement pattern on the liquid crystal panel 130 and the coordinate value of the measurement point in the captured image detected by the pattern detection unit 122. In the following description, the coordinate system in the liquid crystal panel 130 is also referred to as “panel coordinates”, and the coordinate system in the captured image is also referred to as “camera coordinates”. The coordinate conversion coefficient is a coefficient for converting camera coordinate values into panel coordinate values.

  The correction value calculation unit 125 calculates a correction value for performing trapezoidal distortion correction based on the coordinate conversion coefficient calculated by the coordinate conversion unit 124 and the straight line detected by the straight line detection unit 123.

  The distortion correction unit 126 performs trapezoidal distortion correction on the input image acquired from the A / D conversion unit 110 using the correction value calculated by the correction value calculation unit 125.

  The pattern output unit 121 superimposes a measurement pattern on an input image or an input image that has been corrected for trapezoidal distortion during screen frame correction, which will be described later.

C. Initial screen frame correction processing:
FIG. 3 is a flowchart showing a flow of initial screen frame correction processing by the projector 10. The screen frame correction process is a series of processes for performing trapezoidal distortion correction on the input image so that each side of the outer periphery of the display image matches the screen frame. The initial screen frame correction process is executed in accordance with an instruction from the user through the remote controller 181 after the projector 10 is installed, for example. Note that the initial screen frame correction processing may be automatically executed in response to, for example, power-on or image signal input.

  When the initial screen frame correction process is started, the pattern output unit 121 projects the measurement pattern onto the screen 30 (step S110). Specifically, the pattern output unit 121 reads out image data representing the measurement pattern stored in the measurement pattern storage unit 171 of the ROM 170, superimposes it on the input image, and outputs it to the liquid crystal panel drive unit 132. Thus, the measurement pattern is projected. This measurement pattern is represented by panel coordinate values. The panel coordinate value of the measurement point included in the measurement pattern is acquired by the pattern output unit 121.

  FIG. 4 is a diagram showing an example of the measurement pattern PT1 formed on the liquid crystal panel 130. As shown in FIG. The measurement pattern PT1 shown in FIG. 4 includes pattern areas A, B, C, and D formed by four white circular measurement points and a substantially square black area around the measurement points. The pattern area A is near the upper left corner of the liquid crystal panel 130, the pattern area B is near the upper right corner of the liquid crystal panel 130, the pattern area C is near the lower right corner of the liquid crystal panel 130, and the pattern area D is the lower left corner of the liquid crystal panel 130. Located in the vicinity. The outer frame (measurement pattern frame PF) of the measurement pattern PT1 coincides with the image forming frame of the liquid crystal panel 130. 4 is an area where an input image transmitted from the cable 200 to the liquid crystal panel driving unit 132 via the A / D conversion unit 110 is formed.

  FIG. 5 is a diagram showing an example of a display image of the measurement pattern PT1 projected on the screen 30. As shown in FIG. In the present embodiment, the projection lens 152 does not face the screen 30. Therefore, the display image of the measurement pattern PT1 has a trapezoidal shape as shown in FIG. In the display image of the measurement pattern PT1, the measurement point of the pattern region A is near the upper left corner of the screen 30, the measurement point of the pattern region B is near the upper right corner of the screen 30, and the measurement point of the pattern region C is the screen. In the vicinity of the lower right corner 30, the measurement points of the pattern region D are located in the vicinity of the lower left corner of the screen 30.

  Next, the imaging unit 190 captures a display image of the measurement pattern PT1, and the pattern detection unit 122 analyzes the captured image (pattern image) by a known method to detect a measurement point in camera coordinates ( Step S120). FIG. 6 is a diagram illustrating an example of a captured image of the measurement pattern PT1. In the example shown in FIG. 6, the entire display image having the measurement pattern frame PF outside the screen frame SCF is projected in the CCD camera frame (camera frame CF) of the imaging unit 190 in the captured image. ing. By analyzing this captured image, camera coordinate values corresponding to the four measurement points of each pattern region are acquired.

  When the panel coordinate values and camera coordinate values corresponding to the measurement points of each pattern area are acquired, the coordinate conversion unit 124 calculates a coordinate conversion coefficient for each pattern area (step S130). Specifically, the coordinate conversion unit 124 calculates the coordinate conversion coefficient A using the panel coordinate value and the camera coordinate value of the measurement point in the pattern area A. Similarly, the coordinate conversion unit 124 converts the coordinate conversion coefficient B using the panel coordinate value and the camera coordinate value of the measurement point in the pattern area B, and converts the coordinate conversion coefficient B using the panel coordinate value and the camera coordinate value of the measurement point in the pattern area C. The coefficient C is calculated using the panel coordinate value and the camera coordinate value of the measurement point in the pattern area D, respectively. In this embodiment, a projective transformation coefficient is obtained as a coordinate transformation coefficient based on four measurement points. Information on the calculated coordinate conversion coefficients A to D is stored in the RAM 175.

  Next, the pattern output unit 121 projects a black image for screen frame SCF imaging on the screen 30 in order to obtain a frame image (step S140). Specifically, the pattern output unit 121 outputs black image data to the liquid crystal panel drive unit 132. The liquid crystal panel driving unit 132 forms a black image on the liquid crystal panel 130 based on the output image data. When the black image data used in the present embodiment is formed on the liquid crystal panel 130, the entire image forming area of the liquid crystal panel 130 becomes a black area. By doing so, the entire black image is displayed on the screen 30 via the projection optical system 150.

  When a black image is projected, the imaging unit 190 captures the projected black image and acquires a frame image. The straight line detection unit 123 detects a total of eight straight lines on the screen frame from the frame image, two lines per side (step S150).

  FIG. 7 is a diagram for explaining an example of a method for detecting a straight line from a frame image. Hereinafter, a straight line corresponding to the screen frame SCF is referred to as a “screen frame straight line”. In FIG. 7, a portion corresponding to the screen frame SCF in the acquired frame image is indicated by a black frame. Note that the circular measurement points shown in FIG. 7 are those detected in step S120 and are not included in the frame image. The measurement points shown in FIG. 7 are shown together with the frame image for the description of the detection of the screen frame straight line. In the following figures, circular measurement points are also shown.

  Specifically, the screen frame straight line is detected as follows. For example, in order to detect two straight lines on the lower side of the screen frame, first, a line segment C3D4 is formed by connecting the measurement point C3 in the lower right corner and the measurement point D4 in the lower left corner detected in step S120. Divide C3D4 into three equal parts. Then, edge detection is performed by applying a contour extraction filter such as a differential filter or a Laplacian filter from the measurement point C3 and the measurement point D4, starting from a line segment inside one third of the line segment C3D4 and starting downward. (FIG. 7A). The straight line detection unit 123 detects the screen frame straight lines LD1 and LC1 within the detected edge range by using, for example, the least square method or the Hough transform (FIG. 7B). Similarly, the straight line detection unit 123 connects the measurement points D4 and A1, the measurement points A1 and B2, and the measurement points B2 and C3 to create line segments, and generates two screen frame straight lines for one screen frame. To detect. In this way, the straight line detection unit 123 detects a total of eight screen frame straight lines.

  When the straight line detection unit 123 detects eight screen frame straight lines, the coordinate conversion unit 124 converts the eight screen frame straight lines in the camera coordinates to the screen frame in the panel coordinates using the coordinate conversion coefficient calculated in step S130. Conversion into a straight line (step S160). FIG. 8 is a diagram for explaining a method of converting a screen frame straight line from camera coordinates to panel coordinates. FIG. 8A shows screen frame straight lines LA1, LA2, LB1, LB2, LC1, LC2, LD1, and LD2 at the camera coordinates detected in step S150. The coordinate conversion unit 124 converts the screen frame straight lines LA1 and LA2 in the camera coordinates into screen frame straight lines LA1i and LA2i in the panel coordinates by performing projective conversion using the coordinate conversion coefficient A calculated in step S130. Similarly, the coordinate conversion unit 124 uses the coordinate conversion coefficient B to convert the screen frame straight lines LB1 and LB2, the coordinate conversion coefficient C to the screen frame straight lines LC1 and LC2, and the coordinate conversion coefficient D to the screen frame straight line LD1. LD2 is converted into a screen frame straight line in panel coordinates. By doing so, the screen frame straight line is converted from the camera coordinates to the panel coordinates using the coordinate conversion coefficients calculated by the four measurement points closer to each screen frame straight line.

  When the screen frame straight line in the camera coordinates is converted into the screen frame straight line in the panel coordinates, the correction value calculation unit 125 calculates a correction value for performing trapezoidal distortion correction (step S170).

  FIG. 9 is a diagram for explaining a method of calculating a correction value. First, in order to calculate the correction value, the vertex of the screen frame in the panel coordinates is obtained by obtaining the intersection V of the screen frame straight lines LA1i and LA2i as shown in FIG. Similarly, the vertex of the screen frame in the panel coordinates is obtained by obtaining the intersection of the screen frame straight lines LB1i and LB2, the intersection of the screen frame straight lines LC1i and LC2i, and the intersection of the screen frame straight lines LD1i and LD2i. Then, a correction value (for example, a projective transformation coefficient) for correcting the trapezoidal distortion of the input image is calculated from the obtained vertex coordinates by a known method (projective transformation). The calculated correction value is stored in the RAM 175.

  When the correction value is calculated, the distortion correction unit 126 performs trapezoidal distortion correction on the input image using the calculated correction value (step S180). The calculation of the correction value and the trapezoidal distortion correction can be performed using a known method (for example, a method described in JP 2011-176705 A). The liquid crystal panel driving unit 132 forms an input image on the liquid crystal panel 130 that has been subjected to trapezoidal distortion correction based on the input image data. As a result, the image subjected to the trapezoidal distortion correction is displayed on the screen 30 via the projection optical system 150.

  In general, a straight line included in a captured image captured by a CCD camera may be gently curved due to the influence of lens distortion of the CCD camera. Therefore, it is difficult to accurately detect each side of the screen frame SCF as one straight line. However, according to the projector 10 of the present embodiment described above, the screen frame SCF is detected as a total of eight straight lines, two for each corner of the screen 30. Therefore, in the captured image, a straight line at the corner portion of the screen 30 that is easily bent due to the influence of lens distortion included in the CCD camera can be detected more accurately. Therefore, the shape of the screen 30 can be detected with high accuracy by converting such a straight line into panel coordinates.

  Further, the eight screen frame straight lines can be detected by performing edge detection in the screen frame direction from the line segment created by the captured image (pattern image) of the measurement point projected near the corner of the screen 30. Therefore, it is possible to detect eight straight lines in a short time compared to the case where edge detection is performed on the entire projected black image. Furthermore, in this embodiment, a coordinate conversion coefficient is calculated for each corner of the screen 30. Therefore, the coordinates of the corners of the screen 30 and the corner of the liquid crystal panel 130 that are easily affected by the distortion of the projection lens 152 that projects the measurement pattern PT1 and the distortion of the lens of the CCD camera can be more accurately performed. it can. As a result, the keystone distortion can be corrected with high accuracy.

  Further, in this embodiment, since the detected eight straight lines are converted into panel coordinates and the vertex of the screen frame is obtained, the shape of the screen 30 at the panel coordinates can be detected with high accuracy. Therefore, the keystone distortion correction can be performed with higher accuracy.

D. Automatic screen frame correction processing:
Next, a case where automatic screen frame correction processing is performed will be described. The automatic screen frame correction process is a process for automatically correcting trapezoidal distortion when movement of the projector 10 is detected. Also in the automatic screen frame correction processing, the initial screen described above is used in that a total of eight screen frame straight lines are detected for each corner of the screen 30 and a coordinate conversion coefficient is calculated for each corner of the screen 30. This is the same as the frame correction process. However, in the initial screen frame correction process, the measurement pattern (step S110) and the black image (step S140) are separately projected, whereas in the automatic screen frame correction process, the screen frame straight line detection is performed. A difference is that the measurement pattern including both the black region and the measurement point for calculating the coordinate conversion coefficient is projected on the image after the trapezoidal distortion correction (step S50). Hereinafter, the automatic screen frame correction process will be described focusing on these differences.

  FIG. 10 is a flowchart for explaining the automatic screen frame correction process. When the user moves the projector 10 (step S10), the vibration detection unit 129 detects the vibration (movement) of the projector 10 (step S20).

  When the vibration detection unit 129 detects vibration, the CPU 160 causes the image processing circuit 120 to execute the following processing from step S50 to step S180.

  In step S <b> 50, the pattern output unit 121 reads out image data representing the measurement pattern PT <b> 2 stored in the measurement pattern storage unit 171 of the ROM 170 and projects it on the screen 30.

  FIG. 11 is a diagram illustrating an example of the measurement pattern PT2 formed on the liquid crystal panel 130 in step S50. A measurement pattern PT2 shown in FIG. 11 has L-shaped black areas AL, BL, CL, DL having four measurement points at the corners at the respective corners of the liquid crystal panel 130. Two straight line area portions where the L-shaped black areas AL, BL, CL, DL intersect each other extend from the inside to the outside of the image forming area of the liquid crystal panel 130. A region indicated by hatching in FIG. 11 is a region where an input image after distortion correction is displayed. In the first execution of step S50, the input image N after the trapezoidal distortion correction calculated by the initial screen frame correction process is displayed. As shown in FIG. 11, in the present embodiment, the measurement pattern PT2 is superimposed and displayed on the input image after distortion correction. In FIG. 11, it is assumed that an area displayed by cross-hatching outside the input image N after the trapezoidal distortion correction is a background area H, which is actually filled with black.

  When the measurement pattern PT2 is projected, like the initial screen frame correction process, the pattern detection unit 122 detects a measurement point from the captured pattern image (step S120), and the coordinate conversion unit 124 detects each corner. A coordinate conversion coefficient is calculated for each step (step S130). The straight line detection unit 123 performs edge detection from the measurement point side included in the L-shaped black region toward the screen frame SCF, and detects two screen frame straight lines for each corner, for a total of eight lines (step S155). . When the screen frame straight line and the coordinate conversion coefficient are thus calculated, the screen frame straight line is converted from the camera coordinates to the panel coordinates as in the initial screen frame correction process (step S160), and the correction value is calculated (step S170). . Then, the keystone correction of the input image is performed according to the correction value (step S180).

  Next, the CPU 160 determines whether or not a fixed time has elapsed since the projector 10 was stationary according to the vibration detection result by the vibration detection unit 129 (step S185). When the projector 10 is stationary and a predetermined time has not elapsed (step S185: No), steps S50 to S180 are executed again. Therefore, the pattern output unit 121 superimposes the measurement pattern PT2 on the currently displayed input image after the trapezoidal distortion correction. When a new correction value is obtained, the currently displayed input image after the trapezoidal distortion correction is replaced with the updated input image after the trapezoidal distortion correction. When the user continues to move the projector 10 in this way, Step S50 to Step S180 are repeatedly executed, the correction value is calculated, and the updated input image after the keystone distortion correction is displayed. That is, the input image N after the trapezoidal distortion correction is displayed on the screen 30 while changing one after another in real time.

  If the CPU 160 determines that a predetermined time has elapsed since the projector 10 was stopped (step S185: Yes), the CPU 160 causes the pattern output unit 121 to stop outputting the measurement pattern PT2, and the measurement pattern superimposed on the video image. PT2 is erased (step S190).

  According to the automatic screen frame correction processing by the projector 10 described above, two screen frames SCF are detected for each corner of the screen 30 as a total of eight straight lines, and coordinate conversion coefficients corresponding to the respective corners are detected. Since the correction is performed using the above, the same effect as the above-described initial screen frame correction process is obtained. Further, the measurement pattern to be superimposed on the input image after trapezoidal distortion correction is a black region having measurement points as shown in FIG. 11, and only covers a part of the input image after trapezoidal distortion correction. Therefore, the user can appropriately install the projector 10 in real time while confirming the input image after the trapezoidal distortion correction. Furthermore, even if the projector 10 moves, the input image after distortion correction is continuously displayed in real time, so that viewing is not interrupted and stress applied to the user is reduced.

E. Variations:
As mentioned above, although one Embodiment of this invention was described, this invention is not limited to such Embodiment, A various structure can be taken in the range which does not deviate from the meaning.

E1. Modification 1:
The measurement pattern used in step S110 in the initial screen frame correction process and in step S50 in the automatic screen frame correction process includes four pattern areas having four measurement points, so that a total of 16 points are obtained. Has measuring points. However, the number of measurement points included in the measurement pattern is not limited to this, and may be four or more in total. If there are four or more measurement points, a coordinate conversion coefficient for performing projective conversion from camera coordinates to panel coordinates can be calculated. For example, it is good also as having one measurement point in each measurement area | region, and the pattern for a measurement which consists of one measurement area | region provided with four measurement points may be sufficient. The shape of the measurement point may be, for example, a square shape or a cross shape as long as it can be detected by image processing. Furthermore, in step S110 in the initial screen frame correction process, if a measurement point can be detected, a measurement pattern having only a measurement point that does not include a black region portion can be used.

E2. Modification 2:
In step S140 in the above-described initial screen frame correction processing, the entire black image is projected in order to detect the screen frame straight line in step S150. However, the black image to be projected may not be the entire black image. FIG. 12 is a diagram illustrating an example of a black image projected to detect a screen frame straight line in step S140. For example, a black image having a background area at the center and a black area K on the outer periphery so that the black area of the display image overlaps the screen frame SCF, such as a black image PT3 shown in FIG. Also good. Further, a black image having black areas Ka, Kb, Kc, and Kd at the respective corners of the liquid crystal panel 130 may be used like a black image PT4 shown in FIG. Further, the black area (including the L-shaped black area and the black image) described in the above initial screen frame correction process and automatic screen frame correction process is not limited to complete black. For example, it may be a region where the luminance is low regardless of the color (for example, luminance of 0% to 10%).

E3. Modification 3:
In step S150 in the initial screen frame correction process described above, the straight line detection unit 123 divides the line segment C3D4 formed by connecting the measurement points into three equal parts, and performs edge detection with respect to the width of the screen frame SCF. However, the width for edge detection is not limited to this. The width for edge detection can be set arbitrarily.

E4. Modification 4:
In the screen frame correction process described above, the intersection of the straight lines is obtained after converting the screen frame straight line from the camera coordinates to the panel coordinates. After obtaining the intersection point of the straight lines on the camera coordinates, the intersection point is converted into a coordinate conversion coefficient. May be converted into panel coordinates to obtain the vertex of the frame.

E5. Modification 5:
Depending on the installation state of the screen 30, the lighting may be reflected on the screen 30 or the color of the screen frame may be partially different, so that eight screen frame straight lines may not be detected by edge detection. Even in such a case, the projector 10 can perform the above-described screen frame correction processing. FIG. 13 is a diagram for explaining processing when a part of eight screen frame straight lines is not detected by edge detection. In the panel image shown in FIG. 13A, the screen frame straight line LA2i (see FIG. 8A) that should have a relationship intersecting the screen frame straight line LA1i is not detected. In such a case, the straight line detection unit 123 extends the screen frame straight line LD2i estimated to be on the same screen frame SCF as the screen frame straight line LA2i, and complements the screen frame straight line LA2i that has not been detected (FIG. 13 (B)). In the above-described screen frame correction processing, the screen frame straight lines are detected by dividing them into eight, so even if there are screen frame straight lines that are not detected in this way, the screen frame straight lines that are estimated to be approximated are complemented. Can do. Therefore, trapezoidal distortion correction can be performed even if there is a straight line that is not detected in the frame image.

  In the case where it is not possible to detect both the screen frame straight line LA2i and the screen frame straight line LD2i that are estimated to be on the same screen frame SCF, as shown in Japanese Patent Application Laid-Open Nos. 2010-05042 and 2008-212355. It is good also as complementing two screen frame straight lines by a well-known method. If this method is used, it is only necessary to detect at least one straight line out of the eight screen frame straight lines, and the success rate of straight line detection is improved.

E6. Modification 6:
FIG. 14 is a diagram for describing processing when the straight line detection unit 123 erroneously detects a screen frame straight line when detecting an edge of a frame image. In the panel image shown in FIG. 14A, the screen frame straight line LA2i estimated to be on the same screen frame and the screen frame straight line LD2i are detected separately. If a vertex is obtained based on such a screen frame straight line and trapezoidal distortion correction is performed, there is a possibility that the screen frame does not match each side of the display image. In order to prevent such a situation, the straight line detection unit 123 first obtains a relationship value between the position and inclination of the two screen frame straight lines LA2i and the screen frame straight line LD2i that are estimated to be on the same screen frame. For example, the intersection point between the center line L in the panel image and each screen frame straight line is calculated, and the distance h between the respective intersection points is calculated (FIG. 14A). Further, the straight line detection unit 123 calculates the absolute value m2 of the inclination of the screen frame straight line LA2i and the absolute value m1 of the inclination of the screen frame straight line LD2i, and obtains the absolute value | m1-m2 | of the difference. Whether the distance h is equal to or smaller than a predetermined value and whether the absolute value | m1-m2 | of the difference between the absolute values m1 and m2 of the slope is equal to or smaller than a predetermined value. judge. As a result of the determination, if at least one of the values is equal to or greater than a predetermined value, one of the screen frame straight line LA2i and the screen frame straight line LD2i is discarded. In FIG. 14A, since the slopes of the screen frame straight line LA2i and the screen frame straight line LD2i are substantially equal, | m1-m2 | is within the default value, but the distance h is greater than or equal to the default value. In the case of the positional relationship as shown in FIG. 14A, the straight line detection unit 123 discards the screen frame straight line LA2i located farther from the measurement point and extends the screen frame straight line LD2i to the screen. The frame straight line LA2i is complemented (FIG. 14B).

  FIG. 15 is another diagram for explaining processing when the straight line detection unit 123 erroneously detects a screen frame straight line when detecting an edge of a frame image. The distance h between the two screen frame straight lines LA2i estimated to be on the same screen frame SCF and the screen frame straight line LD2i shown in FIG. 15A is within a predetermined value, but the absolute values m1 and m2 of the inclinations. The absolute value of the difference | m1-m2 | In such a case, the straight line detection unit 123 discards the screen frame straight line LA2i having an inclination different from that of the left frame of the image forming area of the liquid crystal panel 130, and extends the screen frame straight line LD2i to extend the screen frame straight line LA2i. Is complemented (FIG. 15B). Even if there is a screen frame straight line that is erroneously detected in this way, the screen frame straight line is detected by dividing it into eight lines. Can be complemented. Therefore, even if there is a false detection, it becomes possible to perform keystone distortion correction with high accuracy.

  Note that the screen frame straight line complementation processing of Modification 5 and Modification 6 can be performed on camera coordinates, but is preferably performed on panel coordinates. This is because complementation can be performed with a screen frame straight line that is estimated to be more approximate if the compensation process is performed after the distortion of the camera lens of the projector 10 is eliminated.

E7. Modification 7:
As the input image after the trapezoidal distortion correction obtained by the automatic screen frame correction process projected in step S180 described above, the current correction value, the correction value calculated before calculating the current correction value (the previous correction value), and It is also possible to calculate an intermediate correction value and display an image in which the trapezoidal distortion correction is performed based on the intermediate correction value. For example, first, an intermediate correction value between the correction value calculated by the fifth process and the correction value calculated by the sixth process is calculated. Then, after displaying the image subjected to the trapezoidal distortion correction based on the intermediate correction value, the input image subjected to the trapezoidal distortion correction based on the correction value calculated by the sixth process may be displayed. . Furthermore, after calculating a plurality of intermediate correction values (for example, 3 to 4) and displaying the images in which the trapezoidal distortion correction is performed based on them in sequence, the trapezoidal distortion correction is performed based on the sixth correction value. The input image may be displayed. During the automatic screen frame correction process, an image that has been corrected for trapezoidal distortion based on an intermediate correction value between the current correction value and the correction value calculated before the current correction value is calculated. After the display and the automatic screen frame correction processing is completed, an image that has been subjected to trapezoidal distortion correction based on the current correction value may be displayed. By doing so, the input image can be displayed with a smooth change, so that the user can confirm the input image after the trapezoidal distortion correction more comfortably.

E8. Modification 8:
In step S50 described above, the measurement pattern including both the black area for detecting the screen frame straight line and the measurement point for calculating the coordinate conversion coefficient is superimposed and projected on the image after the trapezoidal distortion correction. The black region, the measurement point, and the image after the trapezoidal distortion correction may be alternately displayed in a time division manner. In step S50, the measurement points may be continuously displayed, and the black area and the image after the trapezoidal distortion correction may be alternately displayed in a time division manner.

E9. Modification 9:
In step S185 described above, the measurement pattern PT2 superimposed on the video when the predetermined time has elapsed is deleted, but eight screen frame straight lines cannot be directly detected from the frame image in step S155 for the predetermined time. Can be set to be longer. Even if eight screen frame straight lines cannot be detected by edge detection, it is possible to complement the screen frame straight lines as described in the above modification. However, since the screen frame straight line actually detected from the frame image reflects the effect of camera lens distortion rather than the complemented screen frame straight line, the input image after correcting the trapezoidal distortion is more appropriate. Can be calculated. Therefore, when the screen frame straight line cannot be detected, the time until the measurement pattern PT2 is erased from the input image is lengthened so that the user places the projector 10 at a position where the screen frame straight line can be detected from the frame image. Can be re-installed.

E10. Modification 10:
In the above-described embodiment, the vibration detection unit 129 detects vibration from the gyro sensor included in the projector 10, but the vibration detection unit 129 includes another vibration detection sensor such as an acceleration sensor instead of the gyro sensor. Also good. Further, the vibration detection unit 129 may detect the vibration of the projector 10 by analyzing a change in the background image without using a sensor.

E11. Modification 11:
In the above-described embodiment, the area where the input image N after trapezoidal distortion correction is displayed is an image representing only the borderline of the input image N after trapezoidal distortion correction, or the borderline of the input image N after trapezoidal distortion correction. An image in which the inside is painted with a color (for example, white or blue) different from the color outside the frame line may be displayed. If these images are displayed, the processing load for performing the trapezoidal distortion correction is reduced, so that the shape of the input image after correction can be displayed at high speed. These images are preferably similar to the image forming frame of the liquid crystal panel, and more preferably the same.

E12. Modification 12:
The projector 10 in the above-described embodiment performs the initial screen frame correction process and the automatic screen frame correction process. However, only one of the screen frame correction processes may be performed.

E13. Modification 13:
In the above-described embodiment, the projector 10 has been described using the transmissive liquid crystal panel 130 as the light modulation device. However, the light modulation device is not limited to the transmissive liquid crystal panel 130. For example, a configuration in which light from the illumination optical system 140 is modulated using a digital micro-mirror device (DMD), a reflective liquid crystal panel, or the like as the light modulation device may be used. Also, a CRT projector that projects an image on a small CRT (cathode ray tube) onto a projection surface may be used.

E14. Modification 14:
In the above-described embodiment, the image processing apparatus of the present invention is applied to the projector. However, the image processing circuit 120 shown in FIG. 2 can be regarded as an image processing apparatus. In the above-described embodiment, the image processing circuit performs the screen frame correction processing in hardware. However, the CPU may execute the screen frame correction processing in software by executing a program.

DESCRIPTION OF SYMBOLS 10 ... Projector 30 ... Screen 102 ... Bus 110 ... A / D conversion part 120 ... Image processing circuit 121 ... Pattern output part 122 ... Pattern detection part 123 ... Straight line detection part 124 ... Coordinate conversion part 125 ... Correction value calculation part 126 ... Distortion Correction unit 129 ... Vibration detection unit 130 ... Liquid crystal panel 132 ... Liquid crystal panel drive unit 140 ... Illumination optical system 150 ... Projection optical system 152 ... Projection lens 154 ... Lens drive unit 160 ... CPU
170 ... ROM
171 ... Pattern storage unit for measurement 175 ... RAM
180 ... remote control unit 181 ... remote control 190 ... imaging unit 200 ... cable A, B, C, D ... pattern area AL, BL, CL, DL ... L-shaped black area A1, B2, C3, D4 ... measurement point LA1, LA2, LB1, LB2, LC1, LC2, LD1, LD2, LA1i, LA2i, LB1i, LB2i, LC1i, LC2i, LD1i, LD2i ... Screen frame straight lines K, Ka, Kb, Kc, Kd ... Black regions h ... Distance N ... Input image after trapezoidal distortion correction H ... Background region V ... Intersection SCF ... Screen frame CF ... Camera frame L ... Middle line PF ... Measurement pattern frame PT1, PT2 ... Measurement pattern PT3, PT4 ... Black image

Claims (9)

An image processing apparatus,
A pattern detection unit that detects the pattern based on a pattern image in which a pattern having a predetermined shape projected on the projection surface is projected from the projection unit having the light modulation device;
A straight line detection unit that detects two straight lines that intersect each corner of the projection surface based on an image of a frame in which at least a part of the frame of the projection surface is captured;
Based on the coordinates of the pattern in the light modulation device and the coordinates of the pattern detected from the pattern image, a coordinate conversion coefficient for converting coordinates in the captured image into coordinates in the light modulation device is calculated. A coordinate conversion coefficient calculation unit for
A correction value calculation unit for obtaining a vertex of the frame based on the two detected straight lines for each corner and the coordinate transformation coefficient, and calculating a correction value for performing trapezoidal distortion correction based on the vertex;
An image processing apparatus comprising: The image processing apparatus according to claim 1,
The pattern is located at each corner of the light modulation device,
The coordinate conversion coefficient calculation unit calculates the coordinate conversion coefficient for each corner of the projection surface,
The correction value calculating unit obtains the vertex based on two detected straight lines intersecting each corner of the projection surface and the coordinate conversion coefficient corresponding to the corner. apparatus. The image processing apparatus according to claim 1 or 2,
The straight line detection unit estimates the undetected straight line based on another straight line when one straight line is not detected from two straight lines on one frame side of the projection surface. Image processing device. An image processing apparatus according to any one of claims 1 to 3, wherein
When the straight line detection unit does not satisfy a predetermined relationship at least one of the relationship between the positions of two detected straight lines estimated to be on one frame side of the projection surface and the relationship between the inclinations An image processing apparatus that discards one of the detected two straight lines and estimates one straight line based on the other straight line. The image processing apparatus according to claim 4,
If the relationship between the positions does not satisfy the predetermined relationship, the straight line detection unit discards a straight line that is far from the pattern among the two detected straight lines, and the distance from the pattern is An image processing apparatus that estimates the one straight line based on a close straight line. The image processing apparatus according to claim 4 or 5, wherein
The straight line detector, when the relationship between the tilts does not satisfy a predetermined relationship, the tilt between the detected two straight lines and the frame side estimated to be the two detected straight lines An image processing apparatus that discards a straight line that differs from each other and estimates the one straight line based on a straight line that is closer in inclination to the frame side. An image processing apparatus according to any one of claims 1 to 6, wherein
The image processing apparatus, wherein the correction value calculation unit obtains the apex of the frame after the straight line detection unit converts the two detected straight lines for each corner into coordinates in the light modulation device.   A projector comprising: the image processing device according to any one of claims 1 to 7, a trapezoidal distortion correction unit, and the projection unit. An image processing method comprising:
Detecting the pattern based on a pattern image in which a pattern having a predetermined shape projected on a projection surface is projected from a projection unit having a light modulation device;
Detecting two straight lines in a relationship intersecting each corner of the projection surface based on a frame image obtained by capturing at least a part of the frame of the projection surface;
Based on the coordinates of the pattern in the light modulation device and the coordinates of the pattern detected from the pattern image, a coordinate conversion coefficient for converting coordinates in the captured image into coordinates in the light modulation device is calculated. And a process of
Obtaining a vertex of the frame based on the two detected straight lines and the coordinate transformation coefficient for each corner, and calculating a correction value for performing trapezoidal distortion correction based on the vertex, and Image processing method.

Images