JP2015139006A - Information processing apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent reduction of correction accuracy even if a pattern image partially includes a portion that is not imaged.SOLUTION: An information processing apparatus includes: projection image acquisition means 11 for acquiring a first pickup image that images a first line group and a second pickup image that images a second line group crossing the first line group; defective part detection means 19 which detects a partial or entire defect for each line of the first line group and detects a partial or entire defect for each line of the second line group in the second pickup image; interpolation means 15 for interpolating the defect of the line; intersection detection means 18 for detecting a first intersection of the line of the first line group and the line of the second line group, the defect of the line being interpolated, and a second intersection of the line of the first line group and the line of the second line group in the projection image; correction coefficient calculation means 20 for calculating a correction coefficient by comparing a range formed from a plurality of first intersections with a range formed from a plurality of second intersections; and correction means 21 for correcting the projection image while using the correction coefficient.

Description

  The present invention relates to an information processing apparatus that corrects a projection image projected on a projection object before projection.

  A projector is a device that projects an image onto a projection object such as a screen. Some projectors project a projected pattern image, correct the distortion of the image based on the captured image, and then project the image (see, for example, Patent Document 1). In Patent Document 1, a pattern image including feature points arranged at equal intervals is projected, a distortion amount of a feature point of a captured image in which a projection surface is captured is calculated, and an image to be projected according to the calculated distortion amount A technique for correcting the above is disclosed. Further, in Patent Document 1, a threshold value for creating a binary image for extracting feature points is calculated, so that image distortion occurs even if ambient light changes.

  However, in Patent Document 1, since a plurality of discrete points are used as a pattern image, there is a problem that a reduction amount of information amount when one point is missing from the pattern image is large and correction accuracy may be reduced. There is.

  In view of the above problems, an object of the present invention is to provide an information processing apparatus that can suppress a reduction in correction accuracy even when a part of a pattern image is not captured.

  In view of the above problems, the present invention is an information processing apparatus that corrects a projection image projected onto a projection object before projection, and includes a first captured image in which the projected first line group is captured; Projection image acquisition means for acquiring a second captured image obtained by capturing a second line group that intersects the first line group; and for each line of the first line group in the first captured image Defect portion detection means for detecting a partial or entire defect and detecting a partial or entire defect for each line of the second line group in the second captured image; and a line of the first line group Or an interpolating means for interpolating the defect of the line of the second line group, a first intersection of the line of the first line group and the line of the second line group in which the defect is interpolated, and Intersection detection means for detecting a second intersection of the line of the first line group and the line of the second line group of the projection image; and a plurality of the first intersections A correction coefficient calculation unit that calculates a correction coefficient by comparing a range formed by the plurality of second intersections, and a correction unit that corrects the projection image using the correction coefficient. It is characterized by having.

  It is possible to provide an information processing apparatus that can suppress a decrease in correction accuracy even if a part of the pattern image is not captured.

It is a figure which shows the structural example of the image correction system in which the projector of this embodiment is used. 2A and 2B are a schematic external view of a front surface and a schematic external view of a back surface showing an example of a projector according to the present embodiment. 1 is an example of a schematic external view of an information processing apparatus. It is explanatory drawing explaining an example of the projection operation | movement of a projector. It is explanatory drawing explaining an example of the operation | movement which a projector calculates a correction parameter. It is explanatory drawing explaining an example of the operation | movement which projects the projection image (correction image) which the projector correct | amended. It is an example of the figure explaining the hardware constitutions of a projector and information processing apparatus. It is a figure which shows an example of the functional block diagram of information processing apparatus. It is a figure which shows an example of the projection pattern A and the projection pattern B. It is a figure which shows an example of the synthetic | combination pattern of the projection patterns A and B. FIG. It is a figure which shows an example of a binary code. It is an example of the figure explaining distortion of the projection image which the projector projected. It is a figure which shows an example of the captured image on which the projection pattern was projected. It is an example of the flowchart figure which shows the procedure of a preparation stage. It is an example of the flowchart figure which shows the procedure of a projection stage. It is an example of the flowchart figure explaining the procedure of the corresponding point extraction process of step S4 of FIG. It is a figure explaining an example of a line pattern area. It is a figure which shows an example of the horizontal line in which one part was missing. It is an example of the figure explaining the correspondence between the coordinates of the feature points of the composite pattern image and the coordinates of the intersection of the composite patterns of the projection patterns A and B. It is an example of the figure for demonstrating calculation of the projective transformation matrix H. FIG. It is an example of the figure explaining correction | amendment of a projection image. FIG. 15 is an example of a flowchart for explaining the procedure of feature point extraction processing in step S4 in FIG. 14 (Example 2); It is an example of the figure explaining the coordinate value of the point which is not interrupted before a missing part. It is a figure which shows an example of the horizontal line which does not have an end point on the right side of the defect | deletion part of a horizontal line. It is an example of the figure explaining the interpolation of a defect | deletion part (Example 3). It is an example of the figure explaining the interpolation in case the nth and n + 1th horizontal line is missing. It is an example of the figure explaining the interpolation at the time of combining a present Example and Example 1 or 2 (Example 3). It is an example of the figure explaining the interpolation when the uppermost or lower horizontal line is not completely detected. FIG. 15 is an example of a flowchart for explaining the procedure of feature point extraction processing in step S4 of FIG. 14 (Example 3); It is an example of the figure explaining the hardware constitutions and functional block of a projector. It is an example of the figure explaining the process sequence of the information processing apparatus of a present Example. It is an example of the figure explaining measurement of a three-dimensional shape. It is an example of the figure which shows the coordinate system of an imaging means typically. It is an example of the figure explaining the perspective projection matrix at the time of moving a projector virtually.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In the present embodiment, a non-limiting exemplary embodiment of the present invention will be described using an information processing apparatus that captures an image of a projection target onto which a projection pattern is projected and corrects the projection image using the captured pattern image. To do.

  In the present embodiment, the image includes a still image, a moving image, a continuous still image, and the like. Audio may or may not accompany. Projecting an image includes projection, projection, projection, irradiation, and the like.

  Taking an image includes taking and acquiring images. Projection objects include those that can project an image on a screen, a wall, an outer surface such as a white boat, an outer wall of a building, and the surface of a moving object.

  In the present embodiment, the projected image refers to an image without distortion projected by the projector 100, the captured image refers to an image with distortion projected on the projection object, and the corrected image is distorted when projected onto the projection object. Refers to a projection image that has been corrected so as to be reduced. The projection image includes a projection pattern for calculating a correction parameter, and the captured image includes a pattern image obtained by capturing the projection pattern.

[Projector, projection operation, and imaging operation]
A projector 100 according to the present invention will be described with reference to FIGS. FIG. 1 is a diagram illustrating a configuration example of an image correction system 200 in which the projector according to the present embodiment is used. 2A and 2B are a schematic external view of the front and a schematic external view of the back showing an example of the projector 100 according to the present embodiment. FIG. 3 is an example of a schematic external view of the information processing apparatus 99. FIG. 4 is an explanatory diagram for explaining an example of the projection operation of the projector 100. FIG. 5A is an explanatory diagram illustrating an example of an operation in which the projector 100 calculates a correction parameter. FIG. 5B is an explanatory diagram illustrating an example of an operation in which the projector 100 corrects a projection image. FIG. 6 is an explanatory diagram for explaining an example of an operation of projecting a projection image (corrected image) corrected by the projector 100.

As shown in FIG. 1, the image correction system 200 includes a projector 100 and an information processing device 99. The image correction system 200 operates as follows.
(i) The projector 100 projects a projection pattern.
(ii) The information processing apparatus 99 captures the projected pattern image.
(iii) The information processing apparatus 99 calculates a projective transformation matrix H.
(iv) The information processing apparatus 99 corrects the projected image with the projective transformation matrix H (the corrected projected image is referred to as a corrected image).
(v) The projector 100 projects a corrected image.

  A configuration example of the image correction system 200 or a configuration in which the projector 100 instead of the information processing apparatus 99 calculates the projective transformation matrix H is possible. Such a configuration will be described later in a fourth embodiment.

  The information processing apparatus 99 only needs to have an imaging means and a function for calculating the projective transformation matrix H. For example, a tablet-type information processing apparatus, a smartphone, a notebook PC (Personal Computer), a PDA (Personal Digital Assistant), a mobile phone Any device such as a telephone or an electronic book terminal may be used.

  As illustrated in FIG. 2A, the projector 100 includes a projection unit 100 </ b> PJ that projects a projection image on the front surface of the projector 100. Further, as shown in FIG. 2B, the projector 100 has a start button 100Ba on the back of the projector 100 for inputting a desired operation execution timing and a setting button 100Bb for setting a desired operation by the user. A selection button 100Bc for the user to select desired information.

  2A and 2B are merely external views simplified for the sake of explanation, and the external appearance and the like of the projector that can use this embodiment are not limited to those shown in FIG. . In addition to the hardware keys, the projector 100 may have soft keys that project a menu screen from the projection unit 100PJ and accept menu selections as shown in FIG. Operation equivalent to or higher than that of the setting button 100Bb and the selection button 100Bc can be received by the soft key. Further, the projector 100 may have a display device such as a liquid crystal, and may accept an operation from a touch panel integrated with the display device.

  As illustrated in FIG. 3, the information processing apparatus 99 includes an imaging unit 110 that includes a lens and an imaging device for imaging a projection target, and an operation button 106 that receives a user operation such as imaging. Note that the imaging unit 110 may be provided on the back side, or may be provided on both the front side and the back side. The appearance of the information processing apparatus 99 is mainly a tablet-type information processing apparatus 99, and is not limited to the illustrated appearance. The display 120 of the information processing apparatus 99 is integrated with the touch panel and can accept user operations in addition to imaging.

  As shown in FIG. 4, the projector 100 projects an image on a screen or the like (hereinafter referred to as “projection object”). For example, when the user presses the start button 100Ba, the projector 100 starts projecting an image. In FIG. 4, the projector 100 projects an image onto the projection region Rng-PJ using the projection unit 100PJ.

  Further, in FIG. 4, the information processing apparatus 99 images the imaging region Rng-CM (projected region) using the imaging unit 110. The information processing apparatus 99 starts capturing an image when the user presses the operation button 106, for example.

  Further, in FIG. 4, the projector 100 can accept selection of a projection target region Rng-Tg as a region on which a projection image is projected. In the projector 100, for example, the size and position of the projection target region Rng-Tg are selected by the user using the selection button 100Bc, and the projection target region Rng-Tg is set using the setting button 100Bb.

  As illustrated in FIG. 5A, the information processing apparatus 99 captures, for example, a captured image Img-CM illustrated in the drawing when the imaging unit Rng-CM is captured using the imaging unit 110. That is, the imaging unit 110 captures a captured image Img-CM that is deformed corresponding to the shape (for example, the outer shape) of the projection target. The information processing apparatus 99 calculates a projective transformation matrix H described later for distortion correction so that the captured image Img-CM becomes the captured image Img-CMr.

  As illustrated in FIG. 5B, the information processing apparatus 99 corrects the projection image Img-PJ using the calculated projective transformation matrix H, and newly generates a corrected image Img-RV. The information processing apparatus 99 calculates the projective transformation matrix H automatically, for example, after the captured image Img-CM is captured or when the user presses the operation button 106.

  As shown in FIG. 6, the projector 100 uses the newly generated corrected image Img-RV as the projected image Img-PJ. That is, the projector 100 projects the corrected image Img-RV, thereby projecting a projection image Img-PJ that cancels the deformation corresponding to the shape of the projection target.

  Hereinafter, the configuration, function, and operation of the projector 100 according to the embodiment of the present invention will be specifically described.

  FIG. 7 is an example of a diagram illustrating the hardware configuration of the projector 100 and the information processing apparatus 99.

  First, the projector 100 includes a communication unit 35, a projection control unit 36, a storage unit 37, a projection lens 31, a color wheel 32, a DMD (Digital microMirror Device) 33, and a light source 34.

  The communication unit 35 communicates with the information processing apparatus 99 and receives a projection pattern, a corrected image, or the like. Any communication implementation method may be used. For example, wireless LAN, Bluetooth (registered trademark), infrared communication, IC communication, WiGig (Wireless Gigabit), or the like is used. In addition, you may communicate by wire, such as wired LAN and a USB cable.

  The projection control unit 36 operates as a control unit that controls the entire projector 100. For example, when the projector 100 receives a projection pattern or a correction image, or when the user presses the start button 100Ba, for example, the DMD 33 is controlled based on the projection pattern or the correction image, and the projection image (the projection image as the correction image is displayed). Project) onto the screen.

  The storage unit 37 stores the projection image (including the projection image as the correction image) received from the information processing apparatus 99.

  The light source 34, the DMD 33, the color wheel 32, and the projection lens 31 are DLP (Digital Light Processing) type projection means. The white light emitted from the light source 34 is applied to the DMD 33 whose ON / OFF is controlled by the projection image decomposed into RGB, passes through the color wheel 32, and is condensed by the projection lens 31, and then the projection object. Projected on. Note that the projection unit may be realized by an LCD method or the like.

  The information processing apparatus 99 includes a CPU 101, a ROM 102, a RAM 103, an SSD (Solid State Drive) 104, a media drive 105, an operation button 106, a power switch 107, a telephone communication unit 108, a network I / F 109, an imaging unit 110, a microphone 111, a speaker. 112, a display I / F 113, an external device I / F 114, and a GPS receiver 115.

  The CPU 101 executes a program 116 stored in the SSD 104 and controls the operation of the entire information processing apparatus. The ROM 102 stores, for example, a startup program and user settings. The RAM 103 is used as a work area for the CPU 101. The SSD 104 stores a program 116. The program 116 is distributed in the form of being downloaded from a server (not shown), or is distributed in a state stored in the recording medium 117.

  The media drive 105 controls reading or writing of data with respect to the recording medium 117. The recording medium 117 is a non-volatile memory such as a flash memory. The operation button 106 is a button for accepting an operation on the information processing apparatus 99 by the user, and includes a touch panel described later. The power switch 107 is a switch for switching on / off the power of the information processing apparatus 99. The telephone communication unit 108 communicates with an information processing apparatus 99 such as a server or another information processing apparatus via a base station such as a mobile phone network. Audio data and various data are transmitted and received by communication.

  The network I / F 109 is a communication card that communicates with the communication unit 35 of the projector 100 via an access point such as a wireless LAN network. A corrected image or the like is transmitted by communication. The imaging unit 110 captures the captured image Img-CM according to the control of the CPU 101. The microphone 111 collects sound and converts it into an electrical signal. The speaker 112 outputs sound.

  Display I / F 113 outputs captured image Img-CM to display 120 under the control of CPU 101. The display 120 is configured using a liquid crystal element, an organic EL element, or the like. The display 120 integrally has a touch panel, displays operation icons and the like, and detects user operations.

  The external device I / F 114 is, for example, a USB host, Bluetooth (registered trademark), NFC (Near Field Communication), TransferJet (registered trademark), or the like that transmits and receives various data to and from an external device. A GPS (Global Positioning System) receiving unit 115 receives radio waves from GPS satellites and estimates the position of the information processing apparatus 99.

  In addition, a bus line 118 such as an address bus or a data bus is provided for electrically connecting the above components. The recording medium 117 is detachable from the information processing apparatus 99.

  Note that the imaging unit 110 may be externally connected via the external device I / F 114, for example, in addition to being built in the information processing device 99.

  FIG. 8 is a diagram illustrating an example of a functional block diagram of the information processing apparatus 99. The information processing apparatus 99 includes a pattern imaging unit 11, a communication unit 12, a binarization unit 13, a pattern selection unit 14, a pattern interpolation unit 15, an image processing unit 16, an imaging control unit 17, a corresponding point extraction unit 18, and a missing part extraction. A unit 19, a correction coefficient calculation unit 20, a projection target imaging unit 21, and a storage unit 22. Each of these functions is a function or means realized by the CPU 101 in FIG. 7 executing the program 116 and cooperating with hardware.

  The pattern imaging unit 11 captures an image of the projection pattern projected onto the projection target using the imaging unit 110 and creates a pattern image.

  The communication unit 12 is a function or means realized by the network I / F 109 and communicates with the communication unit 35 of the projector 100 to transmit a corrected image in which the projection pattern and distortion are corrected to the projector 100. The communication method may be communication via a network in addition to one-to-one communication.

  The binarization unit 13 binarizes the pattern image. The pattern selection unit 14 selects a horizontal line projection pattern and a vertical line projection pattern, and causes the communication unit 12 to transmit the projection pattern to the projector 100. Also, the binarization unit 13 selects whether the binarized pattern image is a horizontal line projection pattern or a vertical line projection pattern.

  The pattern interpolation unit 15 interpolates a horizontal line or vertical line missing part of the pattern image binarized by the binarization unit 13. The image processing unit 16 corrects the projection image using the projective transformation matrix H calculated by the correction coefficient calculation unit 20.

  For example, when detecting the operation of the operation button 106 by the user, the imaging control unit 17 causes the pattern imaging unit 11 to capture a projection pattern and create a captured image (which will be described later).

  The corresponding point extraction unit 18 combines the captured image of the horizontal line and the vertical line in which the missing part is interpolated, and obtains the intersection of the horizontal line and the vertical line in the captured image. Further, this intersection point is associated with the intersection point of the horizontal line and the vertical line of the projection image.

  The missing part extraction unit 19 extracts a missing part of a horizontal line or a vertical line of the captured image binarized by the binarization unit.

  The correction coefficient calculation unit 20 calculates a projective transformation matrix H that corrects the projection image so that the projection image is not distorted by the projection target. The projection target imaging unit 21 images the projection content that is the projection target.

  The storage unit 22 is a function or means realized by one or more of the RAM 103, the SSD 104, or the recording medium 117 in FIG. The storage unit 22 stores a projection image (including a projection pattern) and a binary code described later. Further, the storage unit 22 stores a pattern image that is a captured image obtained by capturing the projection pattern. In addition, projection content and projection images are stored.

<Pattern image>
The pattern image will be described with reference to FIGS. The pattern image is used to create a projective transformation matrix H (to correct distortion of the projection image on the projection target). FIG. 9A shows an example of projection patterns A and B projected as projection images. In the projection pattern A, a group of lines in which vertical line segments having the same length are arranged at equal intervals form a pattern. In the projection pattern B, a line group in which line segments of the same horizontal length are arranged at equal intervals forms a pattern. Further, the horizontal line and the vertical line have lines superimposed in the longitudinal direction.

  FIG. 9B illustrates an example of a pattern image that is a captured image obtained by capturing an image of the projection pattern by the pattern imaging unit 11. The pattern image A is obtained by capturing the projection pattern A. The pattern image B is an image of the projection pattern B.

  FIG. 10A shows an example of a combined pattern of projection patterns A and B. FIG. By combining the projection patterns A and B, a vertical line segment and a horizontal line segment intersect, and a lattice-shaped combined pattern in which a plurality of rectangles are arranged vertically and horizontally is obtained. In other words, the projection patterns A and B are obtained by taking out a vertical line segment and a horizontal line segment from the composite pattern in FIG. One of the grids may be referred to as a mesh.

  FIG. 10B is a diagram illustrating an example of a combined pattern image in which the pattern image A and the pattern image B are combined. Because the pattern images A and B are distorted due to the projections and depressions of the projection object, the lattice-shaped composite pattern image is distorted.

  In this embodiment, the intersection of the lattice of the composite pattern image corresponding to the intersection of the lattice of the composite pattern is specified, and the projective transformation matrix H is created.

  Note that the projection pattern A and the projection pattern B do not need to be parallel lines or at equal intervals, and the projection pattern A and the projection pattern B only need to intersect. If corresponding points can be extracted by obtaining several intersections, projective transformation is possible.

<Binary code>
The binary code will be described with reference to FIG. FIG. 11 is a diagram illustrating an example of a binary code. The dotted lines in FIG. 11 are explanatory lines that do not actually exist. 11A shows the binary code A, FIG. 11B shows the binary code B, and FIG. 11C shows the binary code C. The binary code is a binary image with black and white stripes, and is used to determine a region for detecting horizontal and vertical lines.

  The binary codes A to C are black and white stripes. The binary code A is a stripe having an upper half black pixel and a lower half white pixel in an area having the same size as the projection pattern. The binary code B is a stripe in which the black pixels of the binary code A are equally divided into black and white and the white pixels are equally divided into black and white. The binary code C is a stripe obtained by equally dividing the black pixels of the binary code B into black and white and equally dividing white pixels into black and white. In this way, the binary codes A to C are fringe image data whose density increases by a factor of two.

  In FIG. 11, the binary codes A to C are horizontal stripes, but vertical stripes can be obtained by rotating 90 degrees. In the following, since the binary codes A to C are horizontal stripes, the detection of the horizontal line will be described as an example, but the vertical line is also detected in the same manner.

  First, each horizontal line of the projection pattern B in FIG. 9A is located at the boundary of the finest binary code C stripes. That is, each horizontal line of the projection pattern B is formed at the boundary of the binary code C stripes (or each horizontal line of the projection pattern B is formed at the boundary of the binary code C stripes). Thereby, the information processing apparatus 99 can estimate the position of the horizontal line or the vertical line by using the binary codes A to C.

The binary codes A to C have a width that is an integral multiple of the line width of the projection pattern B. The binary code A is 4 times, the binary code B is 2 times, and the binary code A is 1 time. Therefore, when the binary codes A to C are superimposed, a monochrome pattern that does not overlap in the overlapping direction is obtained. One of the black and white patterns is called a band area. According to the binary codes A to C, band regions 1 to 8 are obtained. The black and white patterns of the band regions 1 to 8 obtained by the binary codes A to C are as follows.
Belt region 1: Black White Black belt region 2: Black White White belt region 3: Black Black Black belt region 4: Black Black White belt region 5: White White Black belt region 6: White White White belt region 7: White Black Black belt Region 8: White Black White Therefore, it can be expected that a horizontal line exists at a position where the monochrome pattern is switched. A line pattern region 61 of FIG. 17 is formed at a position where a horizontal line is expected to exist.

<Distortion of captured image>
The distortion of the captured image will be described with reference to FIG. FIG. 12 is an example of a diagram for explaining distortion of a projected image projected by the projector 100. FIG. 12A shows an example in which the short focus (or ultra short focus) projector 100 projects a projection image onto the projection target Scr. FIG. 12B shows an example of a captured image obtained by capturing a projection image projected onto the projection object Scr by the short-focus (or ultra-short focus) projector 100 directly facing the projection object Scr. FIG. 12C shows an example in which the projector 100 with a normal focus (or focal length) projects a projection image onto the projection object Scr. FIG. 12D shows an example of a captured image obtained by capturing the projection image projected onto the projection object Scr by the projector 100 having a normal focus (or focal distance) directly facing the projection object Scr.

  As shown in FIG. 12C, in the projector 100 having a normal focal length, when the projection light (projection images) La, Lb, and Lc is irradiated (projected) onto the projection surface of the projection object Scr, the projection light La, Lb and Lc are reflected by the reflected light Lar, Lbr and Lcr. That is, the normal projector 100 is not significantly affected by the shape (distortion) of the projection surface of the projection target as shown in FIG.

  On the other hand, as shown in FIG. 12A, in the short-focus projector 100, when the projection light (projection images) L1, L2, and L3 is irradiated (projected) onto the projection surface of the projection target, the projection target Depending on the shape (distortion) of the projection surface of the object, the projection light L2 is reflected by the reflected light L2ra. That is, in the short-focus projector 100, the reflection position of the projection light L2 is shifted depending on the shape (distortion) of the projection surface of the projection object, and the reflected light L2r is reflected instead of the reflected light L2ra. As a result, as shown in FIG. 12B, in the short-focus projector 100, the local portion of the captured image Img-CM (from the viewer, for example) is displayed from L2ra at the position facing the projection object Scr. Distorted to L2r. In the projector 100, since the incident angle of the projection light becomes small particularly in the case of a short focal point, the reflection position shift (image distortion) increases even with a slight distortion of the projection surface.

  The projector 100 according to the present embodiment calculates a correction parameter that cancels out local distortion of the captured image Img-CM as a correction parameter. That is, the projector 100 calculates a correction parameter that is deformed so that the distortion L2r in FIG. This correction parameter is the projective transformation matrix H.

<Detection of pattern image>
FIG. 13 is a diagram illustrating an example of a captured image on which a projection pattern is projected. The captured image includes a pattern image B, a screen 52, a storage unit 54 that stores the screen 52, a clock 55, a calendar 56, and the like.

  In FIG. 13, the projection pattern is surrounded by a square black frame 51. Considering the case where such a projection pattern is projected on the screen 52 suspended from the wall, the projection pattern B is included in the area surrounded by the circumscribed rectangle (black frame 51) of the connected component near the center of the captured image. It is. If the black frame 51 of the pattern image B is larger than a predetermined size, it can be estimated that a recognizable pattern element has been captured. Naturally, the user may project the projection pattern B on the center of the screen 52 and capture the pattern image B so as to occupy the center of the captured image.

  Therefore, for example, if a black pixel connected component (a portion of the black frame 51) having a predetermined number of pixels or more can be detected, it can be detected as the black frame 51, and the pattern image B in the black frame can be stably extracted.

  The black frame 51 may be a circle or an ellipse instead of a rectangle. Further, the captured image captured by projecting the projection pattern B is compared with the captured image captured in a state where the projection pattern B is not projected, and the pattern image B is extracted by comparing the two images. May be.

<Preparation stage>
Hereinafter, the creation of the projective transformation matrix H will be described separately in a preparation stage and a projection stage.

FIG. 14 is an example of a flowchart showing the preparation stage procedure.
S1: The projection control means 36 of the projector 100 projects a pattern image. Since the information processing apparatus 99 stores the pattern image, the communication unit 12 of the information processing apparatus 99 transmits it to the projector 100.
S2: For example, when the user presses the operation button 106, the pattern imaging unit 11 of the information processing apparatus 99 captures an area including the projection pattern. The information processing apparatus 99 is fixed by a tripod or the like, and the imaging position is fixed while the projection patterns A and B and the binary codes A to C are imaged.
S3: The pattern imaging unit 11 stores the captured image (pattern image) in the storage unit 22. The pattern imaging unit 11 repeats steps S2 and S3 to capture and store the projection patterns A and B and the binary codes A to C.
S4: The corresponding point extraction unit 18 associates the intersection of the combined pattern image obtained by combining the pattern images A and B with the intersection of the horizontal line and the vertical line of the combined pattern obtained by combining the projection patterns A and B. Record the pair that associates coordinate values. This process is called corresponding point extraction. Details of the corresponding point extraction will be described later.
S5: Assume that each intersection of the composite pattern image and the composite pattern is a vertex, and obtain a projective transformation coefficient (projective transformation matrix H) between regions surrounded by the corresponding four points. This coefficient is an image conversion coefficient and may be called a correction coefficient or a correction parameter.
S6: The pattern images A and B are ideally captured as shown in FIG. 9A, but are projected in consideration of the case where the pattern is distorted as shown in FIG. The image is corrected using the projective transformation matrix H. That is, conceptually, by applying distortion in the reverse direction to the projection image, the distortion due to the unevenness of the screen cancels out, and a projection image without distortion is projected. Details will be described later.
S7: The projective transformation matrix H is stored in the storage unit 22 of the information processing apparatus 99.

<Projection stage>
FIG. 15 is an example of a flowchart showing the procedure of the projection stage.
S11: The image processing unit 16 reads the projective transformation matrix H stored in the storage unit 22.
S12: The projection target imaging unit 21 images the projection content prepared in the information processing apparatus 99 into a projection image. The projection content is, for example, a still image, a moving image, a continuous still image, or the like.
S13: The image processing unit 16 processes (deforms) the projection image according to the projective transformation matrix H.
S14: The communication unit 12 transmits the deformed projection image to the projector 100 in order to cause the projector 100 to project the deformed projection image.
S15: The projector 100 stores the received projection image in the storage unit 37.
S16: The projection control means 36 of the projector 100 controls the light source 34, the DMD 33, and the color wheel 32, and projects a projection image onto the projection object through the projection lens 31. With the above procedure, the user can recognize a projection image with reduced distortion.

[Extraction of corresponding points]
FIG. 16 is an example of a flowchart for explaining the procedure of corresponding point extraction processing in step S4 of FIG. When the image is captured without losing the horizontal and vertical lines while being distorted as in the pattern images A and B in FIG. 9B, the distortion can be corrected by the feature point extraction and the projective transformation matrix H. However, when part of the pattern images A and B is lost, there is a possibility that the corresponding area cannot be correctly associated between the projection pattern and the pattern image.

  Therefore, in the procedure of FIG. 16, feature points are extracted after interpolating the missing portion so that the projective transformation matrix H can be created even if part of the pattern images A and B is missing.

  Note that the processing in FIG. 16 is performed in the order of the projection pattern B (horizontal line) and the projection pattern A (vertical line).

  The corresponding point extraction unit 18 binarizes the captured images of the binary codes A to C, and creates binary images for the number of binary codes A to C (S4-1).

  Next, band regions 1 to 8 are specified from a set of three binarized binary code images (S4-2). As described above, when the binary codes A to C are overlapped, black and white patterns that do not overlap are obtained, so that the band regions 1 to 8 can be specified.

  Next, a boundary portion of the band regions 1 to 8 is determined, and a range widened up and down by a predetermined number of pixels from the boundary portion is set as the line pattern region 61 (S4-3).

  FIG. 17A is a diagram for explaining an example of the line pattern region 61. The predetermined number of pixels is set as a range in which the horizontal line can move due to distortion. As shown in the figure, the line pattern region 61 is set around the boundary between the band regions 1 to 8. In many cases, a horizontal line is detected at the boundary between the band regions 1 to 8, but the horizontal line does not necessarily exist at the boundary due to the binarization result of the pattern image described below and the influence of external light during photographing. There is a case. For this reason, the line pattern region 61 is determined as a range in which the horizontal line can move with respect to the boundaries of the band regions 1 to 8 determined by the binary code. Then, a horizontal line is detected in the line pattern area 61.

  Returning to FIG. 16, the corresponding point extraction unit binarizes the pattern image B (S4-4). Binarization is performed on black pixels below a predetermined threshold and white pixels above the threshold.

  Next, the corresponding point extraction unit 18 extracts black pixels in the line pattern region 61 and extracts continuous black pixels (S4-5). Consecutive black pixels start from an arbitrary black pixel and detect and connect black pixels among adjacent pixels. If there is a black pixel, the process of detecting the black pixel among the pixels adjacent to the pixel is repeated. If there is no black pixel in the adjacent pixel, the same processing is repeated from the black pixel that is not yet connected. Thereby, a horizontal line is detected. When the horizontal line is missing, two or more horizontal lines may be detected.

  Next, the missing part extracting unit 19 detects the line pattern area 61 in which the horizontal line is missing from the line pattern area 61 at the boundary of the band areas 1 to 8 (S4-6).

FIG. 17B is an example of a diagram illustrating determination of the presence / absence of a defect. By detecting the continuous black pixels, it is estimated that the black pixels corresponding to the horizontal lines are continuous in the longitudinal direction in the line pattern region 61. The missing part extraction unit 19 searches for a black pixel from the start or end of the line pattern region 61. The determination criterion of the defect is set as appropriate. For example, it is determined that the defect is present when there is no black pixel in the width direction (row direction) of the line pattern region 61.
(i) There are black pixels in two rows from the first column to the fifth column.
(ii) The sixth column has black pixels in the third row from the top.
(iii) There is a black pixel in the fourth row from the top in the seventh column.
(iv) There is a black pixel in the sixth row from the top in the eighth column.
(v) Since there is no black pixel in the ninth column, the missing part extracting unit 19 determines that the horizontal line is missing in the ninth column.

  Thereafter, the search is further performed in the column direction. Since there is a black pixel in the 12th column, the search for the horizontal line is resumed from the 12th column. Therefore, it is determined that the ninth to eleventh columns are missing portions. In this way, all missing portions are detected for each horizontal line.

  In addition, as the determination of the missing part, it may be determined that there is a defect when the number of columns without black pixels is equal to or greater than a predetermined number. Even if there is a black pixel in one column, it may be determined that there is a defect if there is no black pixel in the same row as the previous column.

  FIG. 18A is a diagram illustrating an example of a missing part 303 of a horizontal line partially missing. In FIG. 18A, the horizontal line between the band regions 4 and 5 is partially missing. As described above, when there is an interrupted portion, the pattern interpolation unit 15 interpolates the missing portion using the horizontal lines 301 and 302 other than the missing portion.

  Returning to FIG. 16, the pattern interpolation unit 15 interpolates the broken horizontal line to be interpolated (S4-7). That is, the end points of the continuous black pixels (the end points of the horizontal lines 301 and 302) are connected by a straight line. As a result, the pattern image is an image in which all line segments are photographed without any defects. FIG. 18B shows an example of a pattern image in which the missing portion 303 is interpolated. In FIG. 18B, interpolation is performed with a straight line. Since interpolation by straight lines only needs to connect the end points, the processing load can be reduced.

  Next, the corresponding point extraction unit 18 determines whether or not all missing portions have been interpolated for the focused line pattern region 61 (S4-8).

  When a missing part remains in the focused line pattern region 61 (No in S4-8), the missing part is interpolated. That is, when there are a plurality of missing portions, interpolation is performed for all the missing portions.

  Next, the corresponding point extraction unit 18 determines whether or not the missing portion has been interpolated for all the line pattern regions 61 (S4-9).

  When the missing portion is not interpolated for all the line pattern areas 61 (No in S4-9), the process is repeated from the process of detecting continuous black pixels for the next line pattern area 61 (S4-5).

  Next, the corresponding point extraction unit 18 performs the same process on the vertical line (S4-10). That is, a vertical line is detected from the pattern image A in which the light projection pattern A is captured, and the missing portion is interpolated. The binary codes A to C can correspond to a vertical line by rotating by 90 °.

  Next, the corresponding point extraction unit 18 combines the pattern images A and B (S4-11). The composition is performed by comparing pixel values of the same coordinates in the horizontal line pattern image A and the vertical line pattern image B and replacing both with white pixels if they are white pixels, and replacing them with black pixels if one is a black pixel. Thereby, a composite pattern image as shown in FIGS. 10A and 10B is obtained. Note that both ends of the horizontal line and the vertical line may be extended before synthesis so that the end of the horizontal line intersects the vertical line, or the end of the vertical line intersects the horizontal line.

  Next, the corresponding point extraction unit 18 obtains the intersection of the horizontal line and the vertical line, and extracts the coordinates as feature points (S4-12).

  FIG. 19A is an example of a diagram illustrating the detection of the intersection. There are various ways of obtaining the intersection point. For example, the coordinates of the black pixels of the pattern image A and the pattern image B are used. Since the coordinates of the black pixels (horizontal line) of the pattern image A and the coordinates of the black pixels (vertical line) of the pattern image B are known, the overlapping point in both the pattern image A and the pattern B is the intersection of the black pixels. (Black square in the figure).

  For example, taking coordinates as shown in the figure, (x, y) = (1, 1) (1, 2) (2, 1) (2, 2) is one of the intersections. Since the overlapping point is not limited to one pixel, for example, the center of gravity (the average of a plurality of overlapping points x-coordinates and the average of y-coordinates) is determined as the intersection.

  By performing the same process on the composite pattern, the feature point can be detected from the composite pattern. Note that feature points may be detected in advance because no distortion or the like occurs in the composite pattern.

The corresponding point extraction unit 18 outputs the intersection as a feature point to the correction coefficient calculation unit 20 (S4-13).
As described above, the intersection point of the composite pattern image as shown in FIG. 10B is obtained as the feature point.

<Associating intersections>
The corresponding point extraction unit 18 associates the coordinates of the feature points of the composite pattern image with the coordinates of the intersection of the composite pattern in FIG. Specifically, for example, the coordinates of the feature points of the composite pattern image are sequentially read in the X direction along the horizontal line from the start end of the horizontal line, and the coordinates of the intersection point may be sequentially associated with the coordinates of the intersection point from the start end of the composite pattern. When the rightmost feature point (end) is reached, the next horizontal line is similarly associated.

  FIG. 19B is an example for explaining the correspondence between the coordinates of the feature points of the composite pattern image and the coordinates of the feature points of the composite pattern of the projection patterns A and B. Since there are nine horizontal lines and nine vertical lines, the coordinates of “81 = 9 × 9” are associated. By making the feature points correspond to each other, the regions surrounded by the four feature points can be made to correspond to each other, and the projective transformation matrix H can be calculated.

  If the number of pixels of the composite pattern image does not match the number of pixels of the composite pattern, the number of pixels of the composite pattern image is matched with the number of pixels of the composite pattern, and both are normalized to the same size.

[Calculation of projective transformation matrix H]
FIG. 20 is an example of a diagram for explaining the calculation of the projective transformation matrix H. First, the correction coefficient calculation unit 20 performs association for each region based on the correspondence of the feature points in FIG. The correspondence between one mesh of the composite pattern image and one mesh of the composite pattern is described by projective transformation. If the point of the pattern image is mp = (xp, yp) and the point of the captured image is mc (xc, yc), the point mp can be expressed as follows using projective transformation using mc.

ｈ１〜ｈ８は未知の射影変換パラメータである。また、同次座標を用いて次のように表現できる。

h1 to h8 are unknown projective transformation parameters. Also, it can be expressed as follows using homogeneous coordinates.

Hが射影変換行列である。

H is a projective transformation matrix.

  Further, the following constraint is obtained from one corresponding point between the pattern elements of the pattern image and the pattern elements of the projection pattern image.

この式により、８つの未知パラメータに対して２つの拘束が得られる。したがって、１つのメッシュの頂点である４つの対応点が得られればＨの全ての要素を決定できる。射影変換の式は、全画素の対応関係を記録したテーブルとして保持しておくことができる。

This equation gives two constraints on the eight unknown parameters. Therefore, if four corresponding points that are the vertices of one mesh are obtained, all elements of H can be determined. The expression for projective transformation can be held as a table in which the correspondence of all pixels is recorded.

  FIG. 21 is an example of a diagram illustrating correction of a projected image. The projection image of the content to be projected is corrected by deforming the projection image projected by the projector 100 in advance. First, the correction coefficient calculation unit 20 determines the projection area so that the pattern image area is rectangular in the captured image. The projection area is the maximum inscribed rectangle of the pattern image. However, since it is preferable to have the same aspect ratio as the projection object, the aspect ratio is actually smaller than the maximum inscribed rectangle. Therefore, the corrected projection image becomes smaller than the original projection image.

Hereinafter, a procedure for reading and determining the pixel value of the point A on the corrected image from the projection image will be described.
(1) First, the position on the captured image where the point A corresponds is determined. This point is called a point Ac. Since the coordinates (xc, yc) of the point Ac can be converted to the point Ap of the pattern image by the projective transformation, the coordinates of the point Ap can be converted to the coordinates of the point Ac of the captured image contrary to the projective transformation. The coordinates (xc, yc) of the point Ac are coordinates in the coordinate system with the upper left corner of the captured image as the origin (0, 0).
(2) Next, it is calculated which position in the projection area the point Ac of the obtained captured image corresponds to. The upper left of the projection area is the origin (x0, y0). The coordinates of the point Ac with respect to this origin are (xc1, yc1).
(3) Let the coordinates of the point Ap of the original image (projection image) corresponding to the point Ac be (xp, yp). Since the projection area has the same aspect ratio as the original image and is different only in size, the coordinates of the point Ap are as follows from the horizontal length Wp of the projection image and the horizontal length Wc of the projection area of the captured image: It can be expressed as
(Xp, yp) = Wp / Wc (xc1, yc1) (a)
(4) Since the origin of the projection area in the captured image is (x0, y0), the coordinate value of the point Ac can be expressed as follows.
(Xc, yc) = (x0, y0) + (xc1, yc1) (b)
(5) Since the coordinates (xc, yc) of the point Ac are obtained, (xc1, yc1) is obtained from the equation (b), and the coordinates (xp, yp) of the point Ap are obtained from the equation (a). Can do. The pixel value of the coordinates (xp, yp) is set as the pixel value of the point A of the corrected image.

  Note that the coordinates of the point Ap corresponding to the point A are not necessarily integers. In this case, the coordinates are obtained by interpolation from the pixel values of the pixels around the point Ap.

  Therefore, the projection transformation formula, projection area, formulas (a) and (b), and the like correspond to the correction information. The calculation of correction information is described in “Experimental study on geometric correction of projected image”, Society of Instrument and Control Engineers Tohoku Branch 235th Research Meeting (2007.5.18).

  As described above, since the information processing apparatus 99 according to the present embodiment calculates the correction coefficient of the projection image using a projection pattern having a plurality of lines instead of a point pattern, interpolation is possible even if there is a missing portion. Further, although the points of the point pattern are discrete, the lines are continuous, so even if they are missing, interpolation can be performed using the end points of the nearby lines. Therefore, the accuracy of correction is improved, and the projection transformation matrix H can be calculated stably even if the screen has irregularities or the projection pattern cannot be photographed with high image quality.

  In the first embodiment, the missing portion is interpolated by a straight line. In this embodiment, an information processing apparatus 99 connected by a cubic spline curve will be described. The spline curve is characterized by smooth interpolation. Screen distortion is caused by smooth fluctuations as long as there is no extreme bend, so there is no sudden direction change (bending of the line pattern) at the end point that tends to occur in the case of linear interpolation, improving the accuracy of correction. Can be made.

  In the present embodiment, the components described in the first embodiment perform the same function, and therefore, only the main components of the present embodiment may be mainly described.

  FIG. 22 is an example of a flowchart for explaining the procedure of the feature point extraction process in step S4 of FIG. In FIG. 22, the process of step S4-7 in FIG. 16 is changed to S4-71 to 4-73.

  If there is a missing part in the line segment of the line pattern area 61 in step S4-6, the pattern interpolation part 15 obtains the coordinate values of the part in front of the missing part one by one (two in total) (S4-71).

  FIG. 23A is an example of a diagram illustrating the coordinate value of a point in front of the missing part. This point is referred to as a coordinate acquisition point. A point further to the left of the left end point 305 centering on the missing portion is the coordinate acquisition point 311, and a point further to the right of the right end point 306 centering on the missing portion is the coordinate acquisition point 312.

  How far away from the end point is the coordinate acquisition point is, for example, about the distance between adjacent lines of the projection pattern. If there is no black pixel at this distance, the black pixel closest to the position is selected.

Next, the pattern interpolation unit 15 obtains a cubic spline curve by using a total of four coordinates of the two end points 305 and 306 and the two coordinate acquisition points 311 (S4-72). A cubic spline curve is based on the idea that three sections between four coordinates are connected by independent cubic curves. Since there are three sections to be obtained (of which the middle section is to be interpolated), it is assumed that three cubic straight-line equations passing through both ends of each partial section are obtained. Each straight line equation
y = a1x ^ 3 + b1x ^ 2 + c1x + d1
y = a2x ^ 3 + b2x ^ 2 + c2x + d2
y = a3x ^ 3 + b3x ^ 2 + c3x + d3
Using the fact that the coordinate values match at the boundary points of each section (in this case, the end points and the coordinate acquisition points), a1 ~ a3, b1 ~ b3, c1 ~ c3, d1 ~ d3 Establish an equation. As a result, six equations are obtained.

  Subsequently, an equation is established so that the respective derivatives and second-order derivatives coincide at the end points or the coordinate acquisition points so that the three curves are smoothly connected at the end points and the coordinate acquisition points. This gives four equations. Since the number of formulas is not enough for the number of variables, a condition is added in which the second derivative before and after the end point becomes zero. As a result, twelve equations can be obtained, so that the variables a1 to a3, b1 to b3, c1 to c3, and d1 to d3 can be determined by solving the 12-ary linear simultaneous equations.

  Note that the spline curve may not be cubic but may be quaternary or higher, and the number of coordinate acquisition points when calculating the cubic spline curve may be increased.

  The pattern interpolation unit 15 interpolates the missing part using the missing part expression among the obtained three curve equations (S4-73). That is, the missing portions of the pattern images A and B are filled with black pixels of spline straight lines. The subsequent processing is the same as step S4-8 in FIG.

  FIG. 23B is an example of a diagram illustrating an example of interpolation of a missing portion by a spline curve. By interpolating with the spline curve, the horizontal lines 301 and 302 can be smoothly interpolated. In addition to the spline curve, Lagrange interpolation or Hermite interpolation may be used.

  Further, in FIG. 23A, it is assumed that a horizontal line is detected on the left and right of the missing part, but there may be a case where there is no horizontal line at one end point. That is, there is only one end point even if the missing portion 303 exists.

  FIG. 24 is a diagram illustrating an example of a horizontal line having no end point on the right side of the horizontal line defect portion 303. In this case, as shown in FIG. 24A, three coordinate acquisition points 313 are obtained in addition to the end points on the side where the horizontal line exists, and the same processing is performed. Then, as shown in FIG. 24B, extrapolation is performed to the end of the X coordinate of the horizontal line by the spline curve in the outermost (end point side) section (between the rightmost coordinate acquisition point 313 and the end point 305). Do. By doing so, it is possible to perform interpolation even if there is no end point 306 in one of the missing portions.

  In the first embodiment as well, when the missing portion has only one end point, for example, a straight line parallel to the X-axis direction can be extrapolated to the end point of the horizontal line. Similarly, a straight line parallel to the Y-axis direction can be extrapolated to the end point of the vertical line.

  As described above, according to the present embodiment, in addition to the effects of the first embodiment, it is possible to improve the correction accuracy because it is difficult for a sudden change in the direction of the horizontal line that may occur at the missing portion in the case of interpolation using a straight line.

  In the present embodiment, an information processing apparatus 99 that interpolates a missing part using not only a horizontal line / vertical line with a missing part but also adjacent horizontal lines / vertical lines will be described.

  FIG. 25 is an example of a diagram for explaining the interpolation of the missing part in the present embodiment. It is assumed that a part of the nth horizontal line from the top is missing as shown. In this case, the pattern interpolation unit 15 interpolates the missing part using the (n−1) th horizontal line from the top adjacent to the upper side and the (n + 1) th horizontal line from the top adjacent to the lower side.

  Since the range of the X coordinate of the missing horizontal line is known, the Y coordinate of the reference point 307 of the (n−1) th and (n + 1) th horizontal lines having the same X coordinate as the X coordinate to be interpolated in this X coordinate range. With reference, the midpoint (average) of the Y coordinates of the two reference points 307 is calculated.

That is, in an arbitrary X coordinate in the missing part, the Y coordinate of the (n−1) th horizontal line is Y (n−1), and the Y coordinate of the (n + 1) th horizontal line is Y (n + 1). Therefore, Y (n) that is the Y coordinate of the nth horizontal line can be expressed by the following calculation.
Y (n) = {Y (n-1) + Y (n + 1)} / 2
In the interpolation method of the present embodiment, even when one horizontal line is missing for a long time, it is possible to accurately interpolate by using the coordinates of adjacent horizontal lines.

Further, as shown in FIG. 25B, interpolation may be performed using reference points of non-adjacent horizontal lines. The Y coordinate of the (n−2) th horizontal line is Y (n−2), and the Y coordinate of the (n + 1) th horizontal line is Y (n + 1). Using the fact that the intervals of the horizontal lines are substantially equal, Y (n) which is the Y coordinate of the nth horizontal line can be expressed as follows.
Y (n) = {Y (n-2) + 2Y (n + 1)} / 3
In FIG. 25B, since there is no missing portion in the (n−1) th horizontal line, such a calculation is not necessary, but when the n−1th horizontal line is missing (the missing portion in the adjacent horizontal line overlaps). ), It can be interpolated with the (n−2) th and (n + 1) th horizontal lines. Therefore, it is not always necessary to use adjacent horizontal lines for interpolation.

  On the other hand, when adjacent horizontal lines are overlapped as shown in FIG. 26, interpolation as shown in FIG. 27 is also effective. As shown in FIG. 26, one of the (n−1) th and (n + 1) th horizontal lines may be missing together with the nth horizontal line. FIG. 26 is an example of a diagram illustrating interpolation when the nth and (n + 1) th horizontal lines are missing. Since there are overlapping portions in the nth and n + 1th horizontal line deficits, the method of this embodiment cannot interpolate the nth horizontal line deficit (the n + 1th horizontal line deficit cannot be interpolated).

  In this case, the pattern interpolation unit 15 performs interpolation by the method of the first or second embodiment. That is, interpolation is performed preferentially by the method of the present embodiment, and a portion where adjacent horizontal line missing portions are overlapped is held without being interpolated. Thereafter, interpolation is performed by the method of the first or second embodiment (interpolation or extrapolation).

  FIG. 27 is an example of a diagram illustrating interpolation when the present embodiment and the first or second embodiment are combined. FIG. 27A is an example for explaining the interpolation of a part of the missing part of the nth horizontal line. Since there is a relationship of “the length of the missing portion of the n-th horizontal line> the length of the missing portion of the n + 1-th horizontal line”, in the ranges 321 and 322 where the n + 1-th horizontal line is detected, the method of FIG. Interpolate the missing part of the horizon.

  A range 323 in which both the nth and n + 1th horizontal lines are missing is reserved. As a result, as shown in FIG. 27B, the horizontal line is interpolated in a part of the missing part of the nth horizontal line (ranges 321 and 322 where the n + 1th horizontal line is present).

  In this way, even when the horizontal line is long missing, a part of the missing part of the nth horizontal line can be interpolated with high accuracy.

  In addition, since the pattern interpolation unit 15 interpolates the missing range 323 by the method of the first or second embodiment, it is possible to interpolate all of the nth horizontal line deficiency and the n + 1th horizontal line deficiency. .

  The missing portion of the (n + 1) th horizontal line is interpolated by the method of the first or second embodiment, similarly to the nth horizontal line.

  By the way, as shown in FIG. 28, the uppermost or lowermost horizontal line (outermost horizontal line) may not be detected completely. FIG. 28 is an example of a diagram illustrating interpolation when the uppermost horizontal line is not completely detected.

Since there is no horizontal line above the first horizontal line, the interpolation of this embodiment becomes difficult. Therefore, when the uppermost or lowermost horizontal line is not completely detected, the pattern interpolation unit extrapolates from the coordinates of the second and third horizontal lines to obtain the coordinates of the first horizontal line. The Y coordinate of the second horizontal line is Y (2), and the Y coordinate of the third horizontal line is Y (3). Therefore, the interval between the horizontal lines is “Y (3) −Y (2)”. Since the first horizontal line and the second horizontal line have the same distance in the Y direction, Y (1) can be obtained as follows.
Y (1) = Y (2)-{Y (3) -Y (2)}
In FIG. 28, all the first horizontal lines are missing, but the interpolation can be similarly performed when a part of the first horizontal line is missing.

  FIG. 29 is an example of a flowchart for explaining the procedure of the feature point extraction process in step S4 of FIG. In this embodiment, all the places that can be interpolated using adjacent horizontal lines or vertical lines are interpolated, and then the missing portion is searched again, and the horizontal lines or vertical lines before and after the missing portion are used as in the first or second embodiment. Interpolation. For this reason, the interpolation processing is performed in two steps of steps S4-7a and S4-23. The main steps are described below.

  If there is a missing portion in the line segment of the line pattern region 61 in step S4-6, the pattern interpolation unit 15 interpolates the missing portion from the horizontal lines adjacent to the missing horizontal line (S4-7a). This process is performed for all missing portions and all horizontal lines (S4-8, S4-9). The same process is performed for the vertical line (S4-10).

  Next, steps S4-5 and S4-6 are performed in order to detect and interpolate the remaining missing part. That is, the corresponding point extraction unit 18 extracts black pixels in the line pattern region 61 and extracts continuous black pixels (S4-5).

  The corresponding point extraction unit 18 detects the line pattern region 61 in which the horizontal line is missing from the line pattern region 61 at the boundary of the band regions 1 to 8 (S4-6).

  And the pattern interpolation part 15 interpolates a missing part by the method of Example 1 or 2 (S4-23).

  Thereafter, the process of S4-23 is performed for all the missing portions and all the horizontal lines (S4-24, S4-25). The same process is performed for the vertical line (S4-26).

  The corresponding point extraction unit 18 combines the pattern images A and B (S4-11). The corresponding point extraction unit 18 outputs the intersection as a feature point to the correction coefficient calculation unit 20 (S4-12).

  Therefore, even when one horizontal line or vertical line is long missing, the information processing apparatus 99 according to the present embodiment can accurately interpolate by using the adjacent horizontal line or vertical line.

  In the distortion correction using the projection pattern projected by the projector 100 and the pattern imaging unit 11 by the pattern image, it is assumed that the user is at the position of the information processing apparatus 99 (pattern imaging unit). For this reason, inconveniences such as fixing the position of the user may occur.

  In this embodiment, therefore, distortion correction that can be viewed even when the user moves the position by performing distortion correction using three-dimensional information obtained by performing three-dimensional measurement of the projection target will be described.

  In order to perform three-dimensional measurement, in this embodiment, correction of a projected image by the projector 100 equipped with the function of the information processing apparatus 99 will be described.

  FIG. 30 is an example of a diagram for explaining the hardware configuration and functional blocks of the projector 100. The projector 100 has the same hardware configuration as the information processing apparatus 99 in FIG.

  As illustrated, the projection image is stored in the storage unit 22 of the projector 100. The storage unit 22 is a removable portable memory such as a USB memory. Further, a PC connected to the projector 100 may transmit the projection image to the projector 100. The projector 100 and the PC may be connected by a video cable such as HDMI (registered trademark) or may be connected by wireless communication such as a wireless LAN.

  In addition, the projector 100 includes a three-dimensional shape measurement unit 23 and a perspective projection matrix calculation unit 24. The three-dimensional shape measurement unit 23 measures the three-dimensional shape of the projection target (acquires distance information). Further, the perspective projection matrix calculation unit 24 calculates a perspective projection matrix.

  FIG. 31 is an example of a diagram illustrating the processing procedure of the information processing apparatus 99 according to the present embodiment. In this embodiment, after extracting corresponding points in step S4 of FIG. 14, the three-dimensional shape measurement unit 23 performs three-dimensional measurement, calculates a perspective projection matrix, and obtains a captured image of the projector at the virtual position obtained thereby ( Using the pattern image, the correction coefficient calculation unit 20 calculates the projective transformation matrix H. Hereinafter, it demonstrates in order.

・ S100
First, measurement of a three-dimensional shape will be described with reference to FIG. FIG. 32A is an example of a diagram for explaining measurement of a three-dimensional shape. If the point A on the projection element is imaged at the point B on the image sensor, the difference in the horizontal distance between the point A and the point B corresponds to the parallax d.

  As shown in FIG. 32B, the projection element (DMD) projects a projection pattern (projection image), and the pattern imaging unit 11 captures a pattern image (captured image). Further, as described above, the correspondence between the mesh intersections is detected. Accordingly, the parallax d is obtained as the horizontal distance between the corresponding intersections. The parallax of the pixels between the intersections can be obtained by linearly interpolating the parallax between the adjacent intersections.

When the base line length (distance between the projection element and the imaging element) is D, the distance Z to the projection point on the screen can be obtained by the principle of triangulation. Note that f is a focal length of the imaging unit 110.
Z = Df / d
Therefore, the three-dimensional coordinates can be calculated for each pixel of the projected image on the screen viewed from the projector (or for each appropriate pixel block).

・ S200
Thus, correspondence between the captured image on the imaging unit 110 and the projected image on the projection object is obtained. However, the captured image on the imaging unit is configured in units of pixels in which a pixel pitch is arranged between pixels. Further, the center position of the projected image, the aspect ratio of the pixels, and the like are specific to the imaging unit 110. Because of these camera internal parameters, the correspondence between the obtained captured image on the imaging means and the projected image on the projection object cannot be expressed simply by the focal length f.

  Therefore, the perspective projection matrix calculation unit 24 calculates the correspondence between the captured image on the imaging unit and the projected image on the projection target in consideration of the internal parameters of the camera.

  FIG. 33A schematically shows the coordinate system of the imaging means. The coordinate system of the imaging means is called a camera coordinate system. First, the correspondence between the captured image on the imaging means using the focal length f and the projected image on the projection target will be described. For example, a coordinate system (Xc, Yc, Zc) with the center of the imaging means as the origin is taken as the camera coordinate system. The point M in the three-dimensional space of the camera coordinate system is associated with the point M ′ (u ′, v ′) on the pixel of the imaging means by the focal length f. Such a relationship is called projection.

M ′ (u ′, v ′) ← M (Xc, Yc, Zc)
If this conversion coefficient is A, it can be expressed as in equation (4-1). Since it is handled as a matrix, equation (4-1) is expressed in a homogeneous coordinate system.

Next, conversion by the camera internal parameter A will be described. As shown in FIG. 33 (b), the point M ′ (u ′, v ′) can be converted to the point m of the imaging means by the camera internal parameter A. Therefore,
m = AM '
It can be expressed. Although there are various expressions for the camera internal parameter A, for example, it can be expressed by translation and scaling (magnification) of the point M ′ according to the equation (4-2).

  By the way, although the point M in the camera coordinate system can be converted into the image coordinate point m in consideration of the camera internal parameters, the point M must be handled in the camera coordinate system. When the projector is moved to an arbitrary location, the point M is represented in a common coordinate system (hereinafter referred to as the world coordinate system) regardless of the location of the projector, so that the distortion is appropriately corrected even when the projector is moved. Projected images can be projected.

  Therefore, the world coordinate system (Xw, Yw, Zw) is converted in advance to a point M (Xc, Yc, Zc) in the camera coordinate system. As shown in FIG. 33C, the transformation of the three-dimensional coordinates can be expressed by rotation and translation. The point Mw in the world coordinate system is converted into a point M in the camera coordinate system that has been rotated and translated by Expression (4-3). A transformation matrix for converting the point Mw in the world coordinate system to the point M in the camera coordinate system is represented by R in the rotation part and t in the translation part (this matrix is represented as R | T). R in the figure is a rotation matrix of rotation around the Z axis, and rotation matrices around the X axis and Y axis can be similarly expressed. In this description, the world coordinate system is converted to the camera coordinate system before conversion using the camera internal parameters. However, the world coordinate system may be converted to the camera coordinate system after conversion using the camera internal parameters.

  Therefore, a point in the world coordinate system can be converted to a point m on the imaging means by “m = A × (R | T) × Mw”. In more detail, it is as follows.

また、
Ｐ＝Ａ×（Ｒ｜Ｔ） …（４−５）
とおけば、ｍ＝Ｐ・Ｍｗと表記できる。Ｐは透視投影行列と呼ばれる３×４の行列である。

Also,
P = A × (R | T) (4-5)
Then, it can be expressed as m = P · Mw. P is a 3 × 4 matrix called a perspective projection matrix.

変数（要素数）が１２個あるが、三次元計測により、少なくとも交点の数（例えば８１）の、Ｍｗとｍの対応が得られているので、１２個全ての要素を求めることができる。

Although there are twelve variables (number of elements), the correspondence between Mw and m of at least the number of intersections (for example, 81) is obtained by three-dimensional measurement, so that all twelve elements can be obtained.

・ S300
Although the point in the three-dimensional space is converted into a point on the captured image by Expression (4-6), even when the user views the projection object from a different viewpoint, if the perspective projection matrix P at the viewpoint is obtained, the position It is possible to calculate the points on the imaging means obtained by

  FIG. 34 is an example of a diagram for explaining a perspective projection matrix when the projector is virtually moved. In the figure, a virtual projector 100A is shown in addition to the projector. The perspective projection matrix P2 when the projector is virtually moved to the position of the virtual projector 100A can be obtained by updating the matrix (R | T) according to the movement position.

  When the camera internal parameter A is known by calibration or the like, P = A × (R | T) in Expression (4-5) by (R | T) when the position of the projector 100 is moved to the position of the virtual projector 100A. ), The perspective projection matrix can be obtained when the virtual projector 100A is at the position. (R | T) is an example of displacement information.

  If the camera internal parameter A is not known, the camera internal parameter A can be obtained by obtaining (R | T) of the camera coordinate system with respect to the world coordinate system and substituting it into the equation (4-5). If the camera internal parameter A is known, the perspective projection matrix P2 of the virtual projector 100A can be obtained by substituting (R | T) when moved to the position of the virtual projector 100A into Expression (4-5).

・ S400
When the perspective projection matrix P2 of the virtual projector is obtained, the perspective projection matrix calculation unit 24 projects the projection image of the projection target onto the imaging unit of the virtual projector, and obtains the captured image captured by the imaging unit of the virtual projector 100A. Can do.

  This completes the process up to storing the captured image in S3 of FIG. Therefore, after this, the process proceeds to S4 in FIG. 14, and corresponding points are extracted again. In the corresponding point extraction process, the detection of the missing portion, the interpolation of the missing portion, and the extraction of the corresponding points described in the first to third embodiments are performed. Then, the correction coefficient calculation unit 20 calculates a projective transformation matrix H based on the corresponding points (S5), and the image processing unit 16 corrects the projection image (S6).

  Therefore, the projector of the present embodiment can correct distortion without fixing the position of the user by measuring the projection object three-dimensionally.

  In this embodiment, the three-dimensional measurement is performed using the parallax between the projection element and the imaging element. However, the three-dimensional measurement may be performed using a stereo camera or a camera capable of measuring TOF (Time Of Flight).

DESCRIPTION OF SYMBOLS 11 Pattern imaging part 12 Communication part 13 Binarization part 14 Pattern selection part 15 Pattern interpolation part 16 Image processing part 17 Imaging control part 18 Corresponding point extraction part 19 Defect part extraction part 20 Correction coefficient calculation part 21 Projection object imaging part 22 Storage unit 99 Information processing apparatus 100 Projector 200 Image correction system

JP 2013-065277 A

Toru Takahashi and three others, “Experimental study on geometric correction of projected images”, Society of Instrument and Control Engineers Tohoku Branch, 235th Research Meeting, 2007.5.18, Document No. 235-5

Claims (10)

An information processing apparatus for correcting a projection image projected on a projection object,
Projected image acquisition means for acquiring a first captured image in which the projected first line group is captured, and a second captured image in which the second line group intersecting the first line group is captured. When,
A partial or total defect is detected for each line of the first line group in the first captured image, and a partial or total defect is detected for each line of the second line group in the second captured image. Deficient part detecting means for detecting
Interpolating means for interpolating a defect in the line of the first line group or the line of the second line group;
A first intersection of the line of the first line group and the line of the second line group in which a defect is interpolated, and the line of the first line group and the second line group of the projection image Intersection detection means for detecting a second intersection with the line of
Correction coefficient calculating means for calculating a correction coefficient by comparing a range formed by a plurality of the first intersections with a range formed by the plurality of the second intersections;
Correction means for correcting the projection image using the correction coefficient;
An information processing apparatus comprising:   The interpolating means interpolates the missing portion of the line of the first line group using coordinates other than the missing portion of the missing line, and lacks the missing portion of the line of the second line group. The information processing apparatus according to claim 1, wherein interpolation is performed using coordinates other than a missing part of a line. The first line group has a plurality of substantially parallel lines on which at least a part of the longitudinal direction is superimposed,
The second line group has a plurality of substantially parallel lines on which at least a part of the longitudinal direction is superimposed,
The interpolating means interpolates the missing portion of the line of the first line group using the coordinates of the other lines of the first line group, and the missing portion of the line of the second line group is The information processing apparatus according to claim 1, wherein interpolation is performed using coordinates of other lines of the second line group.   The interpolation means interpolates the missing portion of the line of the first line group by connecting both ends of the missing portion with a straight line or a curve, and replaces the missing portion of the line of the second line group with the missing portion. The information processing apparatus according to claim 2, wherein interpolation is performed by connecting both ends of the line with a straight line or a curve. When the line of the first line group is missing including the start or end, the interpolation means extrapolates the missing part of the missing line to the start or end side that is missing. To interpolate
If the line of the second line group is missing including the start or end, interpolate by extrapolating the missing part of the missing line to the missing start or end,
The information processing apparatus according to claim 4. The interpolating means calculates the missing portion of the line of the first line group that is missing by calculating the coordinates of the other lines of the first line group in the direction perpendicular to the longitudinal direction of the line. Interpolate at points,
Interpolating the missing part of the line of the second line group that is missing by calculating the coordinates of the other lines of the second line group in the direction perpendicular to the longitudinal direction of the line,
The information processing apparatus according to claim 3. Since the missing part of the first line of the first line group missing and the missing part of the other line of the first line group overlap in the longitudinal direction of the line, the first line group When a part of the missing part of the line cannot be interpolated from the coordinates of the other line of the first line group, the interpolation means replaces the part of the missing part of the first line with the part of the part. Interpolate both ends with straight lines or curves,
Since the missing part of the second line of the second line group that is missing and the missing part of the other line of the second line group overlap in the longitudinal direction of the line, the second line group When a part of the missing part of the line cannot be interpolated from the coordinates of the other line of the second line group, the interpolation means replaces the part of the missing part of the second line with the part of the part. Interpolate both ends with straight lines or curves,
The information processing apparatus according to claim 6. If the outermost line of the first line group is missing,
The interpolating means calculates the coordinates of the second and third lines from the outermost side of the first group of lines in the direction perpendicular to the longitudinal direction of the line, and the missing part of the outermost line. Interpolate
If the outermost line of the second line group is missing,
Interpolate the missing part of the outermost line at the point obtained by calculating the coordinates of the second and third lines from the outermost side of the second line group in the longitudinal direction and the straight direction of the line,
The information processing apparatus according to claim 3. Distance information acquisition means for acquiring distance information between the projection object and the information processing apparatus;
Calculating means for calculating a first perspective projection matrix for projecting a point on the projection object onto a point on the imaging means, using the distance information when the information processing apparatus is arranged at the first location; Have
The calculation means calculates a second perspective projection matrix at the second location using displacement information when the information processing apparatus is arranged from the first location to the second location,
The first captured image and the second captured image are obtained by projecting a point on the projection object onto a point on the imaging means in the second location using the second perspective projection matrix. And
The defect detection means detects a partial or total defect for each line of the first line group, detects a partial or entire defect for each line of the second line group,
The interpolating means interpolates a loss of the line of the first line group or the line of the second line group,
The intersection detection means includes a first intersection of the line of the first line group and the line of the second line group in which a defect is interpolated, and the line of the first line group of the projection image. Detecting a second intersection with a line of the second line group;
The correction coefficient calculation means calculates a correction coefficient by comparing a range formed by a plurality of the first intersections with a range formed by a plurality of the second intersections,
The correction means corrects the projection image using the correction coefficient;
The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. In an information processing apparatus that corrects a projection image projected on a projection object,
A projected image acquisition step of acquiring a first captured image in which the projected first line group is captured and a second captured image in which the second line group intersecting with the first line group is captured. When,
A partial or total defect is detected for each line of the first line group in the first captured image, and a partial or total defect is detected for each line of the second line group in the second captured image. A defect detection step for detecting
An interpolation step for interpolating a defect in the line of the first line group or the line of the second line group;
A first intersection of the line of the first line group and the line of the second line group in which a defect is interpolated, and the line of the first line group and the second line group of the projection image An intersection detection step for detecting a second intersection with the line of
A correction coefficient calculating step of calculating a correction coefficient by comparing a range formed by a plurality of the first intersections with a range formed by the plurality of the second intersections;
A correction step of correcting the projection image using the correction coefficient;
A program characterized by having executed.

Images