JP2011050013A - Video projection apparatus and video projection method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To remedy difficulty in watching a projection video caused by distortion of the projection video or low reflectance of a projection surface in a mobile projector or to project an image easy to watch by efficiently using a region suitable for remaining projection in such a case that a band-like area of low reflectance such as a column or different material surface is present on the projection surface. <P>SOLUTION: A position distortion correction unit 106 generates a position distortion correction parameter corresponding to mapping W from a reference pattern to a light receiving pattern and based on a correspondence relationship, a screen division correction unit 107 detects a low-reflection area where an area corresponding to the reference pattern is missed in the light receiving pattern and divides a video projection screen so as to project a video on any other area than the low-reflection area. Mapping V from an image before division to an image after division is calculated and in accordance with the mapping V, an image division correction parameter is generated. A correction unit 109 uses these parameters to correct video contents. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a video projection apparatus and a video projection method.

  Video projectors called mobile projectors, which are compact and lightweight and easy to carry, are commercialized due to positive factors such as recent advances in wireless communication technology and the appearance of inexpensive ultra-high brightness white LEDs (Light Emitting Diodes). Has been. Some mobile projector products are on the palm and weigh less than 200 grams. There are also products that operate with an internal power supply and operate without an external power supply.

  Such a mobile projector can be used when a business presentation is performed by connecting to a notebook computer. Furthermore, it will be widely used as a secondary external output device for terminals having only a small display such as a mobile phone or PDA (Personal Digital Assistant).

  However, although the current mobile projector has been reduced in size, it is still assumed that projection onto a dedicated flat projection surface is assumed, as in the case of a conventional video projection apparatus. Therefore, in an actual mobile environment, carrying a dedicated projection screen needs to be dramatically improved in size and weight.

  For this reason, it is conceivable that the mobile projector has to use a handy plane such as a wall or a door of a house. However, in this case, the image projection surface often has non-uniform or discontinuous color (hue, saturation and brightness), is non-planar, or is not directly facing the projector even if it is flat. It will be. Furthermore, there may be a case where the positional relationship between the projection surface and the projector changes with time, such as in the case of a hand-held projector.

  Then, when an image is projected on the projection surface having the distortion of the projection surface, the non-uniform reflectance, the color, etc., as described above, a method for solving the difficulty in viewing the image, which has been conventionally used, is described below.

  In Patent Document 1 and Patent Document 2, distortion correction of an image projected on an arbitrary non-planar screen is studied.

  Specifically, in the method described in Patent Document 1, an image (point image) before correction is projected, and at the same time, a laser pointer is projected to a place where the point image is to be displayed. Then, by recognizing these images with a camera, information on the deviation is obtained. This deviation is measured at many points to obtain distortion correction parameters for the entire image correction.

  In the method before Patent Document 1, it is impossible to determine a projection area for a projection plane composed of a plurality of planes unless an expensive device (laser range finder or the like) is used. However, in Patent Document 1, this point has been improved. Yes.

  Moreover, in the method described in Patent Document 2, a special projection pattern is projected onto a projection plane (for example, a folding screen-like screen) composed of a plurality of planes, and the relationship between the pixels and the projection points is captured by the camera. Get. Then, the projection points are clustered to recognize the projection plane as a plurality of planes, and for example, an image is displayed on the widest plane. At this time, image distortion correction is also performed.

  As described above, both Patent Document 1 and Patent Document 2 automatically correct image distortion and position when projecting an image onto a projection surface that does not face the image projection apparatus. , It is possible to show the user a desired video.

  Further, in the method described in Patent Document 2, when there is a region that is not suitable for projection on the projection surface, the projection can be performed while avoiding the region.

Japanese Patent Application Laid-Open No. 2001-169211 JP 2007-142495 A

  However, in an actual mobile environment, it is necessary to use, for example, an inner wall of a house as a projection plane. In this case, it is often the case that the video projector must be installed so that the wall is painted in several colors or the dark-colored pillars cross the projection surface. is there.

  However, even with the above two methods, it is difficult to correct a surface with non-uniform colors (hue, saturation, brightness). In the method described in Patent Document 2, the maximum surface that can be projected can be found while avoiding a region of the projection surface that does not reflect even when projected, but the usable surface may be reduced. In such a case, it is difficult to project an easily viewable image on a large surface even by the above two methods.

  Further, when the relative position between the projector and the projection surface changes with time, it is difficult to cope with either method.

  An object of the present invention is to improve the difficulty in viewing a projected image due to distortion of the projected image or low reflectance of the projection surface in a mobile projector, or a surface with a different column or material on the projection surface. To provide a video projection device and a video projection method capable of efficiently projecting an easy-to-view image by efficiently using a remaining area suitable for projection when a strip having a low reflectance exists.

  The image projection apparatus of the present invention includes a reference pattern generating means for outputting a reference pattern, a pattern projecting means for projecting the reference pattern as an image, a light receiving pattern obtained by receiving the projected reference pattern, and the reference Compares the pattern, identifies the correspondence between the components of the pattern, and generates a correction parameter for positional distortion and a comparison means for calculating the mapping W from the reference pattern to the light receiving pattern based on the correspondence And a screen division correction unit that generates a screen division correction parameter. Based on the correspondence, whether or not the position distortion correction or the screen division correction is necessary is determined, and according to the determination result. A correction management unit that operates the positional distortion correction unit and the division correction unit, and the positional distortion correction parameter or the screen division correction. A correction unit that corrects video content using a parameter; and a video projection unit that projects the corrected video content, wherein the positional distortion correction unit corrects the positional distortion according to the mapping W. Generating a parameter, and the image division correction unit detects a low reflection region including a region in which the region corresponding to the reference pattern is missing from the light receiving pattern, and projects the image on a region other than the low reflection region. As described above, the screen on which the video is projected is divided, a mapping V from the image before the division to the image after the division is calculated, and the correction parameter for image division corresponding to the mapping V is generated.

  The image projection method of the present invention projects a reference pattern as an image, compares the received light pattern obtained by receiving the projected reference pattern with the reference pattern, and shows the correspondence between the components of these patterns. Identify and calculate the mapping W from the reference pattern to the light receiving pattern based on the correspondence, determine the necessity of positional distortion correction or screen division correction based on the correspondence, and need positional distortion correction If the correction parameter for position distortion corresponding to the mapping W is generated and it is determined that screen division correction is necessary, the region corresponding to the reference pattern is missing from the light receiving pattern. A low-reflective area including the image, and dividing the screen on which the video is projected so that the video is projected to an area other than the low-reflective area. A mapping V to an image is calculated, the image division correction parameter corresponding to the mapping V is generated, and the video content is corrected and corrected using the position distortion correction parameter or the screen division correction parameter. The video content is projected.

  According to the present invention, it is possible to improve the difficulty in viewing a projected image due to the distortion of the projected image or the low reflectance of the projection surface, or the reflectance of a surface with a different column or material on the projection surface is low. Even in the case where there is a band-like region, it is possible to efficiently use the remaining region suitable for projection and project an easy-to-view image.

The figure which shows the structure of the image | video projection system which concerns on Embodiment 1 of this invention. Schematic representation of the positional relationship between the projection surface and each part of the video projection device Reference pattern in the first embodiment Other reference patterns in the first embodiment Explanatory drawing of the projection order of the reference pattern in Embodiment 1 Light receiving pattern in the first embodiment Other light receiving patterns in the first embodiment Light spot correspondence table between patterns in the first embodiment Operational explanatory diagram of the pattern light receiving analysis unit of the first embodiment A graphical representation of the mapping W of the first embodiment Operational explanatory diagram of the pattern light receiving analysis unit of the first embodiment Explanatory drawing of the projection order of the reference pattern in Embodiment 1 The figure showing the effect of the mapping W of Embodiment 1 Flowchart diagram of mapping W calculation in the first embodiment Flowchart diagram of mapping W calculation in the first embodiment Light receiving pattern in the first embodiment Explanatory drawing of the screen division correction operation in the first embodiment The figure showing the effect of the screen division correction in the first embodiment The figure showing the effect of maps W and V of the first embodiment Light receiving pattern in the first embodiment Explanatory drawing of the screen division correction operation in the first embodiment The figure showing the effect of the screen division correction in the first embodiment Explanatory drawing of operation | movement of the light reception pattern and screen division | segmentation correction | amendment in Embodiment 1. Explanatory drawing of the screen division correction operation in the first embodiment The figure showing the effect of the screen division correction in the first embodiment Light receiving pattern in the first embodiment Light receiving pattern in the first embodiment Explanatory drawing of the screen division correction operation in the first embodiment Color correction reference pattern in the first embodiment Color-corrected light receiving pattern in the first embodiment The figure showing the effect of maps W, V, and U of the first embodiment Reference pattern in Embodiment 2 of the present invention Schematic representation of the positional relationship between components of the video projector in Embodiment 3 of the present invention

  Embodiments of the present invention will be described below with reference to the drawings.

(Embodiment 1)
FIG. 1 is a configuration diagram of a video projection system according to Embodiment 1 of the present invention.

  In FIG. 1, a projection surface 120 is a surface on which the image projection apparatus 100 projects an image. The projection surface 120 may be a dedicated projection screen, or any object surface having a surface on which an image is projected by projecting an image, such as a wall or a door.

  Specifically, the video content 130 is a movie, a photograph, a presentation material for business, or the like, and is usually stored in a storage device such as a personal computer or a PDA (Personal Digital Assistant) outside the video projection device 100. . The storage device such as a personal computer and the video projection device 100 are connected by a video transfer cable or the like of an appropriate standard or protocol. Of course, the storage device such as a personal computer and the video projection device 100 may be connected not by a wired cable but by a wireless transmission path.

  The reference pattern generation unit 101 outputs one or more types of reference patterns to the pattern projection unit 102 and the pattern comparison unit 104. Details of the reference pattern will be described later.

  The pattern projection unit 102 projects the reference pattern input from the reference pattern generation unit 101 as an image. Note that this image is not limited to an image of visible light, and the image is a laser beam (light spot) from a structure that illuminates with a powerful light source from the back on the surface acting as a movie film such as a liquid crystal projector. ), And the like that display the image by raster scanning the projection surface.

  The pattern light reception analysis unit 103 receives the image projected by the pattern projection unit 102 onto the projection surface 120, analyzes the received image, and converts it into a pattern. A pattern obtained as a result of analyzing the received video is called a “light receiving pattern”. The optical image of the received image and the optical image of the image projected on the projection surface 120 are referred to as a “pattern image”.

  This pattern image is a concept different from the pattern of the reference pattern. The pattern image is an optical image, and the simple pattern is obtained as a result of analyzing the pattern image that is an optical image. That is, the pattern is a source for generating an optical image, and is a geometric concept such as “point”, “line”, or “square”.

  The pattern comparison unit 104 compares the light reception pattern generated by the pattern light reception analysis unit 103 with the original reference pattern, and generates a mapping from the light reception pattern to the reference pattern by associating the components of these two patterns. To do.

  The correction management unit 105 determines whether or not position distortion correction, screen division correction, and color correction are necessary based on the correspondence relationship between the light receiving pattern and the reference pattern generated by the pattern comparison unit 104. Judging. Then, the correction management unit 105 calls up the positional distortion correction unit 106, the screen division correction unit 107, or the color correction unit 108 in an appropriate order according to the determination result, and calculates the video projection correction parameters. . The correction management unit 105 outputs the calculated video projection correction parameters to the correction unit 109.

  The positional distortion correction unit 106 calculates a positional distortion correction parameter for correcting positional distortion. The screen division correction unit 107 calculates a screen division correction parameter for dividing the screen. The color correction unit 108 calculates color correction parameters for correcting colors (hue, saturation, brightness). The correction management unit 105 calculates a video projection correction parameter by combining the position distortion correction parameter, the screen division correction parameter, and the color correction parameter. A method for calculating these correction parameters will be described later.

  The correction unit 109 corrects the projection of the video content 130 according to the video projection correction parameter generated by the correction management unit 105.

  The video projection unit 110 projects the video content 130 on the projection surface 120 in a state corrected by the correction unit 109.

  Now, it is shown that the image projection apparatus 100 configured as shown in FIG. 1 can perform positional distortion correction, screen division correction, and color correction.

  First, these correction types are defined. Note that specific examples appear in the following description.

[1] Positional distortion correction Positional distortion is when the video projection unit 110 is not directly facing the projection plane 120, when the projection plane 120 is non-planar, or when the position of the video projection unit 110 with respect to the projection plane 120 is time. This means image distortion and positional deviation as seen from the user of the image projection apparatus 100 that occurs when the image changes. The positional distortion correction refers to returning video distortion or positional deviation to a normal state when viewed from the user.

[2] Screen division correction In the screen division correction, when there is an area that is not suitable as a projection plane on the plane on which the image is to be projected, one piece of projection content is divided into several pieces so as to avoid the inappropriate area. This refers to correction for dividing, reducing, moving and the like.

[3] Color correction Color correction refers to correction applied to cancel the difference in hue, saturation, or brightness when the hue, saturation, or brightness differs for each partial area of the projection plane. Alternatively, the color correction is to correct unevenness in brightness that occurs when the projection surface 120 is not directly facing the video projection device 110 or when the projection surface 120 is a curved surface.

  Now, the description returns to the embodiment.

  Now, it is assumed that the pattern projection unit 102, the pattern light reception analysis unit 103, the video projection unit 110, and the projection plane 120 are installed in a positional relationship as shown in FIG. That is, the pattern projection unit 102 is installed near the user's eyes (head) viewing the projection plane 120, and is further installed so that the user's line of sight and the image projection axis of the pattern projection unit 102 are substantially parallel. ing. When there are a plurality of users, one of the users is appropriately selected, and the pattern projection unit 102 is installed in the vicinity of the user's eyes. It is assumed that this user is in front of the projection plane 120.

  The pattern light reception analysis unit 103 and the video projection unit 110 are installed adjacent to each other, and the central axis of the pattern light reception analysis unit 103 and the optical axis of the video projection unit 110 are parallel or both axes are the center of the projection plane. It is assumed that they are installed so as to cross each other. Then, it is assumed that the video projection unit 110 and the pattern light reception analysis unit 103 are installed on the lower left side toward the projection plane 120.

  When the installation is completed, the video projection device 100 starts calculating the video projection correction parameters.

  First, the correction management unit 105 instructs the reference pattern generation unit 101 to generate a reference pattern.

  Then, the reference pattern generator 101 generates a reference pattern for positional distortion correction and image division correction. As described above, the pattern is a geometric concept such as a point, a line, or a polygon, and has graphic attributes such as color (hue, saturation or brightness), position, or size of a figure. Have.

  FIG. 3 is a graphical representation of the reference pattern used in the present embodiment, that is, a pattern image (hereinafter referred to as “reference pattern image”) 200 with respect to the reference pattern. Here, the reference pattern used in the present embodiment is hereinafter referred to as “reference pattern d”.

  This reference pattern d is a set of points arranged in a lattice pattern, and is expressed as follows in a descriptive manner.

  Reference pattern d = {Point (0, 0), Point (0, 160), Point (0, 320), (middle omitted), Point (960, 1120), Point (960, 1280)}.

  Here, Point (x, y) means a point located at coordinates (x, y) on the two-dimensional plane. Since these points are expressed as bright points on the projection plane (because of the area, they are expressed as small circles instead of dots), these points are called light spots.

  In addition, expressing these light spots with Point (x, y) and coordinates is referred to as light spot position expression. Further, the light spot Point (x, y) of the reference pattern d is represented as d (x, y).

  Furthermore, in the present embodiment, the reference pattern d is rectangular and arranged on the grid. For this reason, it is convenient if it can be expressed using rows and columns in addition to the coordinates. Therefore, for the sake of convenience, in the present embodiment, each point on the reference pattern d or a pattern image generated from the reference pattern d is expressed as d [i, j] or the like. i indicates a column number in the positive direction of the x-axis, and j indicates a row number in the positive direction of the y-axis. For example, Point (0, 0) can be expressed as d [0, 0], and the point 201 in FIG. 3 can be expressed as d [4, 5]. This expression such as d [i, j] is referred to as a light spot index expression.

  For convenience of explanation, the number of light spots of the reference pattern d is 7 by 9 in length, but in actuality, the number of light spots must be determined by a trade-off between precision and calculation time. As the number of light spots is increased, the correction can be performed more precisely, while the processing time of the entire correction process becomes longer.

  In addition, although the vertical and horizontal sizes of the reference pattern d are described as 1280 pixels in width and 960 pixels in length, this is also a size used for convenience of description, and it is not necessary to have this size in particular. .

Now, in the reference pattern d, the relationship between the indexed light spot d [i, j] and the position-displayed light spot Point (x, y) is determined as follows.
d [i, j] ≡Point (i × 160, j × 160)
However, 0 ≦ i ≦ 7, 0 ≦ j ≦ 5, i and j are integers

  These light spots are within the rectangle whose upper left corner is 1280 pixels wide and 960 pixels long in the origin coordinates. This rectangle corresponds to the projection range of the video projection unit 110 as it is. That is, in this case, the video projection unit 110 can project a screen having a maximum width of 1280 pixels and a height of 960 pixels.

  If the reference pattern d is a set of vertically and horizontally aligned points, it is advantageous in the analysis operation of the pattern light receiving analysis unit 103 in the subsequent process. As the shape of the reference pattern, for example, a lattice pattern as shown in FIG. 4 or various geometric patterns (not shown) can be considered. A necessary condition that a certain pattern can suffice as a reference pattern is that it is basically possible to calculate which part of the reference pattern corresponds to which part of the light receiving pattern.

  Now, the reference pattern d generated by the reference pattern generation unit 101 is transferred to the pattern projection unit 102, converted into a reference pattern image 200 as shown in FIG. 3, and the image is projected on the projection plane 120. In FIG. 3, each point of the reference pattern image 200 is indicated by a black circle. However, this is for the convenience of drawing, and in fact, the black portion in FIG. 3 is a portion with high luminance. The white portion of the ground is a portion where the luminance is 0 (pure dark). In all of the following description, it is assumed that black and white inversion is expressed unless otherwise specified regarding the pattern image.

  Note that it is not necessary to project all of the constituent elements (points in the example shown in FIG. 3) at the same time. For example, as shown in FIG. 5, the pattern projection unit 102 may project four points at intervals. In FIG. 5, the pattern projection unit 102 simultaneously applies four points (d [i, j], d [i, j + 1], d [i + 1, j], d [i + 1, j + 1]) that are adjacent vertically and horizontally. Although an example of projection is shown, the points to be projected at the same time do not need to be adjacent vertically and horizontally. For example, a method of projecting four points at random from the light spot of the reference pattern, or d [i, j], d [i, j + 1], d [i, j + 2], d [i, j + 3] In addition, four points may be projected vertically, or four points may be projected similarly. Further, the pattern projection unit 102 has four points (d [i, j], d [i + n, j], d [i, j + n], d [i + n, j + n) in which several rows (n) are skipped in the vertical and horizontal directions. ]) May be projected simultaneously.

  As to how the pattern projection unit 102 is effective when projecting a light spot, it must be considered from the purpose of projecting a reference pattern and analyzing it in the first place.

  That is, the purpose of obtaining the light receiving pattern by projecting the reference pattern is (1) obtaining a mapping from the reference pattern to the light receiving pattern (correcting distortion and misalignment of the projected image by this mapping), (2) These are two points: finding a region with a low reflectance on the projection surface that exceeds the correctable range depending on the intensity of the projected light (finding and using it for subsequent screen division correction).

  However, as will be described in detail later, for the purpose of (1), it is only necessary to know at least four correspondences from the reference pattern to the light receiving pattern. For the purpose of (2), it is only necessary to know an area where the reflectance of the projection surface is extremely low (an area where the corresponding light receiving pattern cannot be obtained even if the reference pattern is projected). do not have to.

  Thus, for example, a method is conceivable in which only four points are initially projected from the reference pattern, and after the mapping from the reference pattern to the light receiving pattern is calculated, all points are projected. As will be described in detail later, if the projection plane is a plane, the mapping can be calculated as long as the correspondence between the four points is known. Alternatively, only the points in even rows and even columns are projected without projecting all points, and if a region with low reflectance is found, only that region is projected in detail (that is, odd rows or odd columns). The method of projecting the point) can be considered. It is also an effective method to project several points at random and project a light spot in a concentrated manner when a region with low reflectance is found.

  Note that what points should be projected at the same time will be described again in the explanation of the operation of the pattern light receiving analysis unit 103. In order to simplify the explanation at this stage, it is assumed that the projection surface 120 has a uniform reflectance, a white color, and a uniform flat wall. The case where the reflectance or color is not uniform or the projection surface 120 is not a flat surface will be described later.

  Now, the correction management unit 105 instructs the pattern image reception analysis unit 103 to receive and analyze the pattern image projected on the projection surface 120 to calculate a light reception pattern.

  First, the pattern image receiving analysis unit 103 receives an optical image reflected on the projection surface 120. The received pattern image (light receiving pattern image) 300 is shown in FIG. The analysis result of the light receiving pattern image 300 is referred to as a light receiving pattern r.

  In the case of the present embodiment, since the reference pattern projected from the front is looked up from the lower left toward the projection plane, the shape of the received light pattern image is not a rectangle but looks distorted as a square in FIG. If the projection surface is not a flat surface and, for example, the central portion is slightly bulging toward the user side, it will appear distorted as shown in FIG.

  Next, when obtaining the light reception pattern image 300, the pattern light reception analysis unit 103 analyzes the light reception pattern image 300 and generates a light reception pattern r. The contents of the light receiving pattern r is a set of recognized points. For example, the point 301 becomes Point (965, 512).

  Regarding the recognition of a point from an image, it is possible to obtain the position of the point by software using a general method of image analysis, and a two-dimensional light spot position detecting element (hereinafter simply referred to as a position). The position of the point may be obtained directly using a detection element).

  However, software image analysis requires a relatively high-speed computing device and a large amount of memory, and further, an answer cannot always be obtained at high speed. On the other hand, the method using the position detection element can detect the position of each light spot at high speed with inexpensive components. For this reason, it is advantageous to use a position detection element in order to produce the device at a low cost.

  On the other hand, with the position detection element, the number of light spots whose position can be known at a time is limited to 2, 4, 8, etc., the light-receiving pattern image itself cannot be taken out, and the color of the light spot is unknown. There are disadvantages. Therefore, when only the positions of the n light spots are captured at a time as in the case of using the position detection element, as described above, the pattern projection unit 102 does not project all the light spots at one time. It is sufficient to project n points at a time, detect the position for each projection, and detect the position of the light spot. Which method is used to calculate the position of the light spot is determined in consideration of cost and required performance.

  When the pattern light reception analysis unit 103 obtains the light reception pattern r, the pattern comparison unit 104 starts comparison between the reference pattern d and the light reception pattern r, and the light spot d [i, j] of the reference pattern d and the light reception pattern. The correspondence relationship between r and the light spot r (x, y) is specified. That is, the pattern comparison unit 104 fills in the “light receiving pattern light spot position” column (shaded portion in the figure) of the correspondence table as shown in FIG. Hereinafter, the light spot r (x, y) of the light receiving pattern r corresponding to d [i, j] is expressed as r [i, j].

  There are the following three methods for specifying this correspondence.

[First method]
First, the first method will be described.

  This is because the light spot d [i, j] of the reference pattern is sequentially projected point by point, and the position Point (x, y) on the projection plane 120 is read each time, thereby d directly. The correspondence between [i, j] and r (x, y) is specified.

  For example, when the reference pattern is used, instead of projecting all the light spots at the same time, each pattern is projected one by one, and the pattern light receiving analysis unit 103 performs the light receiving operation when one light spot is projected. Then, it can be directly known without calculating, which one of the light spots r (x, y) of the light receiving pattern corresponds to the light spot d [i, j] of the reference pattern.

  This method is simple and can easily cope with a case where there is a region with low reflectance on the projection surface and there is a partial defect in the light receiving pattern image. On the other hand, this sequential projection method has a drawback that it takes time because light spots must be projected one by one. For example, as shown in FIG. 3, when a reference pattern having 63 light spots is used, if the pattern projection unit 102 projects 10 images (0.1 second per sheet) in 1 second, 63 points are obtained. It takes 6.3 seconds to project light spots one by one. In order to increase the accuracy, in actuality, the number of light spots must be increased to further reduce the distance between the light spots, and in addition to this, the operation time of the light receiving section of the pattern light receiving analysis section 103 can be reduced. Considering this, it will take more time.

[Second method]
Next, the second method will be described.

  This is because the light spot d [i, j] of the reference pattern and the light spot r (x, y) of the light receiving pattern obtained as a result of the position detection after the position detection of all the light spots of the light receiving pattern is completed. Is calculated. Specifically, the correspondence is calculated by the following process from Step A1 to Step A4 and the process from Step B1 to Step B6. Steps A1 to A4 are calculation methods when light spots corresponding to the light spots at the four corners of the reference pattern exist in the light receiving pattern. On the other hand, Steps B1 to B6 are calculation methods when one or more light spots are missing among the light spots corresponding to the light spots at the four corners of the reference pattern.

  Now, it is assumed that a light receiving pattern image as shown in FIG. 9 has been obtained, and all the positions of the respective light spots have been detected.

(Step A1)
In the light receiving pattern r, the point having the smallest y coordinate value is designated as r0, and the largest point is designated as r1.

ここでｎ、ｍは定数であり、ｙはｘの１次関数となる。 (Step A2)
A straight line r0-r1 connecting r0 and r1 is obtained. Then, a straight line 403 passing through r0 and r1 is defined as an expression (Addition 1).

Here, n and m are constants, and y is a linear function of x.

(Step A3)
Of the light spots of the light receiving pattern r, two points that are farthest from the straight line r0-r1 are obtained. Of these two points, the point in the region on the straight line r0-r1 (the region in the negative direction on the y axis) is r2, and the point in the lower region is r3.

Specifically, out of the light spots of the light receiving pattern r, two points that are most distant from each other between the area above the line 403 and the area below the line 403 are obtained. However, since it is sufficient that the point with the maximum distance is known and the actual distance is not necessary, a simple calculation method is sufficient. That is, among the respective light spots (xi, yi) of the light receiving pattern r, the distance R shown in the equation (2) is the minimum (on the straight line 403, that is, the negative direction of the y axis) and the maximum point (the straight line). What is necessary is just to obtain | require under 403, ie, the positive direction of an axis | shaft.

  For example, in the example shown in FIG. 9, r2 is the light spot 404 and r3 is the light spot 405.

(Step A4)
R0, r1, r2, and r3 obtained in steps A1 to A3 correspond to the light spots at the four corners of the rectangle forming the light receiving pattern image. That is, r0 or r2 is r [0,0] or r [8,0], that is, the upper two vertices among the four vertices of the rectangle forming the light receiving pattern image. Of r0 and r2, the smaller coordinate x value is r [0,0]. In the case of the light receiving pattern image as shown in FIG. 9, r0 is r [0,0] and r2 is r [0,8]. Similarly, r1 is r [6,8] and r3 is r [0,6].

  As a result, the correspondence relationship between the light spots at the four corners of the rectangle of the reference pattern as shown in FIG. 10 and the light spots at the four corners of the rectangle of the light receiving pattern is specified.

  If the correspondence between the four points is known, the mapping from the reference pattern to the light receiving pattern can be calculated, and the correspondence between the remaining points can be understood by using the mapping. The correspondence between the remaining points will be described again in detail after describing the mapping calculation method.

  The case where the above-mentioned light spot is not missing is established when there is a premise that the projection surface 120 has uniform reflectance and color and can read all the projected light spots. However, in an actual mobile environment, the reflectance or color is not always uniform. Therefore, it may be difficult to read all four corners of the reference pattern rectangle.

  Therefore, next, when light spots are missing from the light receiving pattern image, a case where light spots corresponding to the four corner points of the reference pattern rectangle are included will be described. In this case, (1) the four points at the four corners that are not estimated can be complemented by guessing from the read points, or (2) the four points at the four corners that correspond to d [i, j] from the read points. It is necessary to select.

  Here, the first case (1) will be described first.

  Now, as shown in FIG. 11, it is assumed that r [8,0] (light spot 505) and some surrounding light spots are missing in the light receiving pattern image. However, at the present time, the number of light spots is less than 63, and it can be understood that some point is missing, but the position of the missing point is r [8,0] and its I don't know if it's around.

  In such a case, a method for finding the missing light spot and estimating the position on the light receiving pattern image will be described.

(Step B1)
First, the same operations as in steps A1 to A4 are performed, and r0 ′, r1 ′, r2 ′, and r3 ′ are obtained as candidates for the light spots at the four corners of the rectangle forming the light receiving pattern image. In FIG. 11, r 0 ′ corresponds to the light spot 501, r 1 ′ corresponds to the light spot 502, r 2 ′ corresponds to the light spot 503, and r 3 ′ corresponds to the light spot 504. Here, since r [0,8] (light spot 505) is lost on the light receiving pattern image, r2 'is not actually one of the four corner points of the rectangle.

  Now, when the total number of light spots of the light receiving pattern image is less than the total number of light spots of the reference pattern, the process proceeds to the following steps.

式（３）は、ｘの１次関数となる。 (Step B2)
First, a straight line expression (3) representing a straight line connecting four points r0 ′, r1 ′, r2 ′, and r3 ′ is obtained.

Equation (3) is a linear function of x.

  A straight line connecting r0′r2 ′ is a straight line 506, a straight line connecting r2′r1 ′ is a straight line 512, and similarly, a straight line connecting r1′r3 ′ is a straight line 508, and a straight line connecting r3′r0 ′ is the same. A straight line 509 is assumed.

(Step B3)
Now, assuming that the light spot is (x, y), the x coordinate of the light spot of the light receiving pattern is substituted for each straight line equation, and the y coordinate value (Y = lmn ′ (x)) obtained as a result is The actual y-coordinate value is compared, and all the light spots are assigned to the following three groups.

As group 1 (on a straight line), select an intersection where:
| Y−Y | ≦ α (α is a value corresponding to an error of the position detection element)
For group 2 (below the straight line), choose an intersection where:
| Y−Y |> α and y−Y> 0
For group 3 (above the straight line), select the intersection where:
| Y−Y |> α and y−Y <0
Here, a straight line is selected in which the elements of group 2 are not empty and the elements of group 3 are not empty. Since light spots exist above and below the straight line selected in this way, it can be seen that the straight line is not a rectangular side forming the light receiving pattern image.

  In the example shown in FIG. 11, the straight line 512 (light spot 507) is a straight line in which the elements of the group 2 are not empty and the elements of the group 3 are not empty. In the example shown in FIG. 11, at this stage, two points at the four corners of the correct rectangle are determined. The correct rectangle means a pattern image rectangle corresponding to the reference pattern rectangle.

  Specifically, when the element of group 2 is empty and the straight line of element 3 of group 3 is lmn ′, the point of i′′n and i ≠ m in ri ′ is There are two points each in the four corners of the correct rectangle. In the example shown in FIG. 11, since the straight line in which the group 2 element is empty and the group 3 element is empty is a straight line l21 ′, the light spots at the two corners of the right rectangle are r0. 'And r3'.

  In this step, if there are two or more such straight lines, there are too many missing light spots, and this method makes it difficult to correct the missing light spots.

(Step B4)
Next, out of the remaining two light spots, the light spots at the four corners of the correct rectangle are selected.

  The following processing is performed for both of the remaining two points.

(Step B4-1)
Of the light spots assigned to group 2 and group 3 in step B3, a group that does not include light spots at the four corners of the correct rectangle is designated as group g. Further, the straight line selected in step B3 is assumed to be lx. Specifically, in the example illustrated in FIG. 11, the straight line lx is a straight line 512 (light spot 507).

  Among the light spots included in the group g, a point that is the farthest from the straight line lx is selected and is set as v0. Then, a straight line connecting the point assumed to be the four corner points of the correct rectangle and the point v0 is obtained. Let this straight line be lv0. Then, the same processing as that performed in step B3 is performed on the straight line lv0, and groups 1, 2, and 3 are obtained.

(Step B4-2)
Here, if there is no light spot included in group 2 or group 3, there is a possibility that straight line lv0 represents a correct rectangular side, and the processing ends here, and step B4- Run from 1. If there are no remaining light spots, the process proceeds to step B5.

  On the other hand, when both the group 2 and the group 3 are not empty, the line lv0 is newly set as the line lx and the process is repeated from step B4-1.

(Step B5)
In this case, two correct rectangular side candidates can be calculated. Of these, the side with the larger number of elements in group 1 for each straight line is defined as the correct rectangular side.

  In the example illustrated in FIG. 11, the straight line 513 and the straight line 512 remain as candidates for a correct rectangular side. At this time, the number of groups 1 of the straight lines 513 and 512 (that is, the number of light spots on each straight line) is two or three. Accordingly, since the number of elements in group 1 for the straight line 512 is larger than the number of elements in group 1 for the straight line 513, the straight line 512 can be recognized as a correct rectangular side. It can also be seen that the light spot that was not the four corners of the correct rectangle, that is, the light spot that was incorrect, is r2 '.

(Step B6)
An intersection point between the remaining correct rectangular side and the correct rectangular side including the light spot which is erroneously set as the four corner points is obtained. This intersection is the last one of the four corner points of the correct rectangle. In the example shown in FIG. 11, this is the intersection of the remaining correct rectangular side, that is, the straight line 512, and the correct rectangular side including the light spot r2 ′ that is erroneously set as the four corner points, that is, the straight line 506. The light spot 505 is obtained.

[Third method]
Finally, the third method will be described. The third method is an intermediate method between the first method and the second method. The first method sequentially projects one point at a time and obtains a light receiving pattern for the projection. In the second method, all the points are projected, and then the light receiving pattern of all the points is obtained. In the third method, a plurality of points that are not all points are projected simultaneously, and the light reception pattern is obtained.

  The number of a plurality of points to be projected simultaneously is set to be equal to or less than the number of light points that the pattern light receiving analysis unit 103 can detect simultaneously. For example, when the number of light spots that can be simultaneously detected by the pattern light reception analysis unit 103 is four, the number of light spots to be projected is set to four or less simultaneously. In this case, the light spots of d [0, 0], d [0, 6], d [0, 8], d [6, 8], that is, the four corner points of the reference pattern are projected.

  Here, if the light spots of the light receiving pattern with respect to the projected four points are obtained as they are, a mapping from the reference pattern to the light receiving pattern is obtained. Therefore, after the mapping is obtained, the light of the reference pattern other than the light spots at the four corners is obtained. The correspondence between the points and the light spots of the light receiving pattern is obtained. A method for obtaining the mapping and a method for obtaining the correspondence between the light spot of the reference pattern after the mapping is obtained and the light spot of the light receiving pattern will be described later.

  If there are missing light spots among the four corner points of the rectangle of the reference pattern, the surrounding points of the light spots lacking the reference pattern are projected to check whether a light receiving pattern image can be obtained. For example, the surrounding points for d [0, 0] of the reference pattern are d [1, 0], d [0, 1], or d [1, 1]. The points to be selected as the surrounding points may be random or may follow an appropriate rule. For example, in FIG. 12, the order from d [0, 0] to d [0, 3] is indicated by arrows. If a light reception pattern image for d [0, 0] cannot be obtained, for example, one light spot of the reference pattern is projected in the order of the arrows in FIG. 12, and the process is repeated until a light reception pattern image is obtained. Good.

  The description of the method for finding the missing light spot and estimating the position on the received light pattern image is completed.

[Mapping]
Next, a method for obtaining a mapping from the reference pattern to the light receiving pattern will be described.

ただし、（Ｘ，Ｙ）は、受光パターンｒ上の点で、（ｘ，ｙ）は、基準パターンｄ上の点である。 When the projection surface 120 is a plane, the mapping from the point d (x, y) on the reference pattern d to the point r (X, Y) on the light receiving pattern r is projective transformation. In the case of projective transformation, the mapping can be expressed as Equation (4) and Equation (5).

However, (X, Y) is a point on the light receiving pattern r, and (x, y) is a point on the reference pattern d.

In the equations (4) and (5), the coefficients a _0, a ₁ , a ₂ , b _0, b ₁ , b ₂ and the constants c _0, c ₁ , c ₂ are the mapping, that is, the projective transformation. This is a mapping parameter. In order to calculate these nine mapping parameters, it is only necessary to know which light spots on the light receiving pattern r correspond to at least four light spots on the reference pattern d which is the original side of the mapping.

  First, as these four points, rectangular vertices forming the reference pattern and the light receiving pattern, that is, d [0, 0], d [0, 6], d [8, 0], d [8, 6 ] And r [0, 0], r [0, 6], r [8, 0], r [8, 6]. These points are selected because it is possible to obtain a better accuracy by calculating the map in as large a region as possible. FIG. 10 shows the correspondence between these points.

Then, d [i, j] = (Xi, yi) and r [i, j] = (xi, yi) are set, and (Xi, yi, xi, yi) is expressed by the equations (4) and (5). Substituting into (X, Y, x, y) and solving the simultaneous formula, the mapping parameters a ₀ , a ₁ , a ₂ , b ₀ , b ₁ , b ₂ , c ₀ , c ₁ , c ₂ are Determined.

  FIG. 13 is a diagram schematically showing an example of this mapping. FIG. 13A is an original image, and FIG. 13B is an image obtained by mapping conversion with respect to FIG. 13A.

  When the mapping is determined in this way, the correspondence between points other than the light points at the four corners of the rectangle used for determining the mapping parameter can also be determined. Assume that the mapping is W and the result (xdi, ydi) of projective transformation on d [i, j] is obtained.

(Xdi, ydi) = W (d [i, j])
Here, dist (p1, p2) is a function for calculating the distance between the points p1 and p2. In addition, p1 and p2 show the coordinate of a point.

なお、αは、位置検出素子の誤差相当の値である。 At this time, if there is a point (X, Y) on the light receiving pattern that satisfies Expression (6), the point is a point corresponding to d [i, j], that is, r [i, j].

Α is a value corresponding to an error of the position detection element.

  Then, when the corresponding r [i, j] is found for the light spots of all the light receiving patterns, that is, when the light spots of the light receiving patterns that can be observed in the shaded portion of the correspondence table of FIG. The pattern comparison unit 104 ends the operation once. This correspondence table is recorded in the pattern light reception analysis unit 103 and used in a later process.

  Here, since the operations of the pattern projection unit 102, the pattern light reception analysis unit 103, the pattern comparison unit 104, and the position distortion correction unit 106 are relatively complicated processes, they are summarized as flowcharts in FIGS.

  The flowchart of FIG. 14 shows the operations of the pattern projection unit 102 and the pattern light reception analysis unit 103 in the first method of projecting light points of the reference pattern one by one. On the other hand, in the flowchart of FIG. 15, the pattern projection unit 102, the pattern light reception analysis unit 103, and the pattern comparison unit 104 in the third method of projecting two or more light points simultaneously among the light points of the reference pattern. Shows the operation.

  In any of the first method to the third method, finally, a correspondence table between the light spots of the respective reference patterns and the light spots of the light receiving pattern corresponding thereto is generated as shown in FIG. Based on the generated correspondence table, the correction management unit 105 determines whether position distortion correction, image division correction, or color correction is necessary.

  Next, a method for determining whether various corrections are necessary in the correction management unit 105 will be described.

  First, the correction management unit 105 checks a correspondence table between the light spots of the reference pattern and the light spots of the light receiving pattern as shown in FIG. Here, if the position of the light spot of the reference pattern is different from the position of the light spot of the light receiving pattern corresponding to the light spot of the reference pattern, the correction management unit 105 determines that position distortion correction is necessary.

  Further, if there are light spots of the light receiving pattern for all the light spots of the reference pattern, it can be estimated that there is no region on the projection surface 120 that has a low reflectance and is not suitable for projection. That is, when there are light spots of the light receiving patterns corresponding to the light spots of all the reference patterns, the correction management unit 105 determines that the screen division correction is unnecessary. In this case, the correction management unit 105 performs a pattern comparison operation for color correction. The pattern comparison operation for color correction will be described later.

  On the other hand, what happens if the light spot of the light receiving pattern is missing with respect to the light spot of a certain reference pattern?

  An outline of the operation of the correction management unit 105 when the light spot of the light receiving pattern is lost will be described using two simple examples. Thereafter, a general operation will be described.

  FIG. 16 schematically shows a light receiving pattern image of the first simple example. The light reception pattern image does not always look rectangular. However, since the mapping from the reference pattern to the light receiving pattern should have been determined at this time, therefore, FIG. I want you.

  In FIG. 16, the light spot included in the hatched area 601 on the light receiving pattern 600 is missing. For example, when there is a black column, the light spot of the reference pattern projected on that area is lost. In FIG. 16, the light spots that should be originally shown are indicated by white circles.

  Now, from the comparison between the light receiving pattern 600 and the original reference pattern (FIG. 3), the correction management unit 105 determines that the screen correction division is necessary when there is a region where the light spot is missing. The correction management unit 105 determines that the screen correction division is necessary when there are no light spots of the light receiving patterns corresponding to the light spots of all the reference patterns. Then, the correction management unit 105 estimates a region where the light spot is missing, and detects a region having a low reflectance and not suitable as a projection surface (hereinafter referred to as a “low reflection region”). For example, in the example illustrated in FIG. 16, the correction management unit 105 detects the area 602 as a low reflection area.

  Next, the correction management unit 105 operates the subordinate position distortion correction unit 106, the color correction unit 108, and the screen division correction unit 107 to obtain each correction parameter. The output of each correction unit is a correction parameter for mapping from one screen to another. These three correction units may be activated in any order and may operate in parallel.

  The positional distortion correction unit 106 uses the mapping W from the reference pattern d to the light receiving pattern r to generate a positional distortion correction parameter. The mapping W is a mapping from the reference pattern to the light receiving pattern that has already been calculated by the pattern comparison unit 104.

  The color correction unit 108 generates a color correction parameter for correcting the color (hue, saturation, brightness). Specifically, the color correction unit 108 generates a mapping U that performs correction from the color brightness luminance of the pixels on the original screen to the color brightness luminance of the pixels in the same region on the correction screen. Details of the color correction will be described later. The color correction unit 108 outputs the mapping U.

  The screen division correction unit 107 divides the screen to be projected, and further generates a mapping V that corrects the screen so as to avoid the low reflection region 602 and project. The screen division correction will be described using the specific example of FIG.

[Screen division correction]
First, the screen division correction unit 107 divides the projection screen by the center line of the low reflection region 700 and further reduces the divided screen so as not to cover the low reflection region 700. In the example illustrated in FIG. 17, the screen division correction unit 107 reduces the area 701 on the left side of the divided screen to the area 702. Similarly, the screen division correction unit 107 reduces the area 703 of the right screen to the area 704 so as to avoid the low reflection area 700.

  Then, the screen division correction unit 107 calculates a mapping V to a screen obtained by dividing and reducing the original screen (reference pattern).

  FIG. 18 schematically shows this mapping V. FIG. 18 is an example in which the image shown in FIG. 13A is an original image. For the convenience of drawing, the white part in the figure indicates a dark part in the actual image, and the black part in the figure indicates a bright part in the actual image. By applying the mapping V to the original image (FIG. 13A), the original image is displayed in the region 801 and the region 802, avoiding the low reflection region 800, as shown in FIG.

Then, the correction management unit 105 obtains a correction map H by integrating the above maps W, U, and V. The corrected map H is a map from the original image (g) to the image (G) to be projected, and the relationship of the following expression (7) is between the corrected map H and each of the maps W, U, and V. There is.

  FIG. 19 schematically shows the correction map H. Note that the mapping U is a mapping for color correction, and is difficult to express in the drawing. Therefore, in the example shown in FIG. 19, it is assumed that the mapping U does nothing. First, FIG. 19B is a result of performing screen division correction on the pattern of FIG. 19A using the mapping V, and FIG. 19C is a result of performing positional distortion correction on the pattern of FIG. 19A using the mapping W. is there.

  The correction management unit 105 outputs the correction map H shown in Expression (7) to the correction unit 109.

  The correction unit 109 converts the image projected by the projection unit 110 using the mapping parameter of the correction map H, and the video projection unit 110 projects the converted image. As a result, from the user's viewpoint, it is possible to view all the images with less distortion.

  FIG. 20 schematically shows a light receiving pattern image of the second simple example. The example shown in FIG. 20 is an example when there is a cross-shaped low reflection region 900 on the projection surface. As shown in FIG. 20, when the projection surface has a cross-shaped low reflection region 900, the screen division correction unit 107 divides the screen into four. Then, the screen division correction unit 107 reduces each screen so that each divided screen does not cover the low reflection region 900. FIG. 21 shows a state of screen division and screen reduction. In the example illustrated in FIG. 21, the upper left area 901 of the four divided areas is reduced to the area 902. As a result, the image shown on the original screen (FIG. 13A) is divided and displayed as shown in FIG.

  The outline of the operations of the pattern comparison unit 104 and the correction management unit 105 when the light spot of the light receiving pattern is lost has been described above using two simple examples.

  Finally, a general example is shown.

  In FIG. 23, an area 1000 indicates a light spot group obtained by performing reverse mapping of the map W on the light receiving pattern, and shows a state in which the low reflection area 1001 crosses over the area 1000 and some light spots are missing. Yes. In FIG. 23, the missing light spot is indicated by a white circle.

  First, in order to determine a screen that does not cover the defect region 1001, the light spots on the upper side AE and the lower side BF of the rectangle ABFE are viewed from the left, and the last point (D ′ and C ′) entering the defect region 1001 Then, the points (E ′, F ′) that are missing from the defect region 1001 are obtained. That is, the points C ′, D ′, E ′, and F ′ are light spots on the upper side or the lower side of the rectangle closest to the defective area 1001 among the light spots not included in the defective area 1001.

  Then, the midpoints of D′ E ′ and C′F ′ are obtained and set as D and C, respectively.

Here, the dividing line for dividing the screen is DC. A straight line expression represented by DC is y = Ldc (x), a projective transformation for transforming the rectangle ABCD to ABC′D ′ is V1, and a projective transform for transforming the rectangle DCFH to E′F′FE is V2. The mapping V to be obtained converts (x, y) to (X, Y).
V: (x, y) → (X, Y)

However,
When Ldc (x) ≦ 0 V is V1
When Ldc (x)> 0 V is V2
And

  FIG. 24 schematically shows a part (V1) of this conversion. Of the screen ABFE, the area of the rectangle ABCD, that is, the area of Ldc (x) ≦ 0 is moved to the rectangle ABC′D ′. According to this conversion V, for example, the video in FIG. 13A is divided and deformed as shown in FIG.

  This division conversion can be processed in the same way even when the defect area 1001 is a cross shape and is further divided into a plurality of vertical and horizontal areas as shown in FIG. In the case of FIG. 26, the image is divided into three parts in the vertical direction (direction parallel to the Y axis) and into two parts in the horizontal direction (direction parallel to the X axis), and each is reduced and displayed in a region with high reflectivity.

  Further, when the low reflection region is trapezoidal as shown in FIG. Similarly in this case, as shown in FIG. 28, a map V to be divided (divided by a straight line CD) and deformed (rectangle ABCD to ABC′D ′, and the right half of the map is omitted as part of the map) is calculated. The

  This is the end of the description of the operation of the screen division correction unit 107.

[Color correction]
Next, color correction will be described.

  In the case of colored paper correction, the basic framework of projecting a reference pattern, obtaining a light receiving pattern, comparing the reference pattern and the light receiving pattern, and calculating a correction parameter based on the comparison result is the other framework. Same as correction. However, in color correction, a more appropriate mapping U can be obtained by using a different reference pattern from the reference pattern used for position distortion correction and screen division correction. Therefore, a method using another reference pattern will be described below.

  FIG. 29 shows a reference pattern for color correction. In FIG. 29, only a rectangle is drawn. The reference pattern for color correction is a pattern composed of white pixels that fill the screen. In other words, if each pixel is expressed using an RGB representation of 8 bits for each color, the reference pattern is such that all the points on the screen for 960 pixels vertically and 1280 pixels horizontally are RGB (255, 255, 255). , Pattern (video).

  The above-described reference pattern is expressed by black and white reversal (luminance reversal) for the convenience of drawing, whereas the display of the reference pattern 1100 for color correction is also black and white reversal. Express without doing. The number of vertical and horizontal pixels is an example for explanation, and may be changed according to the performance of the projector.

  This reference pattern 1100 is a pattern projection unit under the overall control of the correction management unit 105 “immediately after” the operation for correcting positional distortion or screen division correction for the reference pattern image 200 shown in FIG. Projected from 102. Since the spatial positional relationship between the projection surface 120, the pattern projection unit 102, and the pattern light reception analysis unit 103 may change with time, "immediately after" the operation for position distortion correction or screen division correction ends. Need to be.

  The shorter the time interval from the end of the operation for position distortion correction or screen division correction until the pattern projection unit 102 projects the reference pattern 1100, the better. Further, if possible, the pattern projection unit 102 and the pattern light reception analysis unit 103 are respectively doubled, one is used for the projection and light reception analysis of the reference pattern image 200, and the other is dedicated to the projection and light reception analysis of the reference pattern 1100. It is preferable to constitute so that.

  In the case of performing position distortion correction and screen division correction using the reference pattern image 200, it is desirable to use some position detection element for the light receiving unit of the pattern light receiving analysis unit 103 as described above. On the other hand, when performing color correction using the reference pattern 1100, a normal camera is used as the light receiving unit of the pattern light receiving analysis unit 103, and therefore color correction is performed in addition to positional distortion correction and screen division correction. In this case, at least the pattern light receiving unit needs to have a double configuration.

  However, this dual configuration is not essential. That is, when performing position distortion correction and screen division correction, the position detection element is desirable for the light receiving unit because the position detection element is relatively inexpensive and can detect the position of the light spot accurately at high speed. . Therefore, the same light reception analysis can be performed by receiving light with a normal camera and performing image analysis from the received light image. In this case, the light receiving unit of the pattern light receiving analysis unit 103 can be configured by a normal CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) camera.

  However, in the position detection element, the position of the light spot is detected by hardware, whereas when a CMOS or CCD camera is used, the position detection of the light spot is processed by software. In order to perform position detection as fast as the position detection element by software processing, a relatively high speed CPU (Central Processing Unit) or GPU (Graphics Processing Unit) is required, and the apparatus must be expensive. .

  Here, it can be said that it is not easy to perform both the positional distortion correction, the screen division correction, and the color correction using the position detection element. This is because the position detection element only calculates the position of a light spot having a certain luminance or higher, and it is difficult to determine the luminance. For this reason, it is difficult to calculate information on the reflectance of the projection plane, which is the minimum necessary for performing color correction, only by using the position detection element.

  Now, let's return to the explanation of color correction.

  Immediately after the pattern projection unit 102 projects the reference pattern 1100, the pattern light reception analysis unit 103 receives the image projected on the projection surface and obtains a light reception pattern. FIG. 30 is a light-receiving pattern image obtained by converting the obtained light-receiving pattern by the inverse mapping of the map V obtained at the time of light-receiving analysis of the previous reference pattern image 200. As a precaution, the received light pattern image before conversion by reverse mapping should be distorted as shown in FIG. 31D. Since the mapping V has already been obtained, the pattern light receiving analysis unit 103 does not need to newly calculate the mapping V. Further, since mapping V is the correspondence between the light spot on the reference pattern and the light spot on the light receiving pattern, it is not necessary to create a new correspondence table as shown in FIG. Therefore, at this time, the operation of the pattern image analysis unit 103 ends.

  Next, the pattern comparison unit 104 starts operation. Similarly, the mapping V has already been calculated, and the comparison between the reference pattern and the light receiving pattern is also finished when the inverse mapping image (1200 in FIG. 29) of the light receiving pattern image is calculated.

  Next, the correction management unit 105 starts operating.

  Here, the correction management unit 105 activates the color correction unit 108 and causes the color correction unit 108 to calculate color correction parameters.

  Now, as shown in FIG. 30, it is assumed that a black region 1201 and gray regions 1202 and 1203 are observed in an inverse mapping image 1200 of the mapping V of the light receiving pattern image. The other areas are assumed to be white.

  As described above, the reference pattern for color correction is a pattern composed of white pixels that fill the screen. Therefore, when the light spot d [i, j] (0 ≦ i ≦ 1280, 0 ≦ j ≦ 960) of the reference pattern is expressed in RGB, all are RGB (255, 255, 255). At this time, it is assumed that the following values are obtained for the light spot r [i, j] of the light receiving pattern 1200.

(1) The points on the area 1201 are RGB (1, 1, 1)
(2) The points on the areas 1202 and 1203 are RGB (127, 127, 127)
(3) Other points are RGB (255, 255, 255)
In order to correct the color or brightness unevenness, it is only necessary to project strong light so as to compensate for the weak reflectivity of the R, G, and B colors where the reflectivity is weak.

  That is, the light spot (pixel) d [i, j] of the reference pattern and the light spot (pixel) r [i, j] of the light receiving pattern are expressed in RGB representation as RGBd [i, j] (r, g, b), respectively. ) And RGBr [i, j] (r ′, g ′, b ′), the reflectance of each point is reflected [i, j] (r ′ / r, g ′ / g, b ′ / b).

  Therefore, in order to perform color correction, for each pixel [i, j] of the image to be projected, the intensities of R, G, B are respectively r / r ′ times, g / g ′ times, b / What is necessary is just to multiply by b '. Basically, for all of these points, the multiplication factor to be multiplied by the respective RGB values is the color correction parameter, and is the mapping U calculated by the color correction unit 108.

  That is, it is only necessary to determine the respective RGB multiplication factors (Ar, Ag, Ab) for the point (i, j) on the image to be projected. Here, Ar is a coefficient that increases the intensity of R (red) light, Ag is a coefficient that increases the intensity of G (green), and Ab is a coefficient that increases the intensity of B (blue).

  Therefore, regarding the area 1201, each RGB color is 255 times (Ar, Ag, Ab) = (255, 255, 255), and regarding the areas 1202 and 1203, each RGB color is twice (Ar, Ag, Ab) = (2, 2, 2), and for other regions, (Ar, Ag, Ab) = (1, 1, 1), that is, do nothing.

  However, in actuality, in the case of the current video projection apparatus, the screen projection is usually performed by subtracting with a display unit such as a liquid crystal that displays the video with respect to the light source. For this reason, since it is difficult to display white light brighter than normal white light on the projection surface, color correction is difficult in all cases.

  For example, when the reflectance of the area 1201 is substantially 0, it is practically difficult to perform color correction. Therefore, the correction of the area 1201 may be left to image division correction. That is, it is preferable not to perform color correction for an area where the reflectance is smaller than a certain threshold.

  This threshold value is preferably determined in relation to the screen division correction. That is, the reflectance at which the screen division correction starts to work may be set as this threshold value. This value can be determined in relation to the brightness of the light spot that can be reacted by the position detection element of the pattern light receiving analysis unit 103 used for screen division correction and position distortion correction.

  In addition, even if the reflectance is not less than a certain threshold, projection is difficult when the reflectance is not less than a certain value. For example, regarding the region 1202 and the region 1203, white (RGB (255, 255, 255) )) Is difficult to reproduce. That is, even if the light RGB (255, 255, 255) having the maximum luminous intensity is projected, the gray of RGB (127, 127, 127) is reproduced.

  Therefore, in order to correct the color or brightness unevenness at as many points as possible, the entire coefficient is multiplied by an appropriate coefficient constant α (0 <α <1) for each coefficient of (Ar, Ag, Ab). Processing such as compressing the intensity of light and non-linear correction such as gamma correction may be further performed. However, in any method, the overall contrast or brightness of the image projection apparatus is lowered, so it is necessary to determine these coefficients α and the like by a trade-off between the naturalness of correction and the brightness or contrast of the screen. is there.

  Furthermore, since the reflectance of the projection surface 120 is not always linear in relation to the lightness of the light spot, fine adjustment may actually be required even within a correctable range.

  This is the end of the description of the operation of each of the three correction units.

  FIG. 31 shows an example of combinations of correction results calculated by these correction units. In FIG. 31, there is a low reflection region such as a black column in the center of the projection surface 120, and the lower portion of the projection surface is colored in gray, so that the user can place the video projection device around his left foot. An example of the correction result in a situation where the user is standing and looking at the projection surface is shown.

  FIG. 31A is an image to be presented to the user. Please note that it is a black-and-white inverted image for the sake of drawing. First, this image is subjected to screen division correction (mapping V). As a result, FIG. 31B is obtained.

  Next, when color correction (mapping U) is performed, FIG. 31C is obtained. This is a case where the correction coefficient α for the entire light intensity compression is set to 0.5, and the projection light with respect to the white area on the projection surface is weakened by half as a whole (gray area in FIG. 31C). On the other hand, regarding the gray area of the projection plane (lower part of the projection plane), when the intensity is doubled, 0.5 is applied as a correction coefficient for luminous intensity compression, and as a result, the projection is performed with the same luminous intensity.

  Finally, positional distortion correction (mapping W) is performed, and as a result, FIG. 31D is obtained.

  As described above, in the present embodiment, the pattern comparison unit 104 compares the received light pattern obtained by receiving the projected reference pattern with the reference pattern, and specifies the correspondence between the components of these patterns. Then, the mapping W from the reference pattern to the light receiving pattern is calculated based on the correspondence, and the correction management unit 105 corrects the position distortion or splits the screen based on the correspondence between the constituent elements of the reference pattern and the light receiving pattern. The necessity of correction is determined, and the position distortion correction unit 106 and the screen division correction unit 107 are operated according to the determination result. The position distortion correction unit 106 generates a correction parameter for position distortion corresponding to the mapping W, and the screen division correction unit 107, based on the correspondence relationship, is a region where a region corresponding to the reference pattern is missing from the light receiving pattern. Detecting a low reflection area including the image, dividing the screen on which the image is projected so that the image is projected on an area other than the low reflection area, calculating a mapping V from the image before the division to the image after the division, An image division correction parameter is generated according to the mapping V. The correction unit 109 corrects the video content using the position distortion correction parameter or the screen division correction parameter, and the video projection unit 110 projects the corrected video content. As a result, distortion of the projected image is corrected, and even if there is a band-like area with low reflectivity such as a column or a surface with a different material on the projection surface, the remaining area suitable for projection can be efficiently used. By using it often, it is possible to project an easy-to-see image.

  Hereinafter, the calculation frequency of the correction parameters for these mappings will be described. In the case of positional distortion correction or screen division correction, the pattern projection unit 102 can also project the reference pattern using infrared light that is invisible to humans. When infrared rays that are invisible to humans are used, correction parameters can be calculated as needed in parallel with video projection. Therefore, in this case, even when the positional relationship between the video projection device and the projection surface changes with time, such as when the video projection device 100 is held by hand, the positional distortion correction and the screen are performed in real time. Division correction is possible.

  With respect to color correction, it is difficult to accurately calculate color correction parameters unless the projection surface is actually illuminated with visible light. For this reason, it is difficult to calculate the color correction parameter in a situation in which the reference pattern projected for correction is perceived by the user and is not completely noticed by the user.

  For this reason, as one solution, the image projection apparatus 100 is mounted with some vibration sensor or three-dimensional position sensor, and the correction operation is executed only when the position of the image projection apparatus 100 is changed. Thus, it is conceivable to minimize the start of the correction operation.

  Alternatively, by using only one frame of the video frames projected by the video projection device 100 for a certain period (for example, 1 second) for the projection of the reference pattern, it is difficult for the user to notice the projection of the reference pattern. Various methods are also conceivable. It is also possible to make the reference pattern itself gray instead of white, or to consider the image of the projected video content itself as the reference pattern and calculate the color correction parameters. .

  Note that the projected image itself may be used as the reference pattern instead of the reference pattern for color correction. Even in this case, the method described in this embodiment can be used as it is as the method for calculating the color correction parameter. However, an image with a low overall saturation, lightness or hue, and high brightness (for example, a light cloudy sky or a snow scene) is advantageous in calculating the color correction parameters. For example, it is possible to intensively calculate color correction parameters when a simple image is projected.

(Embodiment 2)
In Embodiment 1, the case where the projection surface is a plane has been described as an example, but the present invention can also be applied to a case where the projection surface is a curved surface. In this embodiment, a case where the projection surface is a curved surface will be described.

  When the projection surface is a curved surface, the calculation method of the mapping W for correcting the positional distortion in the pattern comparison unit 104 is different from that when the projection surface is a flat surface. When the projection surface is a curved surface, the pattern comparison unit 104 approximates the curved surface with a minute rectangular plane, and approximates the whole mapping by using projection mapping between the respective minute rectangular planes.

  As a minute rectangular plane, a minute plane surrounded by four points d [i, j], d [i + 1, j], d [i, j + 1], and d [i + 1, j + 1] is considered. In FIG. 32, each microplane is a square area surrounded by a broken line. For example, the hatched area 1300 is an example of a minute plane.

  The pattern comparison unit 104 performs projective transformation F onto a plane on the received light pattern image surrounded by four points r [i, j], r [i + 1, j], r [i, j + 1], and r [i + 1, j + 1]. Find (i, j). A group of projective transformations F (i, j) is defined as an entire map W.

  Unlike the case where the projection surface is a flat surface, when the projection surface is a curved surface, linear interpolation cannot be performed when obtaining a correspondence from d [i, j] to r [i, j]. A method (second method) in which a large number of light spots are projected and the position is detected cannot be used.

  As described above, when the projection surface is a curved surface, a one-point projection method (first method) or a method of simultaneously projecting a plurality of light spots (third method) as shown in the flowchart of FIG. 14 or FIG. Use. In the method of projecting a plurality of light spots at the same time, the number of light spots to be simultaneously projected is limited to 4 at most, and each time four points are projected, the correspondence between the light spot of the reference pattern and the light spot of the light receiving pattern. It is necessary to take it. In addition, when a missing light spot is generated in the light receiving pattern image, it is necessary to confirm the correspondence of the light spot only in that area by the one-point projection method.

  Note that, when the third method is used, the maximum four-point projection is performed because the positional relationship between the four points is not changed even when the projection surface is a curved surface.

  The other calculations and the operation of each unit are the same as those in the first embodiment, and thus description thereof is omitted.

  As described above, in the present embodiment, when the projection surface is a curved surface, the pattern comparison unit 104 obtains the projective transformation F (i, j) to the plane on the light receiving pattern image surrounded by the four points, and performs the projective transformation. A group of F (i, j) is defined as the entire map W.

(Embodiment 3)
In the first embodiment, the pattern light reception analysis unit 103 is disposed in the vicinity of the video projection unit 110, and the pattern projection unit 102 is a position independent of the video projection unit 110 and close to the position of the user's eyes. It was supposed to be arranged. On the other hand, it is also possible to obtain the same effect by arranging the video projection unit 110 and the pattern projection unit 103 close to each other and arranging the pattern light receiving analysis unit 103 near the position of the user's eyes.

  In the present embodiment, a case will be described in which the image projection unit 110 and the pattern projection unit 103 are arranged close to each other, and the pattern light reception analysis unit 103 is arranged near the position of the user's eyes.

  FIG. 33 is a configuration diagram of a video projection system according to Embodiment 3 of the present invention. In the video projection system according to the present embodiment shown in FIG. 33, the same reference numerals as those in FIG. In the video projection system of FIG. 33, the video projection unit 110 and the pattern projection unit 103 are arranged close to each other, and the pattern light receiving analysis unit 103 is arranged near the position of the user's eyes.

  With the arrangement as shown in FIG. 33, in the present embodiment, the mapping W for correcting the positional distortion is read as an inverse mapping of the mapping W for correcting the positional distortion used in the first embodiment. It will be necessary. That is, after calculating the mapping from the light spot d [i, j] of the reference pattern to the light spot r [i, j] of the light receiving pattern in the procedure in the first embodiment, in this embodiment, the positional distortion correction is performed. As the mapping W for, a reverse mapping of the calculated mapping is used.

  Other calculations and the operation of each unit are the same as those in the first embodiment, and thus the description thereof is omitted.

  As described above, in the present embodiment, the video projection unit 110 and the pattern projection unit 103 are arranged close to each other, and the pattern light reception analysis unit 103 is arranged near the position of the user's eyes. As a result, as can be seen from the configuration diagram of FIG. 33, all or a part of the pattern light reception analysis unit 103 (position detection element part or camera part) can be replaced with an external device having a function equivalent to these parts. .

  For example, when the video projection apparatus 100 according to the present embodiment has a wireless communication function and the user has a mobile terminal with a camera function and a wireless communication function, the video projection apparatus 100 By obtaining the camera image acquired by the mobile terminal from the user's mobile terminal via wireless, all or part of the pattern light receiving analysis unit 103 can be replaced with the user's mobile terminal.

  Furthermore, when the user's mobile terminal can download and execute an application from the outside, by executing various calculation functions of the pattern light reception analysis unit 103 on the mobile terminal, as an alternative to the pattern light reception analysis unit 103, The portable terminal can be used.

  In the above description, the pattern comparison unit 104 generates the mapping W from the reference pattern to the light receiving pattern. However, the positional distortion correction unit 106 generates the mapping W based on the correspondence table. Also good.

  The image projection apparatus and the image projection method according to the present invention efficiently perform the remaining region suitable for projection even in the case where there is a strip-shaped region with a low reflectance such as a column or a surface with a different material on the projection surface. Can be used frequently and can be easily seen by humans, especially when it is difficult to use a dedicated projection surface, such as when used in a mobile environment, due to distortion of the projected image or low reflectance of the projection surface Therefore, it is useful as a video projection device or the like that improves the difficulty of seeing the projected video generated and makes it easy to see.

DESCRIPTION OF SYMBOLS 100 Image projection apparatus 101 Reference | standard pattern generation part 102 Pattern projection part 103 Pattern light reception analysis part 104 Pattern comparison part 105 Correction management part 106 Position distortion correction part 107 Screen division correction part 108 Color correction part 109 Correction part 110 Video projection part 120 Projection surface 130 Video content

Claims (7)

A reference pattern generating means for outputting a reference pattern;
Pattern projection means for projecting the reference pattern as an image;
The light receiving pattern obtained by receiving the projected reference pattern is compared with the reference pattern, the correspondence between the components of these patterns is specified, and the light receiving pattern is derived from the reference pattern based on the correspondence. A comparison means for calculating the mapping W to
A position distortion correction unit that generates a correction parameter for position distortion; and a screen division correction unit that generates a correction parameter for screen division, and determines whether or not position distortion correction or screen division correction is necessary based on the correspondence relationship. And a correction management unit that operates the positional distortion correction unit and the division correction unit according to the determination result;
Correction means for correcting video content using the correction parameter for position distortion or the correction parameter for screen division;
Video projection means for projecting the corrected video content,
The positional distortion correction means generates the positional distortion correction parameter according to the mapping W,
The image division correction unit detects a low reflection region including a region in which a region corresponding to the reference pattern is missing from the light receiving pattern, and the image is projected onto a region other than the low reflection region. Dividing the screen on which the video is projected, calculating a mapping V from the image before the division to the image after the division, and generating the correction parameter for image division according to the mapping V;
Video projection device. When the position of the light spot of the reference pattern is different from the position of the light spot of the light receiving pattern corresponding to the light spot of the reference pattern, the correction management unit determines that positional distortion correction is necessary, and all the reference When there is a light spot of the light receiving pattern corresponding to the light spot of the pattern, it is determined that screen division correction is unnecessary.
The video projection device according to claim 1. The reference pattern generation means generates a pattern for color correction as the reference pattern,
The correction management means further includes color correction means for calculating a color correction parameter for correcting hue saturation lightness based on the correspondence relationship;
The correction means further corrects the video content using the color correction parameter;
The video projection device according to claim 1. The projection part of the optical image of the video projection unit also serves as the projection part of the optical image of the pattern projection unit,
The video projection device according to claim 1. The pattern projecting means projects the components of the reference pattern divided into a plurality of parts at intervals for each part;
The video projection device according to claim 1. The pattern projecting means projects the reference pattern using invisible light;
The image projection apparatus according to claim 1. Project the reference pattern as an image,
The light receiving pattern obtained by receiving the projected reference pattern is compared with the reference pattern, the correspondence between the components of these patterns is specified, and the light receiving pattern is derived from the reference pattern based on the correspondence. To calculate the mapping W to
Based on the correspondence, determine whether or not position distortion correction or screen division correction is necessary,
When it is determined that positional distortion correction is necessary, the positional distortion correction parameter corresponding to the mapping W is generated,
When it is determined that screen division correction is necessary, a low reflection region including a region in which a region corresponding to the reference pattern is missing is detected from the light receiving pattern, and the video is projected to a region other than the low reflection region. Dividing the screen on which the video is projected, calculating a mapping V from the image before the division to the image after the division, and generating the correction parameter for the image division according to the mapping V,
Using the correction parameter for positional distortion or the correction parameter for screen division, correct video content,
Projecting the corrected video content;
Video projection method.

Images