JP2014060611A - Image processing apparatus, image projection system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, an image projection system, and a program for controlling projection light in accordance with a positional relation between a projection object and a projection apparatus.SOLUTION: This image processing apparatus 100 comprises: distance measurement means 106 for measuring a distance between plural points within an image and calculating three-dimensional points; object shape designation means 108 for receiving designation of a three-dimensional shape being the projection object; object shape identification means (108, 110) for identifying object three-dimensional points corresponding to the three-dimensional shape of the projection object, from the three-dimensional points; surface estimation means 112 for estimating the approximate surface of the projection object from the object three-dimensional points; and control data calculation means 116 for calculating control data regulating projection light to project to a projection area on the approximate surface.

Description

  The present invention relates to an image processing device, a video projection system, and a program, and more particularly, an image processing device, a video projection system, and a video projection system for controlling projection light corresponding to a positional relationship between a projection object and the projection device, and The present invention relates to a program for realizing the image processing apparatus.

  2. Description of the Related Art In recent years, a technique is known in which a monocular camera or a stereo camera is combined with a projector, a distance to a projection target such as a screen is measured, and a projected image is corrected.

  For example, Japanese Patent No. 3880582 (Patent Document 1) discloses a projector for the purpose of correcting the distortion of a projected image caused by focus adjustment, tilt projection, and the shape of the surface of a projection target. The projector of Patent Document 1 has a plurality of cameras capable of capturing an image of a projection target, projects a plurality of target points having a predetermined pattern onto the projection target in a plane, and measures a distance. It has a function to correct. In the projector, the projection range corresponding to the distance from the projection lens is calculated based on the zoom position, and the same target point in the imaging screen of the plurality of cameras is handled from the imaging screen and the projection range of the plurality of cameras. Thus, the three-dimensional position of the target point is detected, and the distance to the target point is detected. The projector adjusts the focus of the projection lens based on the calculated average distance.

  On the other hand, the projection surface projected by the projector is not only a wide plane such as a screen, but also an arbitrary object that has a predetermined size, such as a human hand, a vehicle body, or a wall having a depression in the center, and can be regarded as a substantially plane. There is a desire to project on an object. Such an arbitrary object usually does not have a black frame used as a screen feature. For this reason, when trying to project onto an arbitrary object, the above-described arbitrary object is not recognized as a projection object in a system based on the assumption that the above-described conventional projection object is a screen. In some cases, it is difficult to apply the correction technique described above.

  The present invention has been made in view of the above-described problems in the prior art, and image processing that enables projection light control according to a projection target to be performed on a projection target of an arbitrary shape designated by a user. An object is to provide an apparatus, a video projection system, and a program.

  In order to solve the above-described problems, the present invention provides an image processing apparatus having the following characteristics. The image processing apparatus includes a distance measuring unit that measures a distance between a plurality of points in an image and calculates a three-dimensional point group, a target shape specifying unit that receives designation of a three-dimensional shape to be projected, and a calculation Target shape specifying means for specifying a target three-dimensional point group corresponding to the three-dimensional shape of the projection target from the three-dimensional point group, and a surface for estimating an approximate surface of the projection target from the specified target three-dimensional point group Estimation means and control data calculation means for calculating control data defining projection light projected onto the projection area on the approximate plane.

  According to the above configuration, it is possible to perform projection light control in accordance with the projection target with respect to the projection target having an arbitrary shape designated by the user.

1 is a schematic configuration diagram of a projector according to a first embodiment. FIG. FIG. 3 is a diagram illustrating functional blocks and data flow in the projector according to the first embodiment. 6 is a flowchart showing projection image correction control executed by the projector according to the first embodiment. 5 is a flowchart illustrating distance measurement processing executed by a distance measurement unit in the first embodiment. The figure explaining the search process of the stereo corresponding point between the stereo images by this embodiment. The flowchart which shows the object shape designation | designated process which the object shape designation | designated part by 1st Embodiment performs. The schematic diagram showing the three-dimensional point group of the scene calculated by the ranging part. The figure explaining the method to designate the desired three-dimensional shape of the projection object from the three-dimensional point group of a scene in 1st Embodiment. The flowchart which shows the object shape search process which the object shape search part by 1st Embodiment performs. The figure explaining the process which searches the object three-dimensional point group corresponding to the object three-dimensional shape of a projection target object from the three-dimensional point group in a scene. The flowchart which shows the plane estimation process which the plane estimation part by 1st Embodiment performs. The flowchart which shows the correction calculation process which the correction calculation part by 1st Embodiment performs. The figure explaining the relationship between the normal plane located in the normal direction of the approximate plane which approximates the target three-dimensional shape of a projection target object, and an approximate plane. The figure explaining the process which converts a three-dimensional point group into the two-dimensional point group on a normal camera image by perspective projection conversion. The figure explaining the process which obtains a correction rectangle from the two-dimensional point group showing the shape of the projection target object on a normal camera image. Diagram for explaining a two-dimensional point group on normal camera image I _V, a correspondence relationship between the feature points on the projector image I _P to be projected. The figure explaining the geometric transformation by projective transformation matrix _HVP and projective transformation matrix _HPP . (A) The figure explaining the trapezoid correction process of the content image by the projective transformation matrix _HPP , and (B) the projection image actually projected after the correction. The figure which illustrates the utilization aspect which applies a projection image correction following the rotation of a projection target object. The figure which illustrates the utilization aspect which uses the wall surface which has a hollow shape as a projection target object. The figure explaining the method of designating the three-dimensional shape of the desired projection object in 2nd Embodiment. The figure explaining 3rd Embodiment controlled to project projection light according to the shape of a projection target object. The schematic block diagram of the image | video projection system in 4th Embodiment. The schematic block diagram of the video projection system in 5th Embodiment. The figure which illustrates the image | video projection system comprised using the short focus projector and the external distance sensor.

  Hereinafter, although this invention is demonstrated with embodiment, this invention is not limited to embodiment mentioned later.

(First embodiment)
In the first embodiment described below, a projector 100 including a stereo image type distance measuring sensor will be described as an example of an image processing apparatus and a video projection system.

[overall structure]
FIG. 1 shows a schematic configuration of a projector 100 according to the first embodiment. The projector 100 includes a control unit 102 that performs overall control of the projector 100, an imaging unit 104 that forms a stereo image type distance measuring sensor, a distance measuring unit 106 that performs stereo image type parallax calculation, and a focus adjustment unit 120. And a projection unit 122.

  The imaging unit 104 includes a plurality of cameras. Each of the cameras includes a solid-state imaging device such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and an imaging optical system such as a lens for imaging on the solid-state imaging device. In the embodiment to be described, the imaging unit 104 includes two cameras arranged in parallel equiposition in the horizontal direction or the vertical direction. The distance between the plurality of solid-state image sensors defines a base line length and is highly position-controlled in order to improve distance measurement accuracy.

  The projection unit 122 includes a projection apparatus such as a liquid crystal system, a CRT (Cathode Ray Tube) system, a DLP (Digital Light Processing) system, and an LCOS (Liquid Crystal On Silicon) system. The projection unit 122 projects a content image input to the projector 100 or generated from a file by the projector 100 from the projection device toward the projection target. The focus adjustment unit 120 adjusts the focus of the projection apparatus based on the measurement result of the separation distance from the projection apparatus to the projection target.

  The projector 100 according to the present embodiment recognizes the position and shape of the projection object in the scene imaged by the camera of the imaging unit 104 based on the distance measurement result by the distance measurement unit 106, and moves to the area of the projection object. A function for correcting and projecting a content image is provided. In the prior art, the recognizable projection object is limited to a relatively wide plane such as a screen in an area captured within the angle of view, such as a human body, a human hand, a wall with a recessed central part, etc. It was difficult to recognize this object as a projection target. On the other hand, the projector 100 according to the present embodiment realizes a projection light control function for controlling projection light using an object of an arbitrary shape that can be specified by the user as a projection target.

  In order to realize the projection light control function, the projector 100 according to the present embodiment further includes the target shape designating unit 108 for designating the above-described arbitrary shape projection target object, and a scene (scene) captured at a certain point in time. And a target shape search unit 110 for searching for a three-dimensional shape (hereinafter referred to as a target three-dimensional shape) corresponding to the designated projection target. Further, projector 100 calculates correction data from the estimated approximate plane, plane estimation unit 112 that estimates the approximate plane of the searched target three-dimensional shape, position change determination unit 114 that determines a change in the approximate plane. A correction calculation unit 116 and a projection image correction unit 118 that performs image correction on the content image using the calculated correction data are included.

  Hereinafter, the function of each functional unit will be described with respect to the projection light control function with reference to FIG. FIG. 2 is a diagram illustrating functional blocks and data flow in the projector 100 according to the first embodiment.

  In response to the imaging command from the control unit 102, the imaging unit 104 captures a scene in which at least a part of the projection image projected by the projection unit 122 is captured with two cameras substantially simultaneously, and image data of a stereo image Is output. In the stereo image system, one camera may be referred to as a reference camera and the other may be referred to as a reference camera. However, in the embodiment described, the two cameras are arranged in a horizontal direction, and the right camera is The reference camera is used as the reference camera, and the left camera is used as the reference camera. The positional relationship between the left and right cameras is parallel, and when the three-dimensional coordinates in the real three-dimensional space are represented by (X, Y, Z), the X axis of the reference camera and the reference camera are the same and the Y axis is parallel. (3) The Z-axis is also parallel. In the embodiment to be described, it is assumed that parallel shooting is performed in which the sensor optical axes of the left and right cameras are parallel.

  The distance measuring unit 106 receives the input of the stereo image data from the imaging unit 104 and measures the parallax amount of the stereo corresponding point observed in the stereo image. The distance measuring unit 106 calculates the distance of each corresponding point in the image from the measured parallax amount based on the principle of triangulation, obtains a three-dimensional point group constituting the captured scene, and uses it as distance measurement data. Output. In the embodiment to be described, feature points are extracted from each stereo image by a process that will be described in detail later, and feature points that are associated with each other are used as stereo correspondence points. In a preferred embodiment, the feature points between stereo images can be associated with high accuracy by interposing the feature points of each stereo image with the feature points extracted from the projected content image. And it can be performed at high speed. The three-dimensional coordinates of the stereo corresponding points in the actual three-dimensional space are restored from the parallax amount of each stereo corresponding point, and a plurality of three-dimensional coordinate points corresponding to the plurality of stereo corresponding points constitute a three-dimensional point group of the scene. .

  The target shape designation unit 108 accepts designation of a desired projection target by the user in the scene imaged by the imaging unit 104. In the present embodiment, the projection target is not limited to a plane such as a screen, a flat wall surface, or a whiteboard, but has an arbitrary three-dimensional shape such as a human body, a human hand, a car, or a wall surface with a depression. It is possible to specify an object. In a specific embodiment, the target shape specifying unit 108 receives input of image data from the imaging unit 104 and displays one of the stereo images on the display device. Then, in response to a user operation on an input device such as a mouse or a touch panel, an image area corresponding to a projection object (such as a palm or a wall at the back of a hollow) is designated. Upon receiving the designation of the image area from the user, the target shape designation unit 108 uses the distance measurement data input from the distance measurement unit 106 to extract a three-dimensional point group included in the designated image area, It is recorded as a three-dimensional point group constituting the target three-dimensional shape (hereinafter referred to as a target three-dimensional point group), and target shape data is output.

  Note that the display device and the input device are not particularly limited, and each of the display device and the input device may be included in the projector 100 or may be connected to the projector 100. A personal computer or tablet that communicates with the projector 100. You may utilize what a terminal etc. are equipped with.

  The target shape search unit 110 searches for a target three-dimensional point group that matches a three-dimensional shape of a specified projection target object from a scene at a different time point with respect to a projection target specified based on a scene at a certain time point, Output as 3D shape data. As described above, the three-dimensional shape of the projection target designated by the user is specified using the distance measurement data in the scene at a certain time. On the other hand, the position of the projection target object may change over time. The target shape search unit 110 is a means for dealing with such a change with time of the position of the projection target.

  The plane estimation unit 112 estimates an approximate surface that approximates the projection object in the actual three-dimensional space from the three-dimensional coordinates of the target three-dimensional point group that constitutes the searched three-dimensional shape in which the projection object is designated. And output approximate surface data. In the embodiment to be described, the approximate surface is an approximate plane, but in other embodiments, the approximate surface may be an approximate curved surface so as to correspond to a curved screen such as a cylinder. The plane estimation unit 112 further calculates the centroid of the target three-dimensional point group, and outputs the centroid data to the focus adjustment unit 120. The focus adjustment unit 120 receives the centroid data from the plane estimation unit 112 and adjusts the focus according to the separation distance from the projection device to the centroid of the approximate plane.

  The correction calculation unit 116 receives the distance measurement data from the distance measurement unit 106, the target shape three-dimensional data from the target shape search unit 110, and the approximate plane data from the plane estimation unit 112, and controls the projection light. Control data is calculated and output to the projected image correction unit 118. When projecting a content image from a projection device, trapezoidal distortion may occur in the projection image due to the horizontal and vertical tilt angles of the projection surface of the projection object with reference to a plane perpendicular to the optical axis of the projection device. In the embodiment to be described, as the projection light control, trapezoidal distortion correction (projection image correction) for canceling such trapezoidal distortion will be described as an example. Therefore, the correction calculation unit 116 according to the present embodiment calculates correction data that defines the geometric correction processing to be applied to the content image so as to be within a range corresponding to the projection target on the approximate plane.

  In the embodiment to be described, trapezoidal distortion correction is described as an example, but it can be applied to various types of projection light control. In another embodiment, image correction may be performed so that a content image is naturally seen by an observer when a content image is projected onto an estimated approximate curved surface corresponding to a curved screen such as a cylinder.

  The projection image correction unit 118 performs correction processing on the input or generated content image based on the correction data input from the correction calculation unit 116. The projection unit 122 projects the projection image corrected by the projection image correction unit 118 from the projection device. The position change determination unit 114 compares the approximate plane based on the previous scene with the approximate plane based on the current scene, detects the temporal change of the approximate plane, and requires the correction calculation unit 116 to update the correction calculation. Determine no. For example, when it is detected that the positional relationship has changed more than a certain level, the correction calculation unit 116 can recalculate the correction data.

  The overall flow of the projection image correction control executed by the projector 100 according to the first embodiment will be described below with reference to FIG. FIG. 3 is a flowchart showing the projection image correction control executed by the projector 100 according to the first embodiment. The control shown in FIG. 3 is started from step S100 when the projector 100 is started, for example. In step S101, the projector 100 performs focus adjustment after activation, and in step S102, the calibration process is started in response to a calibration start signal.

  In step S103, the projector 100 projects a content image using the projection unit 122. In step S104, the imaging unit 104 causes a scene including at least a part of the projected image to be projected substantially simultaneously with the reference camera and the reference camera. Capture a stereo image. In step S105, the projector 100 uses the distance measurement unit 106 to perform distance measurement processing, which will be described later with reference to FIGS. 4 and 5, calculates the distance between the stereo corresponding points in the stereo image, and captures the captured scene. The three-dimensional point group which comprises is calculated. In step S106, the projector 100 branches the control depending on whether the flow is for the first calibration or for the second and subsequent calibrations.

  If it is the first calibration in step S106 (YES), the control branches to step S107. In step S107, the projector 100 uses the target shape specifying unit 108 to execute a target shape specifying process to be described later with reference to FIGS. 6 to 8, receives the specification of the projection target from the user, and sets the target three-dimensional shape. get. At the time of the first calibration, the target shape specifying unit 108 functions as a target shape specifying unit in the present embodiment. In step S108, the projector 100 causes the plane estimation unit 112 to perform plane estimation processing to be described later with reference to FIGS. 11 and 13 to calculate the approximate plane and the center of gravity of the projection target from the three-dimensional coordinate point of the target three-dimensional shape. And is recorded for comparison of position changes to be described later. In step S109, the projector 100 causes the focus adjustment unit 120 to adjust the focus to the center of gravity obtained by the plane estimation process.

  In step S110, the projector 100 executes correction calculation processing described later with reference to FIGS. 12 to 17 based on the estimated approximate plane, and calculates correction data for applying geometric correction to the content image. In step S111, the projector 100 causes the projection image correction unit 118 to perform trapezoidal distortion correction on the content image to be projected based on the calculated correction data. In step S112, the projector 100 waits for a predetermined interval time to elapse. In step S113, the projector 100 determines whether the use of the projector 100 is finished. If it is determined in step S113 that it is still in use (NO), the process loops to step S103. Thereby, the projection image projected by the projection unit 122 is appropriately corrected according to the size, position, and tilt angle of the projection target. On the other hand, if it is determined in step S113 that use has ended (YES), projector 100 branches to step S117 and ends this control.

  Referring to step S106 again, in step S106, the flow is for the second and subsequent calibrations, and if it is not the first calibration (NO), control branches to step S114. In step S114, the projector 100 uses the target shape search unit 110 to execute a target shape search process, which will be described later with reference to FIGS. 9 and 10, and at the time of the first calibration based on the distance measurement data after the second calibration. Search for a target 3D point cloud that matches the specified target 3D shape. In the second and subsequent calibrations, the target shape searching unit 110 serves as a target shape specifying unit in the present embodiment. In step S115, the projector 100 causes the plane estimation unit 112 to execute the above-described plane estimation process, and again estimates the approximate plane of the projection target.

  In step S116, the projector 100 compares the previous approximate plane with the newly estimated approximate plane by the position change determination unit 114, and determines whether there is a change greater than a preset threshold value. The change is detected as the amount of movement of the center of gravity of the approximate plane and the amount of change in the tilt angle in the horizontal and vertical directions. The threshold value may be a value set on the vendor side of the projector 100 based on numerical calculation, experimental results, or the like, or the user may select a desired threshold value. If it is determined in step S116 that the position change is less than the threshold (YES), the process directly branches to step S112. In this case, the correction data is not recalculated, and the content image is projected based on the correction data calculated last time in step S103.

  On the other hand, when there is a significant position change and it is determined that the change exceeds the threshold (NO), the control branches to step S109. At this time, the approximate plane calculated again can be recorded for comparison after the next time. In this case, focus adjustment is performed in step S109, correction calculation processing is performed again in step S110, and in step S111, the projection image is corrected based on the updated correction data. As a result, it is possible to appropriately correct following the change in position of the projection object.

  Hereinafter, with reference to FIG. 4 to FIG. 18, the distance measurement process in step S105 described above, the target shape designation process in step S107, the target shape search process in step S114, the plane estimation process in steps S108 and S115, and the correction in step S110 Details of each of the calculation processes will be described.

[Distance processing]
First, the details of the distance measurement process in the present embodiment will be described with reference to FIGS. 4 and 5. FIG. 4 is a flowchart showing the distance measuring process executed by the distance measuring unit 106 in the first embodiment. FIG. 5 is a diagram illustrating a process for searching for stereo corresponding points between stereo images according to the present embodiment.

  As described above, in the present embodiment, the distance measurement unit 106 extracts feature points from each stereo image, and uses the feature points associated with each other as stereo correspondence points. There are various methods of searching for stereo correspondence points between stereo images. In a preferred embodiment, each stereo imaged by capturing at least a part of a feature point extracted from the projected content image and the content image is provided. This can be done through the association with the feature points of the image. FIG. 4 shows a process of calculating the three-dimensional coordinates by obtaining the stereo correspondence points by associating the feature points of the content image and the left and right stereo images.

The process shown in FIG. 4 is started from step S200 in response to the call of step S105 in FIG. The ranging unit 106 acquires stereo image data from the imaging unit 104 in step S201, and acquires content image data to be projected in step S202. Hereinafter, among the stereo image, a right stereo image captured by the reference camera (right camera) referred to by I _R, a left stereo image captured by the reference camera (left camera) referred to by I _L, the content image I referred to in the I _C.

In step S203, the distance measuring unit 106 extracts feature points from each of the content image I _C and the stereo images I _R and I _L. In the preferred embodiment, feature points can be extracted by applying an algorithm called SIFT (Scale-Invariant Feature Transform). SIFT is an algorithm that detects a feature point from an image and describes a feature quantity that is invariant to rotation, scale change, illumination change, etc. of the detected feature point. An outline of the algorithm will be described. In the algorithm, first, an extreme value search is performed in a scale space by DoG (Difference of Gaussian) processing, and the position and scale of candidate feature points are determined. Among the detected feature point candidate points, points that are easily affected by noise or the like, such as points with low contrast and points on the edge, are removed. Then, the orientation of the feature point is obtained so as to describe the feature that is invariant to the rotation, and the feature amount is described based on the obtained orientation.

Here, ^V C ₌ a _{N C-number} of feature point group extracted from the content image ^{_{I C {v i C | i}} = 1, ..., N C} expressed in, _{N R} extracted from right stereo image _{I R} The feature point groups are represented by V ^R = {v _i ^R | i = 1,..., N _R }, and the N _L feature point groups extracted from the left stereo image I _L are represented by V ^L = {v _i ^L | I = 1,... N _L } For each feature point, a feature vector including position coordinates in each image and a predetermined number of feature amount elements is obtained. In the standard SIFT algorithm, in the description of the feature amount, a circular region is defined with the determined scale as the radius around the feature point, this peripheral region is divided into 4 × 4 blocks, and each direction has 8 directions. The slope histogram of is calculated. For this reason, the feature vector is a 128-dimensional vector.

The loop of steps S204~ step S213, each stereo image _I L, the processing for each _{I R} is performed. In the loop from step S205 to step S212, a feature point v that forms a corresponding relationship with each feature point v _i ^C (i = 1,..., N _C ) of the content image I _C for a predetermined stereo image I _R or I _{L. j *} ^R or v _{j *} ^L is determined. Hereinafter, processing for the right stereo image will be described as a representative. In the loop of steps S206~ step S208, the distance measuring unit 106, in step S207, the feature points extracted from a given stereo images ^{_{I R v j R (j =}} 1, ..., N R) for a content image I _The distance d (v _i ^C , v _j ^R ) between the feature points of ^C with the i-th feature point v _i ^C is calculated. Here, assuming that v _{i, k} ^C , v _{j, k} ^R are the k-th elements in the feature quantity vectors of the feature points v _i ^C , v _j ^R , respectively, the distance d between the feature points is expressed by the following equation (1). Can be calculated.

In step S209, the distance measuring unit 106, step S206~ step S208 all distances between the feature points _d calculated in ^{_{(v i C, v j R}} ) (j = 1, ..., N R) from among the following The distance d (v _i ^C , v _{j *} ^R ) between the feature points with the feature point v _{j *} ^R of the stereo image I _R having the minimum distance is calculated by Expression (2). “J *” identifies a feature point that minimizes the distance between feature points by the following equation (2).

In step S210, the distance measuring unit 106, the calculated distances between the feature points ^{_{d (v i C, v j}} * R) is equal to or less than a predetermined threshold value (T). In step S210, if it is determined to be less than the predetermined threshold (YES), in step S211, the feature point _{v j} ^{* R} of the stereo image _{I R} having a smallest and a distance less than the threshold T, the corresponding point pairs < v _i ^C and v _i ^R > are registered.

And each feature point of the content image I _C, right and left stereo images I _R, when the correspondence between the feature points of the respective I _L is both complete, through the point A, the process proceeds to step S214. Figure 5 is passed through the characteristic points of the content image I _C, right and left stereo images I _R, describes the correspondence between feature points of I _L. As shown in FIG. 5, when there are both a feature point v _{j *} ^R and a feature point v _{j *} ^L registered as a corresponding pair for the feature point v _i ^C of the content image, this feature point v _{j *} ^R And the feature point v _{j *} ^L is detected as a stereo corresponding point. In the loop from step S214 to step S218, for the i-th feature point v _i ^C of the content image, an attempt is made to acquire both corresponding point pairs of the left and right stereo images (step S215), and whether or not both are found is determined. When the determination is made (step S216) and both corresponding point pairs are found (YES in step S216), they are registered as k-th stereo corresponding points <v _k ^R , v _k ^L ,>.

The subsequently loop of steps S219~ step S221, the distance measuring section 106, correspondence is balanced the _{N S} stereo correspondence point ^{_{<v k R, v k L}} ,> respect, in the real three-dimensional space based on the parallax A coordinate point p _k ^S is calculated.

Here, the coordinates of the stereo corresponding points on the stereo image of the reference camera are m ^R = (x _R , y _R ), and the coordinates of the stereo corresponding points on the stereo image of the reference camera are m ^L = (x _L , and y _L), representing a three-dimensional point in the real three-dimensional space corresponding to the stereo point p = (X, Y, with Z). In this case, the three-dimensional point p can be calculated by the following equations (3) to (5) using the reference camera as a reference. As described above, since the cameras are arranged in parallel equiposition, y _R = y _L is assumed. In the following formula, x _L -x _R represents the parallax in the horizontal direction, B represents the baseline length corresponding to the distance between the left and right cameras, and f represents the same focal length of the left and right cameras.

In step S222, the distance measuring section 106, _{N S} number of stereo corresponding points ^{_{<v k R, v k L}} > for the coordinate _m ^k R of the characteristic points on the _{image, m} ^{k L} and the corresponding three-dimensional points to p _k ^S (k = 1,..., N _S ) is recorded as distance measurement data as a set, and in step S223, this process is terminated and the process returns to the original process shown in FIG.

  In the above description, as a preferred embodiment, the stereo correspondence points between the stereo images are searched based on the correspondence between the feature points of the content image and the feature points of the stereo images, and the parallax between the images of the corresponding points. Based on the above description, it is assumed that the three-dimensional coordinates are calculated. This preferred embodiment is excellent in terms of high-speed operation and accuracy. However, in another embodiment, a method for searching for stereo correspondence points is not directly performed on the basis of the distance between feature points between the feature points of the stereo image without passing through the feature points of the content image for at least some of the correspondence points. Correlation may be performed. In addition, instead of using a feature vector by SIFT, a pixel correspondence point may be searched by directly comparing and associating pixel blocks constituting a stereo image by template matching or the like.

  Further, in the embodiment to be described, a so-called stereo image method is used as a distance measurement method, in which captured images from two different viewpoints with defined base line lengths are used. However, in other embodiments, captured images from three or more different viewpoints may be used, and an active stereo system in which one captured image in a stereo image system is replaced with a projected image from a projection device such as a projector may be used. Good.

[Target shape specification processing]
Hereinafter, the details of the target shape designation processing in the present embodiment will be described with reference to FIGS. FIG. 6 is a flowchart showing the target shape designation process executed by the target shape designation unit 108 according to the first embodiment. FIG. 7 is a diagram schematically illustrating a three-dimensional point group of a scene calculated by the distance measuring unit 106. FIG. 8 is a diagram for explaining a method for designating a desired three-dimensional shape of a projection target from a three-dimensional point group of a scene in the first embodiment.

The process shown in FIG. 6 is called at step S107 in FIG. 3 and starts from step S300. In step S301, the target shape designation unit 108 reads out the distance measurement data output from the distance measurement unit 106, and from the distance measurement data, the three-dimensional point group P ^S {p _k ^S | k = 1,..., N _S } and For example, the coordinate m _k ^R of the feature point on the right stereo image is acquired.

Figure 7 (A) and (B), illustrate 3D point group P ^S restored based projection images of the first time calibration scene captured, FIG. 7 (A) shows a front view thereof, FIG. 7 (B) shows a top view thereof. In Figure 7, the three-dimensional point group P ^S scene, "○" in accordance with the depth (Z-coordinate), which is conveniently expressed by symbol "△" and "×". “◯” is a point group 154 corresponding to the screen located in the middle, “×” is point groups 150 and 152 corresponding to the background at the back of the screen, and “Δ” is the platform in front of the screen A point group 156 corresponding to the person.

  In step S302, the target shape designating unit 108 causes a display device such as a display to display a user interface (UI) for selecting a shape to be projected. In the embodiment to be described, the UI is configured to display a camera image (for example, an image captured by the right camera) as shown in FIG. 8B and accept designation of an image area 162 for this image. As a method for specifying an image area, as shown in FIG. 8B, a selection rectangle is defined by a mouse operation, or a desired image area is filled by a touch panel operation. In the UI, the feature points may be visualized and superimposed on the camera image so that the user can easily identify the feature points.

Further, the method of specifying the projection object is not limited to the above-described image area specification. In another embodiment, instead of designating the image area 162, as shown in FIG. 8C, designation of the cluster 172A configured to the three-dimensional shape of the projection object from the displayed clusters 172A to 172D is performed. It may be accepted. Said cluster, clustering based on the 3D point group P ^S of the scene to the coordinates, and.

In step S303, it is determined whether or not designation by the user has been received, and step S303 is looped until designation is accepted. In step S304, the target area designating unit 108 calculates a three-dimensional point group P ^O {p _k ^O | k = 1,..., N _O } included in the image area selected on the UI. As described above, the coordinate point m _k ^R on the right camera image is acquired corresponding to the three-dimensional point p _k ^S. Accordingly, as shown in FIG. 8 (A), of the 3D point group P ^S of the entire scene, the coordinate point m _k 3D point group of some ^{of R} are included p _k ^O the designated image region is Identified. In step S305, the target shape specifying unit 108 records the acquired three-dimensional point group P ^O {p _k ^O | k = 1,..., N _O } as a target three-dimensional point group constituting the target three-dimensional shape. In step S306, the process ends and returns to the original process shown in FIG.

[Target shape search processing]
The details of the target shape search process in the present embodiment will be described below with reference to FIGS. 9 and 10. FIG. 9 is a flowchart illustrating target shape search processing executed by the target shape search unit 110 according to the first embodiment. FIG. 10 is a diagram illustrating a process of searching for a target 3D point group corresponding to the target 3D shape of the projection target object from the 3D point group in the scene.

  As shown in FIG. 10, the target shape search unit 110 searches for a target three-dimensional point group that matches the target three-dimensional shape specified at the first calibration in the second and subsequent calibration scenes. In the preferred embodiment, the search for the target three-dimensional point group can be performed by least square matching using the specified target three-dimensional shape as a template. As a numerical solution method of least square matching, Gauss-Newton method, steepest descent method, Levenberg-Markert method, and the like can be given. In the embodiment described below, the Levenberg-Marquardt method, which is an effective solution for the nonlinear least squares problem, will be described as an example.

The process shown in FIG. 9 is called at step S114 in FIG. 3 and starts from step S400. In step S401, the target shape search unit 110 acquires distance measurement data output from the distance measurement unit 106, and in step S402, the three-dimensional point group P ^S {p _k ^S | k = 1 of the scene from the distance measurement data. ..., acquires the _{N S},} in step S403, 3-D point cloud object shape _{^{P O {p k O | k}} = 1, ..., acquires _{N O}.}

In step S404, object shape searching unit 110, the 3D point group P ^S of the acquired scene, with estimates the scene approximated surface that approximates the three-dimensional shape of a scene, the three-dimensional point cloud object acquired projection target P A target shape approximate curved surface that approximates the target shape is estimated from ^O. Method of estimating the scene approximated surface and object shape approximated surface include, but are not limited to, three-dimensional point group P ^S and 3D point group P ^O each coordinate point of the sample points, respectively, by the least square method It can be calculated by curved surface fitting. Note that the method of expressing the approximate curved surface is arbitrary, and an arbitrary expression format such as a polynomial curved surface, a Pezier curved surface, or a spline curved surface can be adopted. Here, the scene approximate curved surface is represented by Z = I (X, Y), and the target shape approximate curved surface is represented by Z = T (X, Y). Since there is an error in the three-dimensional coordinates obtained by the distance measurement processing, it is possible to perform good matching by estimating the approximate curved surface in this way.

  In steps S405 to S408, the target shape approximated curved surface Z = T (X, Y) is collated with a partial region in the scene approximated curved surface Z = I (X, Y) by iterative calculation based on the Levenberg-Markert method. The parameter improvement is repeated in the direction in which the error decreases, and a final solution is obtained. The Levenberg-Markert method can be expressed by the following formula (6).

In the equation, χ = {χ ₁ ,..., Χ _n } represents a set of unknown parameters to be obtained, and the target approximate curved surface that is a template includes the geometrical deformation such as translation, deformation, and rotation. This is a parameter of the function f associated with the partial area. In the above equation (6), the matrix J (χ) is a Jacobian matrix, is represented by the following equation (7), and is composed of a row vector representing the gradient with respect to the predetermined parameter χ of the function f. In the above equation, e (χ) is a residual between the target approximate curved surface T (X, Y) and the partial region f (I (X, Y), χ) associated with the scene approximated curved surface with a predetermined parameter χ. The smaller the sum of squares, the higher the degree of matching, that is, the evaluation function to be minimized. λ is an attenuation parameter and can be changed for each iteration. I is a unit matrix. Δχ represents the correction amount of the parameter χ in the iterative process.

In step S405, the target shape search unit 110 first sets an initial value χ ⁰ of an unknown parameter. In step S406, the target shape search unit 110 uses a target approximate curved surface as a template for a given parameter χ ^t (where t represents the number of iterations and is zero for the first time), and a portion of the scene approximate curved surface. A parameter correction amount Δχ is calculated so as to reduce the error by collating with the region. In the Levenberg-Markert method, when the Jacobian matrix J (χ) of the equation (7) is calculated, the parameter correction amount Δχ can be obtained from the equation (6). The correction amount Δχ indicates an improvement direction for reducing the error e (χ) in the parameter space. In step S407, the unknown parameter χ ^t is improved by Δχ to obtain an updated unknown parameter χ ^{t + 1} . In step S408, it is determined whether or not a predetermined convergence condition is satisfied by the repeated calculation so far. The convergence condition is, for example, that the error e (χ) is less than or equal to a predetermined threshold, the change amount of the error e (χ) is less than or equal to the predetermined threshold, or the parameter correction amount Δχ is less than or equal to the threshold. It can be a condition.

If it is determined in step S408 that the convergence condition is not satisfied (NO), the process loops to step S406 and the iterative improvement process is repeated. If it is determined in step S408 that the convergence condition is satisfied (YES), the process proceeds to step S409. As shown in FIG. 10, matching with a partial area in the scene is performed by iterative calculation, and a partial area corresponding to the target shape approximate curved surface in the scene approximated curved surface is specified from the finally obtained parameter solution χ ^t Is done. At step S409, the object shape searching unit 110, based on the obtained parameter chi ^t, the out of the three-dimensional point group ^{P S} in the scene, the three-dimensional point group object corresponding to the target shape ^{_P S} _{'{p k} ^S ′ | k = 1,..., N _{S ′} } is extracted, and in step S410, the present process is terminated and the process returns to the original process shown in FIG.

  In the above-described embodiment, the target three-dimensional shape (target three-dimensional point group) of the projection target object used as the template has been described as being fixedly used in the first calibration. However, in another embodiment, the target shape search unit 110 can also use a target three-dimensional shape specified by matching at a certain time as a template for subsequent calibration. In this embodiment, since the shape as a template is sequentially updated to the latest one, for example, it is possible to satisfactorily follow a projection object whose position and shape change continuously, such as a curtain.

[Plane estimation processing]
Hereinafter, the details of the plane estimation process in the present embodiment will be described with reference to FIGS. 11 and 13. FIG. 11 is a flowchart illustrating the plane estimation process executed by the plane estimation unit 112 according to the first embodiment. FIG. 13 is a diagram for explaining the relationship between the approximate plane that approximates the target three-dimensional shape of the projection target and the normal camera that is positioned in the normal direction of the approximate plane.

The process shown in FIG. 11 is called at step S108 and step S115 in FIG. 3, and starts from step S500. In step S501, the plane estimation unit 112 uses the target 3D point group P ^S ′ {p _k ^S ′ | k = corresponding to the projection target in the scene from the target shape 3D data output by the target shape search unit 110. 1,..., NS _′ } are acquired. In step S502, the plane estimation unit 112 calculates a regression plane Z = R (X, Y) from the acquired three-dimensional point group PS ^{′ of} the target shape. The regression plane is represented by the following equation (8), and the coefficients a, b, and c can be obtained by the least square method using a numerical set of coordinate values of the three-dimensional point group PS ^′ . In addition, since the least square method using an approximation function like the following formula (8) is well known as a solution method using a normal equation, a detailed description is omitted here.

Although the regression plane calculated in step S502 may be used as an approximate plane to be projected as it is, in the embodiment to be described, the three-dimensional point p _i ^{S ′} greatly deviating from the regression plane is calculated in steps S503 to S506. Improve regression plane while excluding from. In step S503, the plane estimation unit 112 calculates the distance of the three-dimensional point p _k ^S ′ from the regression plane and extracts the most distant three-dimensional point p _i ^{S ′} . In step S504, the plane estimation unit 112 determines whether or not the distance is equal to or smaller than the threshold value. If the distance exceeds the threshold value (NO), the farthest three-dimensional point is excluded, and the process loops to step S502. And calculate the regression plane again.

On the other hand, if it is determined in step S504 that the distance is equal to or smaller than the threshold (NO), the process branches to step S506. In step S506, the plane estimation unit 112 assumes that the current regression plane is an approximate plane that approximates the three-dimensional shape corresponding to the projection target, the coefficients a, b, and c of the above equation (8), and the target three-dimensional shape. The center of gravity G is recorded, and in step S507, the process is terminated and the process returns to the original process shown in FIG. FIG. 13 shows the relationship between the target three-dimensional point group PS ^′ extracted as the target three-dimensional shape, the approximate plane Z = R (X, Y), and the center of gravity G.

[Correction calculation processing]
Hereinafter, the details of the correction calculation processing in the present embodiment will be described with reference to FIGS. FIG. 12 is a flowchart showing a correction calculation process executed by the correction calculation unit 116 according to the first embodiment. FIG. 13 is a diagram for explaining a virtual normal camera that is as described above and is positioned in the normal direction of the approximate plane.

The process shown in FIG. 12 is called in step S110 shown in FIG. 3, and starts from step S600. In step S601, the correction calculation unit 116 reads the target shape three-dimensional data output from the target shape search unit 110, and the target three-dimensional point group P ^S '{p _k ^S ' | k of the projection target in the scene. = 1,..., NS _′ }. In step S602, the correction calculation unit 116 reads the approximate plane data output from the plane estimation unit 112, and acquires the coefficients a, b, and c of the approximate plane Z = R (X, Y).

In step S603, the correction calculation unit 116 calculates a perspective projection transformation matrix B that converts a three-dimensional point on the real three-dimensional space into a two-dimensional point on the image. The perspective projection transformation matrix B converts a three-dimensional point in a real three-dimensional space into a point on a normal camera image viewed from a virtual normal camera 130 positioned in the normal direction with respect to the approximate plane. It is. As shown in FIG. 13, the normal camera 130 moves from the position of the center of gravity G of the target three-dimensional point group P ^S ′ in the estimated approximate plane Z = R (X, Y) in the normal direction of the approximate plane. This means a virtually placed camera. The normal camera image is virtual image data when the image is virtually captured from the position of the normal camera. Normal camera images have no color information and are used in the process of geometric transformation of points. Hereinafter, the normal camera image is referred to as _IV .

  The perspective projection transformation matrix B is given by the following equation (9). In the following formula, the matrix A is an internal parameter matrix (3 × 3) representing the optical axis coordinates of the camera, the focal length, and the distortion of the lens. The matrix R is a rotation matrix (3 × 3) representing the rotation from the reference camera (right camera) 104 to the normal camera 130. The vector t is a translation vector (3 × 1) from the reference camera (right camera) 104 to the normal camera 130.

In step S604, the correction calculation unit 116 uses the calculated perspective projection transformation matrix B to convert the target three-dimensional point group PS ^′ in the scene into the two-dimensional point group Q ^{S ′} {on the normal camera image _IV. q _k ^S ′ | k = 1,..., N _{S ′} }. FIG. 14 is a diagram illustrating a process of converting a three-dimensional point group into a two-dimensional point group on a normal camera image by perspective projection conversion. As shown in FIG. 14, the target three-dimensional point group PS ^′ constituting the three-dimensional shape of the projection object 180 is projected onto the two-dimensional point group Q ^{S ′} on the normal camera image _IV , and the two-dimensional points are projected. The group QS ^′ represents the shape 182 of the projection object on the normal camera image.

In step S605, the correction calculation unit 116 calculates a convex hull polygon 184 that wraps the two-dimensional point group Q ^{S ′} on the obtained normal camera image _IV . FIG. 15 is a diagram for explaining processing for obtaining a correction rectangle from a two-dimensional point group representing the shape of the projection target on the normal camera image. This convex hull polygon 184 defines an area considered as a screen on the normal camera image _IV . In step S606, the correction calculation unit 116 calculates the maximum corrected graphic that is included in the obtained convex hull polygon 184 and has the same aspect ratio as the content image. The correction figure is a correction rectangle in the embodiment to be described, and the correction rectangle 186 defines an area on the normal camera image _IV on which the content image is projected.

  In the embodiment to be described, priority is given to the fact that the projection image does not protrude from the projection target, and the correction figure is a correction rectangle included in the convex hull polygon 184. However, a corrected figure having an arbitrary relationship can be obtained from the convex hull polygon 184. In another embodiment, the correction calculation unit 116 may change the inclusion relationship, calculate the minimum correction rectangle that includes the convex hull polygon 184 and has the same aspect ratio as the content image.

In step S607, the correction calculation unit 116 calculates the two-dimensional point group Q ^{S ″} {q _k ^S included in the correction rectangle (projection object 186) obtained from all the two-dimensional point groups Q ^{S ′.} ”| K = 1,..., NS _” } is selected. Figure 16 is a diagram for explaining a two-dimensional point group on normal camera image I _V, a correspondence relationship between the feature points on the projector image I _P to be projected. Similar to the processing described in the section [Ranging processing], feature points can be extracted from the projector image. Then, using the feature quantity vector associated with the two-dimensional point q _k ^{S ″} (which can be associated with the stereo correspondence point) and the feature point vector of the feature point extracted from the projector image, FIG. Corresponding relationships as shown can be obtained. In step S608, the correction calculation unit 116 determines from the projector image the feature point group q _k ^P (k on the projector image corresponding to the two-dimensional point group q _k ^S ″ {k = 1,..., N _{S ″} }. = 1,..., N _{S ″} ).

In step S609, the correction calculation unit 116 includes the two-dimensional point group Q ^{S ″} {q _k ^S ″ | k = 1,..., N _{S ″} } included in the correction rectangle and the corresponding feature on the projector image. From a point on the normal camera image to a point on the projector image, using four or more corresponding point pairs of the point group Q ^P {q _k ^P | k = 1,..., N _{S ″} }. And a projective transformation matrix _HVP to be converted is calculated. The projection transformation matrix H _VP is expressed by the following formula (10), where m _V = (x _V , y _V ) on the normal camera image and m _P = (x _P , y _P ) on the projector image. It is expressed using In the following formula, h _{1 to} h ₈ are unknown parameters.

Then, by dividing by the scale factor, the following equation (11) is obtained, and if the unknown parameters h _{1 to} h ₈ in the equation (10) are obtained, an arbitrary coordinate point (x _V on the normal camera image) is obtained. , Y _V ), the corresponding coordinate point (x _P , y _P ) on the projector image is obtained.

Per corresponding point pairs one in the normal camera image I _V and projector image I _P, since one constraint equation in the least square method is obtained, if there are 4 or more corresponding point pairs, by applying the least square method The eight unknown parameters h _{1 to} h ₈ are obtained.

FIG. 17 is a diagram for explaining geometric transformation by the projective transformation matrix _HVP and the projective transformation matrix _HPP . In step S610, the correction calculation unit 116 converts the four corners of the correction rectangle 186 obtained above using the obtained projective transformation matrix H _VP , so that the four corners of the correction rectangle 186 on the projector image are obtained. The four corners of the correction trapezoid 188 corresponding to are calculated. In step S611, the correction calculation unit 116, further, applied with the four corners of the correction trapezoid 188 on the projector image I _P, using a 4-point between the corners of the content image I _C, the least square method in the same manner as the above-described process Then, a projective transformation matrix _HPP for converting from a point on the content image to a point on the projector image is calculated. The projective transformation matrix _HPP is correction data that cancels out the average trapezoidal distortion and defines the geometric transformation so that the content image fits within the corrected figure converted on the projected image.

[Projection correction processing]
FIG. 18 is a diagram for explaining (A) trapezoidal correction processing of a content image by the projective transformation matrix _HPP , and (B) corrected projected image actually projected. Based on the correction data (projection transformation matrix H _PP ) input from the correction calculation unit 116, the projection image correction unit 118 applies a geometric transformation to the input or generated content image and performs a trapezoidal distortion correction process. Thus, a projector image I _P ′ that is actually projected from the projection device is generated. The projector image I _P ′ corrected by the projective transformation matrix H _PP is projected so as to be within the shape 180 of the arbitrary-shaped projection target specified by the user, and an appropriate content projection image 190 is obtained.

[Operational effects of the first embodiment]
According to the first embodiment described above, it is possible to perform projection image correction control in accordance with a projection target with respect to the projection target having an arbitrary shape designated by the user. Further, the above control is repeatedly performed every predetermined time, and the change in the positional relationship of the projection target is recognized, and the correction data is recalculated. Therefore, even when a change occurs in the positional relationship between the projector 100 and the projection object while using the projector 100, the change in position is detected in substantially real time without interrupting the use of the projector 100, and a projection target having an arbitrary shape is obtained. It becomes possible to continue recognizing an object and project the corrected projection light onto the object. Further, since it is possible to extract stereo-corresponding points and measure three-dimensional coordinates using the content image used by the user in the presentation or the like, even when correction is necessary during use, the projector 100 The use of is not hindered.

  19 and 20 exemplify preferred usage modes of the projector 100 according to the first embodiment. FIG. 19 illustrates a usage mode in which the projection image correction is performed following the rotation of the projection object 180. As shown in FIG. 19A, at a certain point in time, the projection image 190OLD has been subjected to projection image correction so as to be within the projection plane defined by the projection object 180 designated by the user. However, when the projection object rotates as shown in FIG. 19B, a part of the projection image 190a, 190b protrudes from the projection object.

  At this time, the feature point of the protruding part is greatly separated from the depth of the projection target. The projector 100 detects a change in the position and shape of the target three-dimensional shape of the projection target, and recalculates correction data. As a result, as shown in FIG. 19C, the correction rectangle is recalculated to be smaller by the rotation of the designated projection object, and the projection image 190 NEW is accommodated in the tilted projection object by the newly calculated correction data. Correction is applied. As shown in FIG. 19, according to the above configuration, a change in the position of the projection object is detected in substantially real time, the projection object having an arbitrary shape is continuously recognized, and projection light appropriately corrected is projected onto the object. be able to.

  FIG. 20 is a diagram illustrating a usage mode in which a wall surface having a concave shape is used as a projection target. In the prior art, a wide plane in the captured scene is recognized as a projection target. Therefore, when a projection is made on a wall having a special shape as shown in FIG. 20, the front part of the wall is recognized as the projection target. It will be.

  On the other hand, in the projector 100 according to the first embodiment, in the UI, the image region 162 corresponding to the recessed wall surface can be specified as the projection target. When the back wall is designated as the projection target, projector 100 calculates correction data so that projected image 190 NEW fits in the recessed back wall. Therefore, it can be said that the projector 100 according to the first embodiment can be suitably applied to an object having an arbitrary shape such as a complicated shape such as a person or a hand or a wall having a recessed central portion.

  Hereinafter, an embodiment of a modification of the above-described first embodiment will be described.

(Second Embodiment)
In the first embodiment, the target three-dimensional shape to be projected is specified by the user specifying an image region or cluster on the camera image on the UI screen. However, the method for designating the projection object is not limited to the above-described embodiment. In the second embodiment, a projection target can be specified by inputting a template image. Note that the configuration of the projector 100 according to the second embodiment is the same as the configuration of the first embodiment except for the configuration of the target shape designating unit 108, and therefore, the differences will be described below.

In the second embodiment, object shape designation unit 108, as shown in FIG. 21, the input image as a template image I _T, for example, by performing template matching between the partial images on the right camera image I _R Thus, an image region corresponding to the projection target is specified. When the image area is identified, as in the first embodiment, among the three runner-up group P ^S, 3-dimensional point group included in the matching image area in the input template image I _T is the projection object Extracted as a target three-dimensional point group constituting a three-dimensional shape.

  According to the second embodiment, even if the user does not specify an image area corresponding to the projection target on the UI, the projection target can be specified simply by inputting the image of the projection target. Is possible. For example, when a person is to be projected, the projector 100 can be made to recognize the person as a projection target by inputting an image of the same person that has been captured separately. In addition, by registering various projection objects in advance, it is possible to specify the projection object simply by selecting the desired projection object. As a template image input method, the image may be given to the projector 100 as a file, or may be imaged by providing a template imaging mode using the imaging unit 104 provided in the projector 100.

(Third embodiment)
In the first embodiment, the projection image correction function for correcting the trapezoidal distortion on the content image has been described as an example. However, the projection light control is not limited to the geometric correction function for the content image. The third embodiment has a function of controlling the projection light to be projected according to the shape of the projection target. Note that the configuration of the projector 100 according to the third embodiment is the same as the configuration of the first embodiment except for the correction calculation unit 116 and the projection image correction unit 118, and differences will be described below.

In the first embodiment, in the correction calculation process, a correction rectangle included in a convex hull figure is used as a projection plane, and a content image is projected on the entire projection plane. As shown in FIG. 22A, the correction calculation unit 116 according to the third embodiment calculates a three-dimensional convex hull figure 184 corresponding to a speaker, for example, and has an arbitrary size including the convex hull figure 184. Calculate the correction rectangle. The correction calculation unit 116 calculates a projection transformation matrix H _PP for converting the four corners of the content image to the correction rectangle on the normal line camera image, content image to fit on the converted corrected in a rectangle on the projection image Correction data defining the geometric transformation is obtained. As shown in FIG. 22, the projection image correcting unit 118 calculates a region 192 corresponding to the convex hull figure on the normal camera image, and masks a portion other than the calculated region 192 for the white plane image, for example. Thus, a content image as shown in FIG. 22B is generated. That is, a projection image is generated in which mask processing is performed so that light is not projected onto a region corresponding to the outside of the convex hull figure.

By applying the projective transformation matrix _HPP to the mask image as shown in FIG. 22B, the projection light is projected so as to fit in the shape of the speaker specified by the user. According to the third embodiment, the projection light can be made to follow the designated projection object, and can be used as the spotlight 194 that follows the projection object.

(Fourth embodiment)
In the first embodiment, the projector 100 including the stereo image type distance measuring sensor has been described as an example of the image processing apparatus and the video projection system. However, it is not always necessary that the functional units shown in FIG. 1 are provided in a single device, and some arithmetic processing may be performed by an external computer system.

  FIG. 23 is a diagram illustrating a schematic configuration of a video projection system 200 according to the fourth embodiment. A video projection system 200 shown in FIG. 23 includes a projector 210 having a stereo image type distance measuring sensor and an external computer system 230. The projector 210 includes a network communication unit 220 such as a NIC (network interface card) in addition to the control unit 212, the focus adjustment unit 216, the imaging unit 214, and the projection unit 218. On the other hand, the external computer system 230 includes a network communication unit 232 for communicating with the projector 210, a distance measuring unit 234, a target shape specifying unit 236, a target shape searching unit 238, a plane estimating unit 240, and a position change. A determination unit 242, a correction calculation unit 244, and a projection image correction unit 246 are included. In this case, stereo image data can be transmitted to the external computer system 230 by the network communication unit 220, and centroid data and corrected projector image data can be received from the external computer system 230 as a calculation result.

  As shown in FIG. 23, by implementing the main calculation function on the external computer system 230 side, the requirements for the calculation performance required for the projector 210 can be relaxed. The external computer system 230 may be a single computer device, a virtual computer, or a computer system that includes a plurality of computers and performs distributed calculations. Further, the mode of sharing the functional units between the projector 210 and the external computer system 230 is not limited to the example shown in FIG. 23, and can be performed in any mode.

(Fifth embodiment)
In the first embodiment and the fourth embodiment, the projectors 100 and 210 including the stereo image type distance measuring sensor have been described. However, the distance measuring sensor is not necessarily provided in the projector.

  FIG. 24 is a diagram illustrating a schematic configuration of a video projection system 300 according to the fifth embodiment. A video projection system 300 shown in FIG. 24 includes a projector 310 and a stereo distance measuring sensor 340 connected externally. The projector 310 includes a control unit 312, a distance measurement unit 314, a target shape designation unit 316, a target shape search unit 318, a plane estimation unit 320, a position change determination unit 322, a correction calculation unit 324, a projection image correction unit 326, and a focus adjustment unit. In addition to the projection unit 330, an interface unit 322 such as a USB (Universal Serial Bus) is provided. On the other hand, the stereo distance measuring sensor 340 includes an interface unit 342 and an imaging unit 344.

  Even if the focal length of the projector 310 and the focal length of the stereo distance measuring sensor 340 are largely dissociated by providing the projection unit 330 and the imaging unit 344 in the devices 310 and 340 as shown in FIG. The above-described projection light control can be suitably performed. FIG. 25 illustrates an imaging system in which the short focus projector 310 and the stereo distance measuring sensor 340 are separated.

  As described above, according to the embodiment of the present invention, an image processing device that can perform projection light control in accordance with a projection target on a projection target of an arbitrary shape specified by a user, An image projection system and a program can be provided.

  The functional unit can be realized by a computer-executable program written in a legacy programming language such as assembler, C, C ++, C #, Java (registered trademark), an object-oriented programming language, or the like. ROM, EEPROM, EPROM , Stored in a device-readable recording medium such as a flash memory, a flexible disk, a CD-ROM, a CD-RW, a DVD-ROM, a DVD-RAM, a DVD-RW, a Blu-ray disc, an SD card, an MO, or through an electric communication line Can be distributed. In addition, a part or all of the functional unit can be mounted on a programmable device (PD) such as a field programmable gate array (FPGA) or mounted as an ASIC (application-specific integration). Circuit configuration data (bit stream data) downloaded to the PD in order to implement the above functional unit on the PD, HDL (Hardware Description Language) for generating the circuit configuration data, VHDL (VHSIC (Very High Speed) Integrated Circuits) Hardware Description Language)), data described in Verilog-HDL, etc. can be distributed by a recording medium.

  Although the embodiments of the present invention have been described so far, the embodiments of the present invention are not limited to the above-described embodiments, and those skilled in the art may conceive other embodiments, additions, modifications, deletions, and the like. It can be changed within the range that can be done, and any embodiment is included in the scope of the present invention as long as the effects of the present invention are exhibited.

DESCRIPTION OF SYMBOLS 100 ... Projector, 102 ... Control part, 104 ... Imaging part, 106 ... Distance measuring part, 108 ... Object shape designation | designated part, 110 ... Object shape search part, 112 ... Plane estimation part, 114 ... Position change determination part, 116 ... Correction Calculation unit 118 ... Projection image correction unit 120 ... Focus adjustment unit 122 ... Projection unit 150 ... Point group 152 ... Point group 154 ... Point group 156 ... Point group 160 ... UI screen 162 ... Image area 170 ... UI screen 170, 172 ... cluster, 180 ... projection object, 180 ... projection object, 184 ... convex hull figure, 186 ... correction rectangle, 190 ... projection image, 192 ... area corresponding to convex hull figure, 194 ... Spotlight, 200 ... Video projection system, 210 ... Projector, 212 ... Control part, 214 ... Imaging part, 216 ... Focus adjustment part, 218 ... Projection part, 220 ... Net Work communication unit 230 ... External computer system, 232 ... Network communication unit, 234 ... Distance measuring unit, 236 ... Target shape specifying unit, 238 ... Target shape searching unit, 240 ... Plane estimation unit, 242 ... Position change determination unit, 244 ... Correction calculation unit, 246 ... Projection image correction unit, 300 ... Video projection system, 310 ... Short focus projector, 312 ... Control unit, 314 ... Distance measurement unit, 316 ... Target shape designation unit, 318 ... Target shape search unit, 320 ... Plane estimation unit, 322 ... Position change determination unit, 324 ... Correction calculation unit, 326 ... Projection image correction unit, 328 ... Focus adjustment unit, 330 ... Stereo distance measuring sensor, 330 ... Projection unit, 332 ... Network communication unit, 342 ... Network communication unit, 344 ... Imaging unit

Japanese Patent No. 3880582

Claims (10)

A distance measuring means for measuring a distance between a plurality of points in the image and calculating a three-dimensional point group;
A target shape designation means for receiving designation of a three-dimensional shape as a projection target;
A target shape specifying means for specifying a target three-dimensional point group corresponding to the three-dimensional shape of the projection target from the three-dimensional point group;
Surface estimation means for estimating an approximate surface of the projection target from the target three-dimensional point group;
An image processing apparatus comprising: control data calculation means for calculating control data defining projection light projected onto the projection area on the approximate plane. The target shape specifying means includes
Means for estimating a target shape approximation curved surface that approximates the three-dimensional shape of the projection target from a three-dimensional point group constituting the three-dimensional shape of the projection target;
Means for estimating a scene approximation curved surface that approximates a 3D shape of the scene from a 3D point cloud of the scene;
Means for collating the target shape approximate curved surface with the portion of the scene approximate curved surface and estimating the position of the corresponding portion of the scene where the evaluation function for evaluating the degree of matching is maximized;
The image processing apparatus according to claim 1, further comprising: a unit that identifies a three-dimensional point group associated with the corresponding part of the scene as the target three-dimensional point group based on the position of the corresponding part. The target shape specifying means includes
Designation of an image region including a three-dimensional point group constituting the three-dimensional shape of the projection target;
Designation of a cluster constituting a three-dimensional shape of the projection target from one or more clusters configured by classifying a three-dimensional point group of a scene, or
3. The image processing apparatus according to claim 1, wherein a three-dimensional point group included in an image region matched with a template image receives designation of the template image as constituting the three-dimensional shape of the projection target. The distance measuring means includes
Means for extracting a plurality of feature points from the projected content image;
Means for extracting a plurality of feature points from a plurality of captured images each including at least a part of the content image;
Means for searching for a plurality of feature points to be stereo correspondence points between the captured images based on the association between the feature points of the content image and the feature points of each of the plurality of captured images;
The image processing apparatus according to claim 1, further comprising: a unit that calculates a three-dimensional position of each of the stereo correspondence points based on a parallax between the images of each of the stereo correspondence points.   The target shape specifying means calculates a matching with the scene at a second time point using the target three-dimensional point group at the first time point specified as constituting the three-dimensional shape of the projection target as a template. To specify a target 3D point group constituting the three-dimensional shape of the projection target at the second time point, and to use the target 3D point group at the second time point as a next template. The image processing apparatus according to any one of claims 1 to 4. The first approximate surface based on the scene at the first time point is compared with the second approximate surface based on the scene at the second time point, and the second approximate surface is deviated from the first approximate surface. Means for calculating the quantity;
And a means for determining whether or not the deviation amount exceeds a predetermined threshold value,
The image processing according to any one of claims 1 to 5, wherein the control data calculation means calculates updated control data when the deviation amount exceeds the predetermined threshold. apparatus. The control data calculation means includes
Means for converting the target three-dimensional point group into a two-dimensional point group on a normal camera image viewed from a virtual normal camera located in a normal direction with respect to the approximate plane;
Means for calculating a correction figure having an inclusion relation with a convex hull figure that convexly envelops the two-dimensional point group;
The configuration of the correction figure on the normal camera image based on the two-dimensional point group included in the correction figure on the normal camera image and the corresponding point group on the projection image corresponding to the two-dimensional point group Means for converting the points onto the projected image;
7. The image processing apparatus according to claim 1, further comprising: calculating correction data for converting a content image so as to be within the correction figure converted on the projection image as the control data. .   The image processing according to claim 7, wherein the image processing apparatus further includes a projection image generation unit that generates a projection image on which mask processing is performed so that light is not projected onto a region corresponding to the outside of the convex hull figure. apparatus. Projection means for projecting content images;
Imaging means for imaging at least a part of the projected content image;
A distance measuring means for measuring a distance between a plurality of points in a captured image and calculating a three-dimensional point group;
Target shape designation means for receiving designation of a three-dimensional shape of a projection target as a projection target;
A target shape specifying means for specifying a target three-dimensional point group corresponding to the three-dimensional shape of the projection target from the three-dimensional point group;
Surface estimation means for estimating an approximate surface of a projection target from the target three-dimensional point group;
A video projection system, comprising: control data calculation means for calculating control data defining projection light projected onto a projection area on the approximate plane; and projection image generation means for generating a projection image using the control data. Computer
A distance measuring means for measuring a distance between a plurality of points in an image and calculating a three-dimensional point group;
Target shape designation means for receiving designation of a three-dimensional shape of a projection target to be projected
Target shape specifying means for specifying a target three-dimensional point group corresponding to the three-dimensional shape of the projection target from the three-dimensional point group;
Program for functioning as surface estimation means for estimating an approximate surface of a projection target from the target three-dimensional point group, and control data calculation means for calculating control data defining projection light projected onto a projection area on the approximate surface .