WO2018167918A1

WO2018167918A1 - Projector, method of creating data for mapping, program, and projection mapping system

Info

Publication number: WO2018167918A1
Application number: PCT/JP2017/010700
Authority: WO
Inventors: 青柳　寿和
Original assignee: Ｎｅｃディスプレイソリューションズ株式会社
Priority date: 2017-03-16
Filing date: 2017-03-16
Publication date: 2018-09-20
Also published as: JPWO2018167918A1; JP6990694B2

Abstract

Data for mapping is generated with which it is possible to create a projection mapping video that matches the external form of each surface of a solid object. A projector having a mapping data generation unit (403) for generating data for mapping in response to a data creation request. The mapping data generation unit has: a three-dimensional sensor (405) for measuring a solid object in three dimensions and outputting three-dimensional position data that represents, by three-dimensional coordinates, the position and shape of each surface of the solid object to which surface an image is projected; a coordinate conversion unit (406) for converting the coordinate system of the three-dimensional position data into a projector coordinate system in which an image projection area is represented by three-dimensional coordinates, with a point where main beams from pixels located at the opposite angles of an image formation surface intersect set as the origin; and a data generation unit (407) for acquiring the position of each surface of the solid object and a ridgeline indicating the external form of the each surface and generating the data for mapping.

Description

Projector, mapping data creation method, program, and projection mapping system

The present invention relates to a projector, a mapping data creation method, a program, and a projection mapping system.

Recently, projection mapping, which uses a projector to project an image on a three-dimensional object such as a building and express the image, has attracted attention. In this projection mapping, it is necessary to create an image (projection mapping image) for projection from a projector so as to match the outer shape of the surface of the three-dimensional object.
FIG. 1A shows an example of a system that performs projection mapping. This system includes a projector 101 and a video processing apparatus 105 including a personal computer. The projector 101 and the video processing device 105 are connected via a video signal cable.
The video processing device 105 creates a projection mapping video 106 and supplies the video signal to the projector 101. The projector 101 projects an image based on the image signal from the image processing device 105 toward the three-dimensional object 104 in the projection area 103. The projection mapping image 106 is displayed on a projection surface including the five surfaces 104 a to 104 e of the three-dimensional object 104.

FIG. 1B shows an example of the projection mapping video 106.
The rectangular original images 141a to 141e are images to be displayed on the surfaces 104a to 104e of the three-dimensional object 104 shown in FIG. 1A. The images 142a to 142e are images obtained by deforming the rectangular original images 141a to 141e according to the outer shapes of the surfaces 104a to 104e, respectively. The projection mapping image 106 is obtained by assigning these images 142a to 142e so as to correspond to the surfaces 104a to 104e, respectively.
The method for creating the projection mapping video 106 includes a method for creating a projection mapping video in a two-dimensional space and a method for creating a projection mapping video in a three-dimensional space.

FIG. 2 schematically shows a method for creating a projection mapping video in a two-dimensional space. On the video processing device 105, a frame indicating the outer shape of each surface of the three-dimensional object in the projection video is created. In FIG. 2, a frame 146 indicated by a solid line is a frame indicating the outer shape of the surface 104 c of the three-dimensional object illustrated in FIG. 1A. The image of the frame formed on the video processing device 105 is actually projected from the projector 101. The frame is deformed on the video processing device 105 so that the projected image of the frame matches the outer shape of the corresponding surface of the three-dimensional object 104.
In the example illustrated in FIG. 2, the frame 147 projected onto the three-dimensional object 104 corresponds to the frame 146 created on the video processing device 105, but the frame 147 matches the outer shape of the corresponding surface of the three-dimensional object 104. Absent. Therefore, the frame 146 is deformed on the video processing device 105 so that the frame 147 matches the outer shape of the corresponding surface of the three-dimensional object 104.

FIG. 3 schematically shows a method for creating a projection mapping video in a three-dimensional space. A virtual three-dimensional space is formed on the video processing apparatus 105, and a virtual three-dimensional object 104-1 and a virtual camera 148 corresponding to the three-dimensional object 104 are arranged in the three-dimensional space. A video is assigned to each surface of the virtual three-dimensional object 104-1 and reproduced, and a video reproduced on each surface of the virtual three-dimensional object 104-1 is captured using the virtual camera 148. An image captured by the virtual camera 148 is used as the projection mapping image 106.
In general, creating a video using a virtual camera 148 is called rendering. The angle of view of the virtual camera 148 is set to be equal to the angle of view of the projector 101. In an actual three-dimensional space, the projector 101 is placed at the position where the virtual camera 148 was placed, and the projector 101 projects the projection mapping image 106 created using the virtual camera 148 toward the three-dimensional object 104. .
Techniques related to projection mapping are described in Patent Documents 1 and 2.

Utility Model Registration No. 199779 JP 2012-215633 A

In the projection mapping video creation method in the two-dimensional space shown in FIG. 2, the frame 146 is displayed on the video processing device 105 so that the projected frame 147 matches the outer shape of the corresponding surface of the three-dimensional object 104. It is necessary to carry out adjustment work such as deforming. Such adjustment work is very troublesome and troublesome for the operator.

In the projection mapping video creation method in the three-dimensional space shown in FIG. 3, it is necessary to specify the position where the virtual camera 148 was placed in the actual three-dimensional space and to accurately place the projector 101 at that position. There is. However, since it is difficult to accurately match the position of the virtual camera 148 in the virtual space and the position of the projector 101 in the real space, it is possible to obtain a projection mapping image that matches the outer shape of each surface of the three-dimensional object 104. Have difficulty.

An object of the present invention is to solve the above-mentioned problems, and to generate a mapping data capable of creating a projection mapping video that matches the outer shape of each surface of a three-dimensional object, a projector, a mapping data creation method, To provide a program and a projection mapping system.

In order to achieve the above object, according to one aspect of the present invention,
A display element having an image forming surface composed of a plurality of pixels; and a projection lens that projects an image formed on the image forming surface, and the image is projected from the projection lens toward a three-dimensional object. A projector,
A reception unit for receiving a data creation request;
A mapping data generation unit for generating mapping data for creating a projection mapping video in response to the data creation request;
The mapping data generation unit
A three-dimensional sensor unit that three-dimensionally measures the three-dimensional object and outputs three-dimensional position data representing the position and shape of each surface on which the image on the three-dimensional object is projected in three-dimensional coordinates;
The coordinate system of the three-dimensional position data is converted into a projector coordinate system in which the projection area of the image is expressed in three-dimensional coordinates with a point where a principal ray from a pixel located on the diagonal of the image forming surface intersects as an origin. A coordinate transformation unit;
Based on the three-dimensional position data coordinate-converted by the coordinate conversion unit, for each of the surfaces of the three-dimensional object, a position of the surface and a ridge line indicating the outer shape of the surface are acquired, and the position and the ridge line are obtained. And a data generation unit that generates the mapping data based on the projector.

According to another aspect of the invention,
A display element having an image forming surface composed of a plurality of pixels; and a projection lens that projects an image formed on the image forming surface, and the image is projected from the projection lens toward a three-dimensional object. A mapping data creation method performed by a projector,
A mapping data generation step for generating mapping data for creating a projection mapping video in response to the data creation request;
The mapping data generation step includes:
Three-dimensional measurement of the three-dimensional object, to obtain three-dimensional position data representing the position and shape of each surface on which the image on the three-dimensional object is projected in three-dimensional coordinates,
The coordinate system of the three-dimensional position data is converted into a projector coordinate system in which the projection area of the image is expressed in three-dimensional coordinates with a point where a principal ray from a pixel located diagonally on the image forming surface intersects as an origin. ,
Based on the coordinate-converted three-dimensional position data, for each of the surfaces of the three-dimensional object, a position of the surface and a ridge line indicating the outer shape of the surface are acquired, and the mapping is performed based on the position and the ridge line. A method for creating data for mapping is provided that includes generating data.

According to yet another aspect of the invention,
A display element having an image forming surface composed of a plurality of pixels; and a projection lens that projects an image formed on the image forming surface, and the image is projected from the projection lens toward a three-dimensional object. A program for causing a projector computer to execute mapping data generation processing for generating mapping data for generating a projection mapping video in response to a data generation request,
The mapping data generation process includes:
Processing to measure the three-dimensional object three-dimensionally and obtain three-dimensional position data representing the position and shape of each surface on which the image on the three-dimensional object is projected in three-dimensional coordinates;
The coordinate system of the three-dimensional position data is converted into a projector coordinate system in which the projection area of the image is expressed in three-dimensional coordinates with a point where a principal ray from a pixel located on the diagonal of the image forming surface intersects as an origin. Processing,
Based on the coordinate-converted three-dimensional position data, for each of the surfaces of the three-dimensional object, a position of the surface and a ridge line indicating the outer shape of the surface are acquired, and the mapping is performed based on the position and the ridge line. And a process for generating data.

According to yet another aspect of the invention,
The above projector,
An image processing device capable of mutual communication with the projector,
The video processing apparatus is provided with a projection mapping system that creates a projection mapping video based on data for a projection mapping video creation tool generated by the projector.

It is a schematic diagram which shows an example of the system which performs projection mapping. It is a schematic diagram which shows an example of the projection mapping image | video produced | generated with the system shown to FIG. 1A. It is a schematic diagram for demonstrating the projection mapping image | video production method in two-dimensional space. It is a schematic diagram for demonstrating the projection mapping image | video production method in three-dimensional space. 1 is a block diagram illustrating a configuration of a projector that is a first embodiment of the present invention. FIG. It is a schematic diagram for demonstrating the relationship of a field angle in case of launching, a projection optical axis, and a projection center point. It is a schematic diagram for demonstrating the relationship of a field angle in case of no launch, a projection optical axis, and a projection center point. It is a schematic diagram for demonstrating the relationship between a field angle at the time of enlarging a field angle, a projection optical axis, and a projection center point. It is a schematic diagram for demonstrating the relationship of a field angle at the time of performing a lens shift upward, a projection central axis, and a projection center point. It is a schematic diagram for demonstrating the relative positional relationship of the detection range of a three-dimensional sensor, and a projectable area. It is a schematic diagram which shows an example of a projection mapping system. It is a flowchart which shows one procedure of the data creation process for mapping. It is a schematic diagram which shows the projection state in which a projection optical axis passes along the center part of the lower side of a projection area in launch projection. It is a schematic diagram which shows the projection video area before and behind distortion correction. It is a schematic diagram which shows the projection state at the time of shifting a lens to upper left direction. It is a schematic diagram which shows the projection video area before and behind distortion correction. 5 is a flowchart showing a procedure for generating a three-dimensional data file 33; It is a schematic diagram which shows the positional relationship of the coordinate system of a three-dimensional sensor, and the coordinate system of a projector. It is a schematic diagram for demonstrating the installation state of a projector. It is a schematic diagram which shows an example of a segmentation process. It is a schematic diagram which shows the example which made the polygon mesh in both the triangular polygon and the quadrangular polygon. It is a schematic diagram which shows the example which made the polygon mesh only with a triangle polygon. It is a flowchart which shows one procedure of the production | generation process of a projector data file. It is a schematic diagram which shows an example of a projection direction vector, a vertical field angle, a left and right field angle, and a projection center coordinate when no field angle symmetrization is set in the field angle symmetrization selection parameter. It is a schematic diagram which shows an example of a projection direction vector, an up-down view angle, a left-right view angle, and a projection center coordinate when the view angle symmetrization is set in the view angle symmetrization selection parameter. 5 is a flowchart showing a procedure for generating a two-dimensional data file 33; It is a schematic diagram which shows an example of a perspective projection. It is a schematic diagram for demonstrating the angle of view of launch projection. It is a schematic diagram for demonstrating an angle-of-view central axis. It is a schematic diagram for demonstrating the projection target surface which shows one surface of a solid object. It is a schematic diagram for demonstrating the relationship between the virtual imaging surface of a camera for rendering, and an optical central axis. It is a schematic diagram for demonstrating the picked-up image of the camera for rendering. It is a schematic diagram for demonstrating the relationship between the image | video to a projection target surface, and a projection target surface. It is a schematic diagram for demonstrating the angle of view of an image | video when distortion correction is performed so that the image | video projected on a projection surface may become a square. It is a schematic diagram for demonstrating the relationship between the projection target surface which shows one surface of a solid object, and the projection direction and position of a projector. It is a schematic diagram for demonstrating the relationship between the virtual imaging surface of a camera for rendering, and an optical central axis. It is a schematic diagram which shows the state which images the image | video allocated on the virtual projection object surface with a rendering camera. It is a schematic diagram for demonstrating the relationship between the projection center axis | shaft which passes along the center of the projection image | video projected on the projection target surface, and a view angle center axis. FIG. 5 is a block diagram showing a configuration of a projector that is a second embodiment of the present invention. FIG. 5 is a block diagram showing a configuration of a projector that is a third embodiment of the present invention.

Next, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 4 is a block diagram showing a configuration of the projector according to the first embodiment of the present invention.
Referring to FIG. 4, the projector includes a communication control unit 1, a parameter storage unit 3, a projection unit 5, a projector data generation unit 6, a mapping data generation unit 7, a file storage unit 8, a projector projection design data storage unit 9, an image An angle symmetrizing unit 10 and an attitude sensor unit 11 are included.
The communication control unit 1 includes control means such as a CPU (Central Processing Unit), and is connected to an external device via the communication input / output unit 2 so as to be able to communicate with each other. RS-232C, a wired LAN (Local Area Network), a wireless LAN, or the like can be used as a communication means between the communication input / output unit 2 and an external device, but is not limited thereto.

The external device is an information processing apparatus such as a personal computer, for example, and includes a projection mapping video creation tool. The projection mapping video creation tool is a tool for creating a projection mapping video in a three-dimensional space, a tool for creating a projection mapping video in a two-dimensional space, or the like. Since these tools are existing, detailed description thereof is omitted here.
The communication control unit 1 controls the operation of each unit of the projector and transmits / receives data and instruction signals (or control signals) to / from the information processing apparatus. For example, the communication control unit 1 controls the projector data generation unit 6 and the mapping data generation unit 7 in accordance with an instruction from the information processing apparatus, and generates data (three-dimensional object data) necessary to create a projection mapping video. 2D data or 3D data or projector data) is generated.

The parameter storage unit 3 includes parameters used in mapping data creation processing necessary to create a projection mapping video (view angle symmetrization selection parameter, segmentation parameter, polygon meshing parameter, file format parameter, mapping mode parameter, etc.) Is stored.
The field angle symmetrization selection parameter is setting information indicating whether or not there is field angle symmetrization. The segmentation parameter is a threshold value for extracting each surface of the three-dimensional object based on three-dimensional position data (point cloud data) obtained by three-dimensionally measuring the shape of the surface of the three-dimensional object. The polygon meshing parameter is a parameter (polygon shape, mesh roughness, etc.) necessary for polygonal meshing of each surface of the extracted three-dimensional object. The file format parameter is a parameter for designating the formats of the two-dimensional data file and the three-dimensional data file. The mapping mode parameter is a parameter for designating any one of three modes of mapping for two-dimensional space, mapping for three-dimensional space, and no mapping.

A video signal is input from the information processing apparatus to the projection unit 5 via the projection video input unit 4. The projection unit 5 projects a video based on the input video signal from the projection video input unit 4, and includes a video processing unit 12, a distortion correction unit 13, a projection lens unit 14, and a distortion correction coefficient calculation unit 15.
The video processing unit 12 performs processing for converting the resolution of the input video signal to the resolution of the display device, processing for adjusting image quality, and the like. The distortion correction unit 13 corrects distortion (for example, trapezoidal distortion) of the image projected on the projection surface not facing the projector, according to the distortion correction coefficient, with respect to the video signal processed by the video processing unit 12. The distortion correction coefficient calculation unit 15 calculates the distortion in the distortion correction unit 13 based on the information for distortion correction set by the user or the horizontal and vertical inclination information of the projection surface from the view angle symmetrization unit 10. A distortion correction coefficient necessary for correction is calculated. The user can set information for distortion correction using an operation unit (not shown).

The projection lens unit 14 includes a display device that forms an image based on the video signal from the distortion correction unit 13 and a projection lens that projects an image formed by the display device. The projection lens includes a lens that can move in the direction of the optical axis, a zoom mechanism that changes the angle of view according to a zoom position corresponding to the position on the optical axis of the lens, and the projection lens as a whole orthogonal to the optical axis. And a lens shift mechanism that shifts in the direction of movement. The projection lens unit 14 supplies zoom / shift position information indicating the zoom position and lens shift position of the projection lens to the view angle symmetrizing unit 10. Here, the display device can be referred to as a display element having an image forming surface including a plurality of pixels, and an image based on the video signal from the distortion correction unit 13 is formed on the image forming surface. As the display element, a liquid crystal display element, DMD (digital micromirror device), or the like can be used.
The projector projection design data storage unit 9 stores design data related to projector projection. The design data is data related to optics for obtaining the field angle and projection center point from the zoom position and lens shift position of the projection lens, that is, how the field angle and projection center point change depending on the zoom position and lens shift position. It is data for calculating what to do. This data is based on data determined by optical design.

The attitude sensor unit 11 includes an attitude sensor that can detect 360 ° rotation angles and 360 ° rotation angles, for example, a triaxial acceleration sensor, and detects the inclination of the projector with respect to the horizontal plane. The output of the attitude sensor unit 11 is supplied to the projector data generation unit 6 and the mapping data generation unit 7.
The view angle symmetrization unit 10 acquires zoom / shift position information from the projection lens unit 14, acquires design data from the projector projection design data storage unit 9, and acquires view angle symmetrization selection parameters from the parameter storage unit 3. The mapping mode parameter is acquired from the parameter storage unit 3. The angle-of-view symmetrization unit 10 generates a projection direction vector and an angle of view indicating the direction of the projection center axis of the projector based on zoom / shift position information and design data in accordance with the mapping mode parameter and the angle-of-view symmetrization selection parameter. And the horizontal and vertical inclination angles of the projection surface with respect to the projection surface facing the projector are calculated.
Here, the angle of view represents the range of projection light from the projection lens (image projection range) as an angle, and the range of horizontal projection light as an angle is called the horizontal angle of view. A range of vertical projection light expressed as an angle is called a vertical angle of view. The projection central axis is the central axis (corresponding to the central ray) of the projection light from the projection lens, and can be used as a reference for the angle of view. The angle of view, the projection center axis, and the projection center point change according to the zoom position and the lens shift position.

FIG. 5A schematically shows the relationship between the angle of view, the projection optical axis, and the projection center point when launching is performed. FIG. 5B schematically shows the relationship between the angle of view, the projection optical axis, and the projection center point when there is no launch. FIG. 5C schematically shows the relationship between the angle of view, the projection optical axis, and the projection center point when the angle of view is enlarged. FIG. 5D schematically shows the relationship between the angle of view, the projection center axis, and the projection center point when the lens shift is performed upward. Here, the projection optical axis is an axis passing through the center of the image forming surface and perpendicular to the image forming surface. Here, FIGS. 5B to 5D correspond to cross sections in the vertical direction.
Usually, when the projector is placed on a table, a projection form called launching is used in which the image is projected above the projection optical axis so that the image is projected above the height of the table. In addition, there is a projection form called lens shift in which an image is projected by moving it to the vertical and horizontal positions with respect to the projection optical axis, and launch is one form of the lens shift.
For example, as shown in FIG. 5A, the projection center point 102 linearly connects each point of the four corners of the projection area 103 and each point of the four corners of the image forming area of the display device 100 with corresponding points. This is where the lines meet. Here, the projection area 103 is obtained by inverting the image of the image forming area of the display device 100 vertically and horizontally. In practice, since refraction is caused by the lens, lines connecting the four corners of the projection area 103 and the four corners of the image forming area of the display device 100 are not straight lines. The projection center point 102 needs to be determined in consideration of the lens configuration of the projection lens.

For example, the four corner points of the image forming area of the display device 100 are A point, B point, C point, and D point, respectively, and the four corner points of the projection area 103 are respectively a point, b point, c point, and d point. . Points a, b, c, and d correspond to points A, B, C, and D, respectively, and the arrangement of points a, b, c, and d is points A, B, and C. , The positional relationship is reversed up, down, left and right with respect to the arrangement of point D. In this case, the projection center point 102 is emitted from the point A and reaches the point a through the lens, the principal ray that exits from the point B and reaches the point b through the lens, and the point C. The principal ray that exits and reaches the point c via the lens and the principal ray that exits from the point D and reaches the point d via the lens are shown as intersecting points. Such an intersection of principal rays can be defined, for example, at the center of the aperture stop of the projection lens, and can be calculated based on lens design data.
In the example with launch shown in FIG. 5A, the projection optical axis 109 passes through the center of the lower end of the projection area 103, and the projection center point 102 is located above the projection optical axis 109. In this case, the projection center axis does not coincide with the projection optical axis 109.

In the example without launch shown in FIG. 5B, the projection optical axis 109 passes through the center of the projection area 103, and the projection center point 102 is located on the projection optical axis 109. In this case, the projection center axis coincides with the projection optical axis 109.
FIG. 5C is an example in which the angle of view is enlarged as compared with the example of FIG. 5B. 5B, the projection optical axis 109 passes through the center of the projection area 103, and the projection center point 102 is located on the projection optical axis 109. However, the projection center point 102 is a display device than the example of FIG. It is arranged on the 100 side. Also in this case, the projection center axis coincides with the projection optical axis 109.

FIG. 5D is an example in which the lens shift is performed so that the projection area 103 is shifted upward as compared with the example of FIG. 5B. As in the example of FIG. 5A, the projection optical axis 109 passes through the center of the lower end of the projection area 103, and the projection center point 102 is located above the projection optical axis 109. In this case, the projection center axis does not coincide with the projection optical axis 109.
As can be seen from the examples of FIGS. 5A to 5D, the angle of view, the projection center axis, and the projection center point change according to the zoom position and the lens shift position. In other words, the angle of view, the projection center axis, and the projection center point need to be determined according to the zoom position and the lens shift position.

Reference is again made to FIG. The process of calculating the projection direction vector, the view angle, and the tilt angle differs depending on whether the view angle symmetrization is performed or not.
When the mapping mode parameter is set to the 2D space mapping mode, or the mapping mode parameter is set to the 3D space mapping mode, and the view angle symmetrization selection parameter is set to no view angle symmetrization. The angle-of-view symmetrization unit 10 executes the following processes (A1) to (A3).
(A1) Based on the zoom / shift position information (zoom position and lens shift position) and design data, the left and right field angles in the horizontal direction and the vertical field angles in the vertical direction are obtained.
(A2) Considering the projection optical axis as the projection center axis, a vector in which the horizontal and vertical direction components are set to 0 is set as the projection direction vector.
(A3) Both horizontal and vertical inclinations are set to zero.

On the other hand, if the mapping mode parameter is set to the 3D space mapping mode and the view angle symmetrization selection parameter is set to view angle symmetrize, the view angle symmetrization unit 10 performs the following processing. (B1) to (B3) are executed.
(B1) Based on the zoom / shift position information (zoom position and lens shift position) and design data, the left and right field angles in the horizontal direction and the vertical field angles in the vertical direction are obtained.
(B2) The projection plane is perpendicular to the projection center axis, and the horizontal angle of view in the horizontal direction with respect to the projection center axis when the distortion correction is performed so that the image projected on the projection plane is square becomes equal. In addition, the projection center axis and the projection plane are determined so that the vertical angle of view in the vertical direction is equal, and the projection direction vector is set in the direction of the projection center axis.
(B3) The horizontal and vertical inclinations of the projection plane perpendicular to the projection center axis determined in (B2) with respect to the plane perpendicular to the projection optical axis are obtained.

The angle-of-view symmetrization unit 10 supplies the projection direction vector and the calculation result of the angle of view to the projector data generation unit 6 and supplies the calculation result of the tilt angle to the distortion correction coefficient calculation unit 15. Note that when the mapping mode parameter is a mode without mapping, the view angle symmetrization unit 10 does not perform the calculation processing of the projection direction vector, the view angle, and the tilt.
The projector data generation unit 6 generates a projector data file for setting a rendering camera and a projector in a virtual three-dimensional space. This projector data file is used in a tool for creating a projection mapping video in a three-dimensional space on the information processing apparatus side.

The projector data generation unit 6 includes an initial projector data generation unit 16, a projector data world coordinate conversion unit 17, a projector data vertical offset unit 18, and a projector data file generation unit 19.
The initial projector data generation unit 16 uses the origin of the projector coordinate system that is the projection center point of the projector (projection center point 102 as shown in FIGS. 5A to 5D), that is, the coordinates (0, 0, 0) as the projector position. Set to coordinates. The initial projector data generation unit 16 generates projector data including the projector position coordinates, the projection direction vector from the angle-of-view symmetrization unit 10, and the angle of view (up / down / left / right angle of view).

The projector data world coordinate conversion unit 17 converts the direction of the projection vector included in the projector data generated by the initial projector data generation unit 16 based on the inclination of the projector with respect to the horizontal plane detected by the attitude sensor unit 11 into the horizontal plane. Convert to world coordinate system based on.
The projector data vertical offset unit 18 adjusts the vertical coordinate of the three-dimensional position data performed by the mapping data generation unit 7 and adjusts the projector position included in the projector data from the projector data world coordinate conversion unit 17. Change the vertical coordinate of the coordinate. In order to perform this coordinate change, the vertical offset amount is supplied from the mapping data generation unit 7 to the projector data vertical offset unit 18.

The projector data file generation unit 19 generates a projector data file 31 based on the projector data from the projector data vertical offset unit 18. The projector data file 31 is stored in the file storage unit 8.
The mapping data generation unit 7 generates a two-dimensional data file 32 indicating the outer position of each surface of the three-dimensional object or a three-dimensional data file 33 of the three-dimensional object. The two-dimensional data file 32 is used by a tool for creating a projection mapping video in a two-dimensional space. The three-dimensional data file 33 is used by a tool that creates a projection mapping video in a three-dimensional space. The two-dimensional data file 32 and the three-dimensional three-dimensional data file 33 are stored in the file storage unit 8.

The mapping data generation unit 7 includes a three-dimensional sensor unit 20, a calibration data storage unit 21, a three-dimensional position data projector coordinate conversion unit 22, a three-dimensional position data world coordinate conversion unit 23, a vertical offset calculation unit 24, and a three-dimensional position. A data vertical offset unit 25, a three-dimensional position data segmentation unit 26, a perspective projection unit 27, a three-dimensional position data polygon meshing unit 29, a two-dimensional data file generation unit 28, and a three-dimensional data file generation unit 30 are provided.
The three-dimensional sensor unit 20 includes a three-dimensional sensor that is arranged toward the optical axis direction of the projection lens and three-dimensionally measures each surface of a three-dimensional object that is a projection target. The detection range of the three-dimensional sensor includes the entire projectable area where the image of the projection lens can be projected.

FIG. 6 is a schematic diagram for explaining the relative positional relationship between the detection range of the three-dimensional sensor and the projectable area. As shown in FIG. 6, the three-dimensional sensor 108 is arranged toward the optical axis direction of the projection lens 101a. The projection area can be enlarged or reduced using the zoom function, and the projection area can be moved up, down, left and right using the lens shift function.
The projection area 113 is a projection area when the lens is shifted in the upper right direction to minimize the zoom. The projection area 114 is a projection area when the lens is shifted in the upper right direction to maximize the zoom. The projection area 115 is a projection area when the lens is shifted in the upper left direction to minimize the zoom. The projection area 116 is a projection area when the lens is shifted in the upper left direction to maximize the zoom.

The projection area 117 is a projection area when the lens is shifted in the lower right direction to minimize the zoom. The projection area 118 is a projection area when the zoom is maximized by shifting the lens in the lower right direction. The projection area 119 is a projection area when the lens is shifted in the lower left direction to minimize the zoom. The projection area 120 is a projection area when the lens is shifted in the lower left direction to maximize the zoom.
The detection range of the three-dimensional sensor 108 includes the entire projectable area where the image from the projection lens 101a can be projected, that is, the entire projection areas 113 to 120. Therefore, the three-dimensional sensor 108 can measure the three-dimensional position of the three-dimensional object arranged in the projectable area. The three-dimensional sensor unit 20 supplies the three-dimensional position data output from the three-dimensional sensor 108 to the three-dimensional position data projector coordinate conversion unit 22.

As the three-dimensional sensor 108, for example, a TOF (Time-of-Flight) method or a triangulation method three-dimensional sensor can be used, but it is not limited to these methods. The TOF method is a method of performing three-dimensional measurement by projecting light toward an object and measuring the time until the projected light is reflected by the object and returned. Examples of the triangulation method include a passive triangulation method and an active triangulation method. In the passive triangulation method, an object is photographed at the same time with two cameras arranged side by side on the left and right, and the principle of triangulation is used based on the difference in position on the captured image of the object obtained by each camera. This is a method for measuring and is also called a stereo camera method. The active triangulation method is a method of irradiating light on an object and performing three-dimensional measurement using the principle of triangulation based on information on reflected light from the object.

Refer to FIG. 4 again. The calibration data storage unit 21 stores calibration data. The calibration data includes parameters (rotation amount and translation amount) for converting the coordinate system of the three-dimensional sensor 108 to the coordinate system of the projector, a reference zoom position, and a reference lens shift position. The amount of rotation and the amount of translation can be obtained by performing calibration that measures the positional relationship between the coordinate system of the three-dimensional sensor and the coordinate system of the projector. The reference zoom position and the reference lens shift position are the zoom position and the lens shift position when calibration is performed.
The three-dimensional position data projector coordinate conversion unit 22 acquires calibration data (rotation amount, translation amount, reference zoom position, and reference lens shift position) from the calibration data storage unit 21 and designs from the projector projection design data storage unit 9. Data is acquired, and zoom / shift position information is acquired from the projection lens unit 14. The three-dimensional position data projector coordinate conversion unit 22 uses the three-dimensional position data of the three-dimensional object from the three-dimensional sensor based on the calibration data and the design data, and the three-dimensional position of the projector coordinate system with the projection center point as the origin. Convert to position data. The calibration data and design data can be referred to as coordinate conversion data for converting the coordinate system of the three-dimensional sensor into a projector coordinate system in which the projection center point is the origin.

The three-dimensional position data world coordinate conversion unit 23 acquires the inclination of the projector with respect to the horizontal plane from the attitude sensor unit 11, and acquires the three-dimensional position data coordinate-converted into the projector coordinate system from the three-dimensional position data projector coordinate conversion unit 22. . The three-dimensional position data world coordinate conversion unit 23 converts the three-dimensional position data coordinate-converted into the projector coordinate system into a world coordinate system based on the horizontal plane based on the tilt of the projector with respect to the horizontal plane. The three-dimensional position data coordinate-converted into the world coordinate system is supplied to the vertical offset calculation unit 24 and the three-dimensional position data vertical offset unit 25.
The vertical offset calculator 24 obtains the minimum value of the vertical coordinates from the three-dimensional position data coordinate-converted to the world coordinate system from the three-dimensional position data world coordinate converter 23. When the minimum value of the vertical coordinate is a negative number, the vertical offset calculation unit 24 outputs a vertical offset amount indicating the absolute value of the minimum value of the vertical coordinate. When the minimum value of the vertical coordinate is a positive number, the vertical offset calculation unit 24 outputs a vertical offset amount indicating 0. The output of the vertical offset calculation unit 24 is supplied to the projector data vertical offset unit 18 and the three-dimensional position data vertical offset unit 25.

The three-dimensional position data vertical offset unit 25 vertically converts the three-dimensional position data coordinate-converted into the world coordinate system from the three-dimensional position data world coordinate conversion unit 23 based on the vertical offset amount calculated by the vertical offset calculation unit 24. Offset in direction. Specifically, the vertical offset is performed by adding the vertical offset amount to the vertical coordinate of the three-dimensional position data. The three-dimensional position data that has been offset in the vertical direction is supplied to the three-dimensional position data segmentation unit 26, the three-dimensional position data polygon meshing unit 29, and the three-dimensional position data file creation unit 30.
The three-dimensional position data segmentation unit 26 acquires a segmentation parameter from the parameter storage unit 3. The three-dimensional position data segmentation unit 26 detects a surface of a three-dimensional object and a ridge line indicating the shape from the three-dimensional position data from the three-dimensional position data vertical offset unit 25 based on the segmentation parameter. The detection results of the surface and the ridge line are supplied to the perspective projection unit 27 and the three-dimensional position data polygon meshing unit 29.

The perspective projection unit 27 acquires projector data from the projector data vertical offset unit 18. The perspective projection unit 27 uses the projector data to project the ridgeline detected by the three-dimensional position data segmentation unit 26 onto the projection surface, thereby obtaining two-dimensional data indicating the external position of each surface of the three-dimensional object in the projection image. Generate. The two-dimensional data is supplied to the two-dimensional data file generation unit 28.
The two-dimensional data file generation unit 28 acquires file format parameters from the parameter storage unit 3. The two-dimensional data file generation unit 28 generates a two-dimensional data file 32 based on the file format indicated by the file format parameter from the two-dimensional data created by the perspective projection unit 27. The two-dimensional data file 32 is stored in the file storage unit 8.

The three-dimensional position data polygon meshing unit 29 acquires a polygon meshing parameter and a file format parameter from the parameter storage unit 3. The 3D position data polygon meshing unit 29 converts the surface detected by the 3D position data segmentation unit 26 into a polygon mesh based on the polygon meshing parameter and the file format indicated by the file format parameter. Data of the polygon meshed surface is supplied to the three-dimensional data file generation unit 30.
The three-dimensional data file generation unit 30 acquires file format parameters from the parameter storage unit 3. The three-dimensional data file generation unit 30 generates a three-dimensional data file 33 based on the file format indicated by the file format parameter from the surface data polygon- meshed by the three-dimensional position data polygon meshing unit 29. The three-dimensional data file 33 is stored in the file storage unit 8.

Next, the configuration of a projection mapping system using the projector of this embodiment will be described.
FIG. 7 shows an example of the projection mapping system. Referring to FIG. 7, the projection mapping system includes a projector 201 and a video processing device 205 such as a personal computer. The projector 201 and the video processing device 205 are connected to each other via a communication unit 207 so that they can communicate with each other. For example, the communication unit 207 may be composed of a communication cable and a video signal cable. In this case, the video signal cable is used by a video processing device 205 described later to supply a projection mapping video for three-dimensional space to the projector 201. For example, the communication cable is used by the projector 201 described later to supply data such as the projector data file 31, the two-dimensional data file 32, and the three-dimensional data file 33 to the video processing device 205. It is also possible to configure the communication cable and the video signal cable with one cable. The communication unit 207 may include a wireless communication unit.
The projector 201 has the configuration described with reference to FIGS. 4 to 6 and projects video based on the video signal from the video processing device 205 onto each surface of the three-dimensional unit 104. The video processing device 205 includes at least one of a two-dimensional space projection mapping video creation and a three-dimensional space projection mapping video creation tool.

In the projection mapping system shown in FIG. 7, projection mapping is performed according to the following procedure.
First, the projector 201 executes mapping data creation processing in response to a data creation instruction (data creation request) from the video processing device 205. In this mapping data creation process, data for a two-dimensional space projection mapping video creation tool or data for a three-dimensional space projection mapping video creation tool is created. Here, for convenience, it is assumed that the video processing device 205 is equipped with a projection mapping video creation tool for 3D space, and the projector 201 creates data for the projection mapping video creation tool for 3D space. This data includes three-dimensional data of a three-dimensional object and projector data for setting a rendering camera and a projector in a virtual three-dimensional space.

Next, the video processing device 205 acquires the data for the 3D space projection mapping video creation tool generated by the projector 201, that is, the 3D data file 33 and the projector data file 31. Then, the 3D space projection mapping video creation tool creates a 3D space projection mapping video using the 3D data file 33 and the projector data file 31 acquired from the projector 201.
The video processing device 205 supplies a projection mapping video for 3D space to the projector 201. The projector 201 projects an image based on the three-dimensional space projection mapping image on each surface of the three-dimensional unit 104.

When the video processing device 205 is equipped with a projection mapping video creation tool for 2D space, the projector 201 uses data for the projection mapping video creation tool for 2D space instead of the 3D data file 33. A two-dimensional data file 32 is created. The two-dimensional data file 32 is two-dimensional data indicating the external position of each surface of the three-dimensional object in the projection video, and is supplied from the projector 201 to the video processing device 205. In the video processing device 205, the two-dimensional space projection mapping video creation tool uses the two-dimensional data file 32 to create a two-dimensional space projection mapping video.

Next, the operation of the mapping data creation process of the projector 201 of this embodiment will be described in detail.
FIG. 8 shows a procedure of mapping data creation processing.
In step S <b> 10, the operator performs an input operation (data creation request) for setting parameters necessary for creating a projection mapping video on the video processing device 205. In response to this input operation, a parameter setting start instruction and parameter setting screen information are supplied from the video processing device 205 to the projector 201. In the projector 201, the parameter setting screen information is supplied to the projection unit 5, and the communication control unit 1 causes the projection unit 5 to project the parameter setting screen according to the parameter setting start instruction.

The operator inputs necessary parameter information with reference to the parameter setting screen. Here, the parameters include parameters necessary for mapping data creation processing (view angle symmetrization selection parameter, segmentation parameter, polygon meshing parameter, file format parameter, mapping mode parameter, etc.). The parameter input information is supplied from the video processing device 205 to the projector 201. In the projector 201, the communication control unit 1 stores parameters input by the operator in the parameter storage unit 3.
Next, in step S <b> 11, the view angle symmetrization unit 10 acquires zoom / shift position information from the projection lens unit 14, acquires design data from the projector projection design data storage unit 9, and receives the view angle from the parameter storage unit 3. A symmetrization selection parameter is acquired, and a mapping mode parameter is acquired from the parameter storage unit 3. The angle-of-view symmetrization unit 10 generates a projection direction vector and an angle of view indicating the direction of the projection center axis of the projector based on zoom / shift position information and design data in accordance with the mapping mode parameter and the angle-of-view symmetrization selection parameter. And the horizontal and vertical inclination angles of the projection surface with respect to the projection surface facing the projector are calculated.

The calculation process of the angle of view, the projection direction, and the tilt by the angle of view symmetrization unit 10 includes the following first and second processes.
(First process)
When the angle of view is not symmetrized so that the angle of view is vertically and horizontally symmetrical, that is, the mapping mode parameter is set to 2D space mapping or the mapping mode parameter is set to 3D space mapping. When the field angle symmetrization selection parameter is set to “no field angle symmetrization”, the field angle symmetrization unit 10 executes the first process.
In the first processing, the angle-of-view symmetrization unit 10 includes the zoom position and lens shift position of the projection lens unit 14, the angle of view of the projector stored in the projector projection design data storage unit 9, zoom characteristics, lens shift characteristics, and the like. The angle of view is obtained from design data related to the projection. Then, the view angle symmetrizing unit 10 sets the projection direction vector to the direction of the projection optical axis, that is, a vector in which the horizontal / vertical direction components are 0 respectively. As for the inclination of the projection plane, the angle-of-view symmetrization unit 10 does not change the projection plane, and therefore sets both the horizontal inclination and the vertical inclination to zero.

(Second process)
When performing angle-of-view symmetrization to make the angle of view vertically and horizontally symmetric, that is, mapping for the three-dimensional space is set in the mapping mode parameter, and angle-of-view symmetry is set in the view angle symmetrization selection parameter Is set, the angle-of-view symmetrization unit 10 executes the second process.
In the second processing, the view angle symmetrizing unit 10 includes the zoom position and lens shift position of the projection lens unit 14, the angle of view of the projector stored in the projector projection design data storage unit 9, the zoom characteristic, the lens shift characteristic, and the like. The angle of view is obtained from design data related to the projection. Next, the angle-of-view symmetrizing unit 10 is a projection plane perpendicular to the projection center axis, and the horizontal direction with respect to the projection center axis when distortion correction is performed so that the image projected on the projection plane becomes square. A projection plane is defined such that the vertical and vertical vertical angles of view are equal, and the projection direction vector is set in the direction of the projection center axis. Then, the view angle symmetrization unit 10 obtains the horizontal and vertical inclinations of the projection plane perpendicular to the projection central axis with respect to the plane perpendicular to the projection optical axis.

The projection unit 5 performs distortion correction so that the projected image on the projection surface becomes square. However, since the projected image area after this distortion correction is smaller than before correction, the distortion correction is not performed. It also affects the corners. Therefore, in consideration of the change in the projected video area after distortion correction, the projection center axis and the projector apparatus in which the horizontal field angle is equal to each other and the vertical field angle is equal to each other in the vertical direction It is desirable to obtain the inclination of the projection surface with respect to the projection surface when the screen and the screen face each other.

Here, the operations of the view angle symmetrization unit 10 and the projection unit 5 will be described in more detail.
In the projection unit 5, the distortion correction coefficient calculation unit 15 normally calculates a distortion correction coefficient according to the distortion correction setting by the user, and corrects the distortion that occurs when the distortion correction unit 13 projects on a screen that is not directly facing the projector. To do. However, when 2D space mapping or 3D space mapping is set in the mapping mode parameter, the distortion correction coefficient calculation unit 15 does not perform distortion correction setting by the user, but projects from the view angle symmetrization unit 10. A distortion correction coefficient is calculated according to the inclination of the surface, and the distortion correction unit 13 corrects the distortion based on the distortion correction coefficient. The view angle symmetrization 10 and the distortion correction processing in the projection unit 5 are executed every time the user adjusts the zoom position and the lens shift position of the projection lens unit 14. This eliminates the problem of deviation between the projected image and the three-dimensional object when using a tool for creating a projection mapping image in a three-dimensional space in which the angle of view can only be set to be vertically or horizontally symmetrical. It becomes possible.

The solution to the above problem will be described with reference to FIGS.
FIG. 9A shows a projection state in which the projection optical axis passes through the center of the lower side of the projection area in launch projection, and FIG. 9B shows the projection video area before and after distortion correction.
In the example of FIG. 9, the projection center axis 110 does not coincide with the projection optical axis 109. With respect to the vertical field angle with respect to the projection optical axis 109, the upper angle is θ _T and the lower angle is θ _B. With respect to the angle of view in the vertical direction with respect to the projection center axis 110, the upper angle is θ ′ _T and the lower angle is θ ′ _B. Since the lower angle θ _B = 0 and the upper angle θ _T and the lower angle θ _B do not coincide with each other, they are not vertically symmetric. The angle-of-view symmetrizing unit 10 is configured so that the vertical angle of view is vertically symmetrical (θ ′ _T = θ ′ _B ) when distortion correction is performed so that the image projected on the projection surface is square. The image projection central axis 110 and the projection plane perpendicular to the image projection central axis 110 are obtained, and the inclination of the projection plane with respect to the projection plane when the projector 201 and the screen face each other is obtained. Then, the view angle symmetrizing unit 10 supplies the inclination of the projection surface to the distortion correction coefficient calculation unit 15, and the distortion correction coefficient calculation unit 15 calculates a distortion correction coefficient based on the inclination of the projection surface, and the distortion correction unit 13. Based on the distortion correction coefficient, distortion correction is performed so that the projected image on the projection surface becomes square. According to this distortion correction, the projection video area 122 is corrected to the projection video area 123 as shown in FIG. 9B, and as a result, the projection area 103 is corrected to the projection area 121 as shown in FIG. 9A.

FIG. 10A shows a projection state when the lens is shifted in the upper left direction, and FIG. 10B shows a projected video area before and after distortion correction. Also in this example, regarding the angle of view in the vertical direction with respect to the projection center axis 110, the upper angle is θ ′ _T and the lower angle is θ ′ _B. In addition, regarding the horizontal field angle with reference to the projection center axis 110, the right angle is θ ′ _R and the left angle is θ ′ _L.
Because it is θ _'T ≠ θ' _B, the vertical angle of view does not become vertically symmetrical. Also, since θ ′ _R ≠ θ ′ _L , the horizontal field angle is not symmetrical. In this case, the angle-of-view symmetrizing unit 10 makes the vertical angle of view vertically symmetric (θ ′ _T = θ ′ _B ) when correcting distortion so that the image projected on the projection surface becomes square. In addition, the projection central axis 110 and a projection plane perpendicular to the projection central axis 110 are obtained so that the horizontal angle of view is symmetrical (θ ′ _R = θ ′ _L ), and the projector 201 of the projection plane is The inclination with respect to the projection surface when the screen is directly facing is obtained. Then, the view angle symmetrizing unit 10 supplies the inclination of the projection surface to the distortion correction coefficient calculation unit 15, and the distortion correction coefficient calculation unit 15 calculates a distortion correction coefficient based on the inclination of the projection surface, and the distortion correction unit 13. Based on the distortion correction coefficient, distortion correction is performed so that the projected image on the projection surface becomes square. According to this distortion correction, the projection video area 122 is corrected to the projection video area 123 as shown in FIG. 10B, and as a result, the projection area 103 is corrected to the projection area 121 as shown in FIG. 10A.

Again referring to FIG. In step S12, the communication control unit 1 of the projector 201 determines whether or not a mapping data generation start instruction has been received. After adjusting the position, projection direction, zoom position, and lens shift position of the projector 201 so that the projected image is projected onto the three-dimensional object that is the projection target, the user starts mapping data generation on the information processing device 205. An input operation for instructing is performed. In response to this input operation, a mapping data generation start instruction is supplied from the information processing apparatus 205 to the projector 201.
When the mapping data generation start instruction is received, in steps S13 and S14, the communication control unit 1 determines whether the mapping mode parameter is set to the two-dimensional space mapping mode or the three-dimensional space mapping mode.
If the mapping mode parameter is set to the two-dimensional space mapping mode, the communication control unit 1 causes the mapping data generation unit 7 to generate the two-dimensional data file 32 in step S15. On the other hand, if the mapping mode parameter is set to the 3D space mapping mode, in step S16, the communication control unit 1 causes the mapping data generation unit 7 to generate the 3D data file 33 and the projector. The data generation unit 6 generates a projector data file 31. If the no mapping mode is set in the mapping mode parameter, the mapping data creation process is terminated without generating the data file.

Hereinafter, the operation will be described separately for the two-dimensional space mapping mode and the three-dimensional space mapping mode.
(3D space mapping mode)
In the three-dimensional space mapping mode, the mapping data generation unit 7 executes the generation process of the three-dimensional data file 33 in accordance with the mapping data generation start instruction from the communication control unit 1, and the projector data generation unit 6 The generation process of the file 31 is executed.
FIG. 11 shows a procedure for generating the three-dimensional data file 33.
In step S20, the three-dimensional sensor unit 20 measures a three-dimensional object three-dimensionally and outputs three-dimensional position data. This three-dimensional position data is represented by point group data indicated by three-dimensional coordinates in the coordinate system of the three-dimensional sensor 108.

In step S 21, the three-dimensional position data projector coordinate conversion unit 22 acquires the rotation amount, the translation amount, the reference zoom position, and the reference lens shift position from the calibration data storage unit 21, and the current zoom position and the reference lens shift position from the projection lens unit 14. The lens shift position is acquired, and the design data is acquired from the projector projection design data storage unit 9. Then, the 3D position data projector coordinate conversion unit 22 receives the 3D from the 3D sensor 20 based on the rotation amount, translation amount, reference zoom position, reference lens shift position, current zoom position and lens shift position, and design data. The coordinate system of the dimension position data is converted into the coordinate system of the projector 201 with the projection center point as the origin.

FIG. 12 schematically shows the positional relationship between the coordinate system of the three-dimensional sensor 108 and the coordinate system of the projector 201.
As shown in FIG. 12, the origin of the coordinate system of the three-dimensional sensor 108 does not coincide with the origin (projection center point) of the coordinate system of the projector 201, and the detection direction of the three-dimensional sensor 108 is also the projection direction of the projector 201. It does not match. For this reason, the coordinate system of the three-dimensional position data measured by the three-dimensional sensor 108 is different from the coordinate system of the projector 201. The coordinate transformation in step S21 can be defined by a rotation amount with respect to the three coordinate axes (XYZ) and a translation amount representing movement with respect to the three coordinate axes (XYZ). Obtaining the amount of rotation and the amount of translation is called calibration.

In the projector 201, the projection center point that is the origin of the projector coordinate system is not always the same position, but is moved by a zoom or lens shift operation. The shift positions are stored in the calibration data storage unit 21 as the reference zoom position and the reference lens shift position, respectively.
The three-dimensional position data projector coordinate conversion unit 22 first obtains the coordinates of the reference projection center point when calibration is performed using the design data, the reference zoom position, and the reference lens shift position. Next, the three-dimensional position data projector coordinate conversion unit 22 obtains the coordinates of the current projection center point based on the current zoom position and lens shift position, and changes the coordinates of the reference projection center point to the coordinates of the current projection center point. The translation amount for the coordinate transformation of is obtained.

Next, the three-dimensional position data projector coordinate conversion unit 22 converts the three-dimensional position data from the three-dimensional sensor unit 20 based on the rotation amount and the translation amount stored in the calibration data storage unit 21. Next, the three-dimensional position data projector coordinate conversion unit 22 moves the coordinates by the amount of translation from the coordinates of the reference projection center point to the coordinates of the current projection center point based on the current zoom position and lens shift position. . As a result, the coordinate system of the three-dimensional position data from the three-dimensional sensor unit 20 is converted into a projector coordinate system with the current projection center point as the origin.
In step S22, the three-dimensional position data world coordinate conversion unit 23 converts the three-dimensional position data coordinate-converted into the projector coordinate system based on the inclination of the projector with respect to the horizontal plane detected by the attitude sensor unit 11, and uses the horizontal plane as a reference. Convert to the world coordinate system.
The projector 201 is installed tilted in the front-rear direction and the left-right direction, or installed upside down. For example, the installation state of the projector 201 includes various states such as a horizontal state 124, an upside down state 125, an upward state 126, a downward state 127, a right rotation state 128, and a left rotation state 129, as shown in FIG. For this reason, the three-dimensional position data world coordinate conversion unit 23 performs a conversion process to the world coordinate system based on the horizontal plane as shown in FIG. 12 based on the horizontal inclination detected by the attitude sensor unit 11. .

In addition, the projection mapping video creation tool for 3D space can create not only a planar video but also a 3D projection mapping for representing a stereoscopic video as a video projected on the surface of a three-dimensional object. In order to perform three-dimensional expression, it is necessary to set the position of the viewpoint. That is, the viewpoint is set at a three-dimensional position, and an image of the surface of the three-dimensional object is created so that it can be seen as a three-dimensional object when viewed from there. At this time, the viewpoint is the position of the human eye standing on the ground, and it is necessary to convert the viewpoint into a world coordinate system based on a horizontal plane parallel to the ground in order to set the viewpoint.

Since the coordinate in the vertical direction of the coordinate-converted three-dimensional position data may include a negative number, it is necessary to adjust the coordinate in the vertical direction. In this vertical coordinate adjustment, the vertical offset amount calculation unit 24 calculates the vertical offset amount so that all the vertical coordinates of the three-dimensional position data become 0 or more, and the three-dimensional position data vertical offset unit 25 Adds the vertical offset to all the vertical coordinates of the three-dimensional position data. Specifically, the vertical offset amount calculation unit 24 obtains the minimum value of the vertical coordinates of the three-dimensional position data, and if it is a negative number, outputs the absolute value of the minimum value as the vertical offset amount, If it is a positive number, 0 is output as the vertical offset amount.
In step S24, the three-dimensional position data segmentation unit 26 generates a ridge line indicating the surface of the three-dimensional object to be projected and its outer shape from the three-dimensional position data based on the segmentation parameters stored in the parameter storage unit 3. To detect.

FIG. 14 shows an example of the segmentation process. The three-dimensional position data segmentation unit 26 first obtains a normal vector 132 for each point of the point cloud data 130 of the three-dimensional position data. Here, the normal vector 132 is a composite vector of the normal vector 131 of a triangular surface composed of a point of interest and peripheral points. Next, the three-dimensional position data segmentation unit 26 compares the normal vectors of adjacent points, and if the difference is smaller than the threshold value set by the segmentation parameter, the three-dimensional position data segmentation unit 26 sets the same plane. Thereby, the

surfaces

133 and 134 can be extracted. When the surface extraction is completed, the three-dimensional position data segmentation unit 26 calculates an edge line 135 that is an intersection line between the

adjacent surfaces

133 and 134.
In the segmentation process described above, how many curved surfaces are extracted as one surface is determined according to the threshold value set by the segmentation parameter. In this example, the segmentation parameter is a threshold for the difference between the normal vectors, but the value differs depending on the segmentation method.

In step S25, the three-dimensional position data polygon meshing unit 29 performs the three-dimensional position data segmentation unit based on the polygon meshing parameters stored in the parameter storage unit 3 and the file format parameters stored in the parameter storage unit 3. The surface detected at 26 is converted to a polygon mesh.
The polygons that can be handled differ depending on the format of the three-dimensional data file to be generated. For example, there are a format that can handle only triangular polygons, a format that can handle only triangular and quadrangular polygons, and a format that can handle polygonal polygons. For this reason, the three-dimensional position data polygon meshing unit 29 performs polygonization using a file format parameter that specifies the shape of the polygon and a polygon meshing parameter that specifies the roughness of the polygon. If the polygon is made fine, the curved surface can be expressed more smoothly, but the amount of calculation in the tool for creating the projection mapping video increases.

FIG. 15A shows an example in which a polygon mesh is formed using both a triangular polygon and a quadrangular polygon, and FIG. 15B shows an example in which a polygon mesh is formed using only a triangular polygon. As can be seen from FIGS. 15A and 15B, compared to the virtual three-

dimensional object

300, 301 in which the polygon mesh is formed by both the triangular polygon and the quadrilateral polygon, the virtual three-dimensional object 302 in which the polygon mesh is formed only by the triangle polygon. 303 have finer polygons.
In step S26, the 3D data file generation unit 30 determines the 3D position data from the 3D position data vertical offset unit 25 or the polygon mesh from the 3D position data polygon meshing unit 29 based on the file format parameter. A three-dimensional data file 33 suitable for the designated file format is generated from the converted three-dimensional position data. Depending on the file format, the three-dimensional data file generation unit 30 calculates information such as the normal vector of each vertex of the polygon and the normal vector of the mesh surface, and generates a three-dimensional data file 33 including these information. To do.
When the 3D data file 33 is generated using the 3D position data from the 3D position data vertical offset unit 25, the 3D data file 33 is the raw data of the measured 3D position in the world coordinate system. including.

Next, generation processing of the projector data file 31 will be described. FIG. 16 shows a procedure for generating the projector data file 31.
In step S30, the initial projector data generation unit 16 sets the origin of the projector coordinate system, that is, the projection center of the projector, that is, the coordinates (0, 0, 0) as the projector position coordinates. Together with the projection direction vector and the angle of view, initial projector data including the projection direction vector, the angle of view, and the projector position coordinates is generated.
FIG. 17 shows an example of a projection direction vector, a vertical field angle, a left and right field angle, and a projection center coordinate when no field angle symmetrization is set as the field angle symmetrization selection parameter. In this example, the projection optical axis 109 passes through the center of the lower end of the projection area 103. The lower angle θ _B = 0, and the upper angle θ ′ _T and the lower angle θ _B do not match, but the projection optical axis 109 is regarded as the projection center axis 110. The origin of the projector coordinate system that is the projection center point 102 of the projector 201, that is, the coordinates (0, 0, 0) are set as the coordinates of the projector position 138. The projection direction vector 136 is the direction of the projection optical axis 109 (the horizontal and vertical direction components are 0 respectively).

FIG. 18 shows an example of the projection direction vector, the vertical and horizontal field angles, the left and right field angles, and the projection center coordinates when the field angle symmetrization is set as the field angle symmetrization selection parameter. In this example, since the lower angle θ _B = 0 and the upper angle θ _L and the lower angle θ _B do not match, the vertical angle of view among the angle of view 137 is not vertically symmetric. When distortion correction is performed so that the image projected on the projection plane perpendicular to the projection center axis is square, the vertical angle of view is vertically symmetrical (θ ' _T = θ' _B ), and the horizontal direction The projection center axis 110 is set so that the angle of view is symmetrical (θ ′ _R = θ ′ _L ). The projection direction vector 136 is a vector indicating a direction in which the vertical vertical field angle is symmetric and the horizontal horizontal field angle is symmetric.

In step S31, the projector data world coordinate conversion unit 17 generates initial projector data based on the inclination of the projector with respect to the horizontal plane detected by the attitude sensor unit 11 in the same manner as the processing in the three-dimensional position data world coordinate conversion unit 23. The projection direction vector generated by the unit 16 is converted into a world coordinate system based on the horizontal plane.
In step S 32, the projector data vertical offset unit 18 converts the vertical offset amount calculated by the vertical offset amount calculation unit 24 into the vertical coordinate of the projector position coordinate in the same manner as the processing in the three-dimensional position data vertical offset unit 25. Add and adjust vertical coordinates.
In step S33, the projector file generation unit 19 generates a projector data file 31.

After the three-dimensional data file 33 and the projector data file 31 are generated, the information processing apparatus takes in the three-dimensional data file 33 and the projector data file 31. A three-dimensional space projection mapping video creation tool constructs a three-dimensional object in a virtual three-dimensional space based on the contents of the three-dimensional data file 33, assigns an image to each surface of the three-dimensional object, and reproduces it. In accordance with the contents of 31, a rendering camera is set and rendering is performed to create a projected image. If this projected image is projected from the projector, projection mapping can be performed in a state where the outer shape of the three-dimensional object is exactly matched.
It should be noted that, in the 3D projection mapping that performs three-dimensional video expression, in the case where the rendering camera is set at the viewpoint position in order to set the illumination and reproduce the light hitting condition, etc., according to the contents of the projector data file 31 Then, the projector is set, the rendering camera is set at the viewpoint position, and rendering is performed. Using the positional relationship with the three-dimensional object, it is converted into a projector projected image to create a projected image. If this projected image is projected from the projector, projection mapping can be performed in a state where the outer shape of the three-dimensional object is exactly matched.

(Mapping mode for 2D space)
In the two-dimensional space mapping mode, the mapping data generation unit 7 executes the generation process of the two-dimensional data file 32 according to the mapping data generation start instruction from the communication control unit 1, and the projector data generation unit 6 Processing up to the vertical offset unit 18 is executed.

FIG. 19 shows a procedure for generating the two-dimensional data file 33.
In step S40, three-dimensional position data is acquired, in step S41, the coordinate system of the three-dimensional position data is converted into the projector coordinate system, in step S42, converted into the world coordinate system, in step S43, a vertical offset is performed, and in step S44. The surface and ridgeline of the three-dimensional object are detected with. The processes in steps S40 to S44 are the same as the processes in steps S20 to S24 shown in FIG. On the other hand, the projector data generation unit 6 performs the processing of steps S30 to S33 shown in FIG.
After the processing in step S44, in step S45, the perspective projection unit 27 projects the edge of the three-dimensional space detected by the three-dimensional position data segmentation unit 26 based on the projector data from the projector data vertical offset unit 18 by the projector. Two-dimensional data is generated by perspective projection on a surface.
In step S46, the two-dimensional data file generation unit 28 matches the file format specified from the two-dimensional data generated by the perspective projection unit 27 based on the file format parameters stored in the parameter storage unit 3. A data file 32 is generated.

As shown in FIG. 20, in perspective projection, a straight line connecting each point of the point group data of the three-dimensional position data constituting the ridge line in the three-dimensional space and the projector position coordinates indicating the projection center point 102 in the projector data; An intersection point with the projection plane 139 perpendicular to the projection direction vector 136 in the virtually set projector data is obtained. Two-dimensional data indicating the outer position of each surface of the three-dimensional object by converting the projection surface 139 and the projection point of the ridge line on the projection surface 139 into two-dimensional data using the intersection as the projection point and complementing the projection point Is generated.
After the two-dimensional data file 32 is generated, the information processing apparatus takes in the two-dimensional data file 32. Based on the contents of the two-dimensional data file 32, the two-dimensional spatial projection mapping video creation tool allocates video to each surface based on the external position of each surface of the three-dimensional object on the two-dimensional data and creates a projection video. If this projected image is projected from the projector, projection mapping can be performed in a state that matches the outer shape of the three-dimensional object.

According to the projector of the present embodiment described above, the three-dimensional sensor 108 measures a three-dimensional object three-dimensionally, and the three-dimensional position data (position and ridgeline of the projection surface on the three-dimensional object) as the measurement result is the coordinates of the projector. Convert coordinates to system. Thereby, the three-dimensional position data of the three-dimensional object in the projector coordinate system can be obtained.
Further, two-dimensional data indicating the position and outer shape of each projection surface on the three-dimensional object is created using the coordinate-converted three-dimensional position data. A video for two-dimensional projection mapping can be created by assigning a video to each projection surface on the three-dimensional object based on the two-dimensional data. Since the image for two-dimensional projection mapping is created based on the three-dimensional position data of the three-dimensional object in the coordinate system of the projector, when the image is projected from the projector, the range of the projected image is the three-dimensional object. It corresponds to the projection surface of

Further, using the coordinate-converted three-dimensional position data, three-dimensional data indicating the position and outer shape of each projection surface on the three-dimensional object is generated, and a projector including a projection direction vector, an angle of view, and a projector position coordinate Generate data. Based on the three-dimensional data, configure a three-dimensional object in a virtual three-dimensional space, assign and play a video to each surface on the virtual three-dimensional object, set a rendering camera based on the projector data, By performing rendering, a video for three-dimensional projection mapping can be created. Since the image for 3D projection mapping is also created based on the 3D position data of the 3D object in the projector coordinate system, when the image is projected from the projector, the range of the projected image is 3D It corresponds to the projection surface of

Further, in the three-dimensional projection mapping that expresses a three-dimensional image, when setting a rendering camera at the viewpoint position in order to set the illumination and reproduce the light hit condition, etc., the projector is set according to the contents of the projector data file. Set the rendering camera at the viewpoint position and perform rendering. And based on the positional relationship with a solid object, it converts into a projector projection image | video and produces a projection image | video. When this video is projected from the projector, the range of the projected video matches the projection surface of the three-dimensional object.

Further, it is possible to perform distortion correction so that the angle of view of the projector is vertically symmetric and symmetric, and to generate a projector data file including a projection direction vector, an angle of view, and projector position coordinates according to the distortion correction. Therefore, even in a tool for creating a projection mapping image in a three-dimensional space in which only a vertically symmetrical and left / right symmetric angle of view can be set as a rendering camera or projector setting, an image that matches the projection surface of a three-dimensional object is created. be able to.
Furthermore, the data in the world coordinate system based on a horizontal plane parallel to the ground is also used as it is for creating a 3D projection mapping video for a 3D video representation in a tool for creating a 3D space projection mapping video. be able to.

Hereinafter, the operation and effect regarding the view angle symmetrization in the creation of the video for three-dimensional projection mapping will be described in detail.
First, a problem relating to field angle symmetrization will be described with reference to FIGS. 21A to 21F.
As a projection form of the projector, a launch projection as shown in FIG. 21A is common. In this example, the projection optical axis 109 passes through the center of the lower side of the projection area 103. In this case, the left angle θ _L and the right angle θ _R are equal in the horizontal angle of view, but the lower angle θ _B is 0 in the vertical angle of view and does not match the upper angle θ _T.
In a rendering camera in a virtual three-dimensional space, it is common to set the horizontal angle of view to be vertically symmetric and the vertical angle of view to be symmetric. When such a rendering camera is used, as shown in FIG. 21B, the field angle center axis 150 is determined so that the vertical field angle is vertically symmetrical (θ ′ _T = θ ′ _B ), and the projection optical axis 109 Needs to be set to coincide with the angle of view center axis 150.

As illustrated in FIG. 21C, the projection direction, the angle of view (zoom), and the position of the projector 201 are determined so that projection can be performed on the projection target surface 153 that represents one surface of the three-dimensional object. Here, the projection target surface 153 is perpendicular to the ground. Note that the position, angle of view (zoom), and direction of the projector 201 are not changed after this positioning.
An image for three-dimensional projection mapping is created using the three-dimensional data (three-dimensional data of the projection target surface 153) and projector data including a position, an angle of view, and a projection direction vector. Here, the position is the coordinate of the projection center point 102, the angle of view is θ _L θ _R (θ _L = θ _R ), θ ′ _T θ ′ _B (θ ′ _T = θ ′ _B ), and the projection direction vector Is the direction of the view angle central axis 150.

In creating a video for 3D projection mapping, first, projector data is set as a parameter of the rendering camera 148. With this setting, for example, as illustrated in FIG. 21D, the virtual imaging surface 151 of the rendering camera 148 is perpendicular to the optical center axis 152.
Next, an image to be projected on the virtual projection target surface 154 reproduced in the virtual three-dimensional space is allocated based on the three-dimensional data. Then, as shown in FIG. 21E, the video assigned to the virtual projection target surface 154 is captured by the rendering camera 148. The captured image is like a captured virtual projection target surface 155 in the virtual imaging surface 151.

The image created as described above is projected from the projector 201. In this case, for example, as illustrated in FIG. 21F, the projection center axis 110 passing through the center of the projection image 156 projected onto the projection target surface 153 is located above the view angle center axis 150. For this reason, the projection image 156 is shifted upward with respect to the projection target surface 153.
Further, while the virtual imaging surface 151 of the rendering camera 148 is perpendicular to the optical center axis 152, the projection surface of the projection area 103 is inclined with respect to the view angle center axis 150, and the virtual imaging surface 151 and Since the projection surfaces are not parallel, the projected image 156 is distorted. For example, as shown in FIG. 21E, in shooting using the rendering camera 148, between the captured virtual projection target surface 155 and the virtual projection target surface 154, the distance between the upper sides of each other and the lower side of each other. The distance between them is the same. On the other hand, as shown in FIG. 21F, in the projection of the projector 201, the distance between the lower sides of the image 157 and the projection target surface 153 between the lower sides rather than the distance between the upper sides of the projection target surface 153. Is far away, the width of the projected image 156 increases. For this reason, the projected image 156 is distorted. The phenomenon in which this distortion occurs is the same as the phenomenon in which distortion occurs when tilt projection or oblique projection is performed.

According to the projector of this embodiment, generation | occurrence | production of the shift | offset | difference and distortion with respect to the projection target surface of said projection image | video can be suppressed. Hereinafter, the principle will be described with reference to FIGS. 21G to 21K.
As shown in FIG. 21G, the vertical angle of the field angle of the image when the distortion correction is performed so that the image projected on the projection plane perpendicular to the projection center axis becomes square becomes equal (θ " _T = θ" _B ) Specifically, a projection center axis 110 connecting the projection center point 102 and the center of the projected image and a projection plane perpendicular to the projection center axis 110 are obtained, and a distortion correction coefficient is set. The video which has been subjected to distortion correction, to become smaller than the original image, theta _"T and theta" _B is less than the respective theta _'T and θ' _B, θ _"L and theta" _R respectively theta _L and smaller than θ _R. For this reason, the angle of view becomes narrower than in the case where distortion correction is not performed.

Next, as illustrated in FIG. 21H, the projection direction and position of the projector 201 are determined so that the projection can be performed on the projection target surface 153 that indicates one surface of the three-dimensional object. In this example, the projection target surface 153 is perpendicular to the ground. Thereafter, the position, direction, and angle of view (zoom) of the projector 201 are not changed.
An image for three-dimensional projection mapping is created using the three-dimensional data (three-dimensional data of the projection target surface 153) and projector data including a position, an angle of view, and a projection direction vector. Here, the position is the coordinate of the projection center point 102, the angle of view is θ _L θ _R (θ _L = θ _R ), θ ′ _T θ ′ _B (θ ′ _T = θ ′ _B ), and the projection direction vector Is the direction of the projection center axis 110 (= view angle center axis 150).

In creating a video for 3D projection mapping, first, projector data is set as a parameter of the rendering camera 148. With this setting, for example, as illustrated in FIG. 21I, the virtual imaging surface 151 of the rendering camera 148 is perpendicular to the optical center axis 152.
Next, an image to be projected on the virtual projection target surface 154 reproduced in the virtual three-dimensional space is allocated based on the three-dimensional data. Then, as shown in FIG. 21J, an image assigned to the virtual projection target surface 154 is captured by the rendering camera 148. The captured image is like a captured virtual projection target surface 155 in the virtual imaging surface 151.

The image created as described above is projected from the projector 201. In this case, for example, as shown in FIG. 21K, the projection center axis 110 passing through the center of the projection image 156 projected onto the projection target surface 153 coincides with the field angle center axis 150. In addition, while the virtual imaging surface 151 of the rendering camera 148 is perpendicular to the optical center axis 152, the projection surface of the projection area 103 is also perpendicular to the field angle central axis 150, so that the projected video 155 is It coincides with the projection target surface 153 and no distortion occurs.
In the above description, the operation has been described with respect to the example of the launch projection shown in FIG. 21A. However, even when the position of the projected image is shifted vertically and horizontally by the lens shift as shown in FIGS. Problems with image shift and distortion occur. In this case, in addition to vertical distortion correction, horizontal distortion correction is also performed. That is, distortion correction in the vertical direction and the horizontal direction is performed so that the angle of view of the projector 201 is vertically and horizontally symmetrical, and projector data including a projection direction vector, an angle of view, and projector position coordinates is generated.

(Second Embodiment)
FIG. 22 is a block diagram showing a configuration of a projector according to the second embodiment of the present invention.
The projector shown in FIG. 22 includes a user interface unit 34 and an external storage device 35 in place of the communication control unit 1 and the communication input / output unit 2, and is different from the first embodiment in this respect.
The user interface unit 34 is an operation unit that receives an input operation by the user and performs the same control as the communication control unit 1, and includes, for example, an on-screen display and a key input unit.

The external storage device 35 is a removable storage device such as a USB (Universal Serial Bus) memory or an SD card. The projector data file 31, the two-dimensional data file 32, and the three-dimensional data file 33 are supplied from the file storage unit 8 to the external storage device 35. By connecting the external storage device 35 to the information processing apparatus, the information processing apparatus can read the projector data file 31, the two-dimensional data file 32, and the three-dimensional data file 33 from the external storage device 35.
According to the projector of the present embodiment, in addition to the effects described in the first embodiment, it is difficult to perform mutual communication with the information processing apparatus via the communication cable, or wireless communication with the information processing apparatus is difficult. In this case, by using the external storage device 35, the projector data file 31, the two-dimensional data file 32, and the three-dimensional data file 33 can be provided to the information processing apparatus. Here, the communication cable is used to send, for example, the projector data file 31, the two-dimensional data file 32, and the three-dimensional data file 33.

In the first and second embodiments described above, each unit of the projector (projector data generation unit 6, mapping data generation unit 7, angle of view symmetrization unit 10, distortion correction unit 13, distortion correction calculation unit 15, etc.) is supported. A program for causing a computer to execute the processing to be performed may be provided. In this case, a function corresponding to each unit such as the projector data generation unit 6, the mapping data generation unit 7, the view angle symmetrization unit 10, the distortion correction unit 13, and the distortion correction calculation unit 15 by the computer executing the program. Can be realized. Here, the program may be provided in a computer usable or computer readable medium, or may be provided through a network such as the Internet. Here, the computer usable or computer readable medium includes a medium capable of recording or reading information using magnetism, light, electronic, electromagnetic, infrared, or the like. Such media include, for example, semiconductor memory, semiconductor or solid storage devices, magnetic tape, removable computer diskettes, random access memory (RAM), read only memory (ROM), magnetic disk, optical disk, magneto-optical disk, etc. There is.

(Third embodiment)
FIG. 23 is a block diagram showing a configuration of a projector according to the third embodiment of the present invention.
Referring to FIG. 23, the projector includes a display element 400, a projection lens 401, a reception unit 402, and a mapping data generation unit 403.
The display element 400 includes an image forming surface including a plurality of pixels. The projection lens 401 projects an image formed on the image forming surface. An image is projected from the projection lens 401 toward a three-dimensional object.

The accepting unit 402 accepts a data creation request. The mapping data generation unit 403 generates mapping data for creating a projection mapping video in response to a data creation request. The mapping data generation unit 403 includes a three-dimensional sensor unit 404, a coordinate conversion unit 406, and a data generation unit 407.
The three-dimensional sensor unit 404 includes, for example, a three-dimensional sensor 405 that is arranged in a direction in which an image is projected. The three-dimensional sensor unit 404 three-dimensionally measures a three-dimensional object with the three-dimensional sensor 405, and outputs three-dimensional position data that represents the position and shape of each surface on which the image on the three-dimensional object is projected with three-dimensional coordinates. .

The coordinate conversion unit 406 is a projector coordinate system in which the coordinate system of the three-dimensional position data is represented by the three-dimensional coordinates of the image projection area with the point where the principal ray from the pixel located diagonally on the image forming surface intersects as the origin. Convert to
Based on the three-dimensional position data coordinate-converted by the coordinate conversion unit 406, the data generation unit 407 determines the position of the surface and the outer shape of the surface for each surface (surface on which an image is projected) of the three-dimensional object. The ridgeline shown is acquired, and mapping data (two-dimensional data or three-dimensional data of a three-dimensional object) is generated based on the position and the ridgeline.

In the projector of the present embodiment, a zoom mechanism that can move at least some of the plurality of lenses constituting the projection lens 401 in the optical axis direction of the projection lens 401 and a shift direction that is a direction orthogonal to the optical axis direction. The chief ray according to the lens shift mechanism that can shift the projection lens 401, the zoom position that is the position of the at least some of the lenses in the optical axis direction, and the lens shift position that is the position of the projection lens 401 in the shift direction. And a first look-up table showing a change in a position where the crossing points. In this case, the coordinate conversion unit 406 refers to the first look-up table and determines the position where the principal ray intersects from the current zoom position of the zoom mechanism and the current lens shift position of the lens shift mechanism. Also good.

In the projector according to the present embodiment, the coordinate conversion unit 406 may convert the coordinate system of the three-dimensional position data into the projector coordinate system, and then convert the coordinate system into the world coordinate system based on the horizontal plane.
Further, in the projector according to the present embodiment, a second lookup table indicating a change in the angle of view representing the projection range of the image as an angle according to the zoom position and the lens shift position, and a second lookup table are provided. With reference to the current zoom angle of the zoom mechanism and the current lens shift position of the lens shift mechanism, the current angle of view is acquired, and the central ray of the projection light beam from the projection lens 401 is represented based on the angle of view. An angle-of-view symmetrizing unit that sets a projection center axis and outputs angle-of-view symmetrization information including a projection direction vector indicating the direction of the projection center axis and the current angle of view, and the coordinates of the origin of the projector coordinate system A projector data generation unit for generating projector data, including the projector position coordinates, and the projection direction vector and the current angle of view indicated by the angle of view symmetrization information; It may further include a. In this case, the projector data generation unit may convert the direction of the projection vector into a world coordinate system based on the horizontal plane. Furthermore, the data generation unit 407 may generate two-dimensional data by perspectively projecting the ridge line on the projection surface based on the projector data.

In the projector according to the present embodiment, the field angle symmetrizing unit may output projection plane tilt information indicating the tilt of the plane perpendicular to the projection center axis with respect to the plane perpendicular to the optical axis of the projection lens 401. In this case, the projector includes a distortion correction coefficient calculation unit that calculates a distortion correction coefficient for correcting distortion of an image projected on the projection plane based on the projection plane tilt information, and a distortion correction coefficient for the input video signal. A distortion correction unit that performs distortion correction based on the image signal, and an image based on the video signal subjected to the distortion correction may be formed on the image forming surface of the display element 400.
Furthermore, in the projector according to the present embodiment, the data generation unit 407 may create a two-dimensional data file generated by generating two-dimensional data in a predetermined format file format. In this case, the projector may further include output means for outputting a two-dimensional data file.

Furthermore, in the projector according to the present embodiment, the data generation unit 407 may create a three-dimensional data file generated by generating three-dimensional data in a predetermined format file format. Further, the data generation unit 407 may generate three-dimensional data by converting each surface of the three-dimensional object into a polygon mesh. In this case, the projector may further include output means for outputting a three-dimensional data file.
In the projector according to the present embodiment, the data generation unit 403 generates a three-dimensional data file in which three-dimensional data is generated in a predetermined format file format. The projector data generation unit generates the projector data in a predetermined format file. You may further have the output means which produces the projector data file produced | generated in the format, and outputs a three-dimensional data file and a projector data file.
The output means may be a communication means capable of mutual communication with an external information processing apparatus or a removable storage means.

In the projector according to the present embodiment, a mapping data creation method for generating mapping data for generating mapping data is performed in response to a mapping data creation request. In this mapping data creation method, the three-dimensional sensor unit 404 measures a three-dimensional object three-dimensionally using the three-dimensional sensor 405 arranged in the direction in which the image is projected, and the image on the three-dimensional object is projected. 3D position data representing the position and shape of each surface to be expressed in 3D coordinates is output, and the coordinate conversion unit 406 converts the coordinate system of the 3D position data from the pixels located diagonally to the image forming surface. The data generation unit 407 converts the projection area of the image into a projector coordinate system expressed in three-dimensional coordinates with the point where the principal rays intersect as the origin, based on the coordinate-converted three-dimensional position data. For each of the surfaces, obtaining the position of the surface and a ridge line indicating the outer shape of the surface, and generating two-dimensional data or three-dimensional data of the three-dimensional object based on the position and the ridge line.

In the projector of the present embodiment, a program for causing a computer to execute mapping data generation processing for generating mapping data in response to a mapping data creation request may be used. In this case, in the mapping data generation processing, each surface on which the three-dimensional object is three-dimensionally measured using the three-dimensional sensor 405 arranged in the direction in which the image is projected, and the image on the three-dimensional object is projected. The process of outputting the 3D position data representing the position and shape of the image in 3D coordinates, and the coordinate system of the 3D position data at the point where the principal ray from the pixel located diagonally on the image forming surface intersects As a process for converting the projection area of the image into a projector coordinate system expressed in three-dimensional coordinates, and for each surface of the three-dimensional object based on the coordinate-converted three-dimensional position data, the position of the surface and the outer shape of the surface And a process of generating two-dimensional data or three-dimensional data of a three-dimensional object based on the position and the ridge line.

The program may be provided on a computer-usable or computer-readable medium, or may be provided via a network such as the Internet. Here, the computer usable or computer readable medium includes a medium capable of recording or reading information using magnetism, light, electronic, electromagnetic, infrared, or the like. Such media include, for example, semiconductor memory, semiconductor or solid storage devices, magnetic tape, removable computer diskettes, random access memory (RAM), read only memory (ROM), magnetic disk, optical disk, magneto-optical disk, etc. There is.

400 Display element 401 Projection lens 402 Reception unit 403 Mapping data generation unit 404 Three-dimensional sensor unit 406 Coordinate conversion unit 407 Data generation unit

Claims

A display element having an image forming surface composed of a plurality of pixels; and a projection lens that projects an image formed on the image forming surface, and the image is projected from the projection lens toward a three-dimensional object. A projector,
A reception unit for receiving a data creation request;
A mapping data generation unit for generating mapping data for creating a projection mapping video in response to the data creation request;
The mapping data generation unit
A three-dimensional sensor unit that three-dimensionally measures the three-dimensional object and outputs three-dimensional position data representing the position and shape of each surface on which the image on the three-dimensional object is projected in three-dimensional coordinates;
The coordinate system of the three-dimensional position data is converted into a projector coordinate system in which the projection area of the image is expressed in three-dimensional coordinates with a point where a principal ray from a pixel located on the diagonal of the image forming surface intersects as an origin. A coordinate transformation unit;
Based on the three-dimensional position data coordinate-converted by the coordinate conversion unit, for each of the surfaces of the three-dimensional object, a position of the surface and a ridge line indicating the outer shape of the surface are acquired, and the position and the ridge line are obtained. And a data generation unit that generates the mapping data based on the projector.
The projector according to claim 1, wherein
A zoom mechanism capable of moving at least some of the plurality of lenses constituting the projection lens in the optical axis direction of the projection lens;
A lens shift mechanism capable of shifting the projection lens in a shift direction that is a direction orthogonal to the optical axis direction;
A change in position at which the principal rays intersect according to a zoom position that is the position of at least a part of the lens in the optical axis direction and a lens shift position that is the position of the projection lens in the shift direction. A lookup table, and
The coordinate conversion unit refers to the first look-up table and determines a position where the principal ray intersects from a current zoom position of the zoom mechanism and a current lens shift position of the lens shift mechanism. projector.
The projector according to claim 2,
The coordinate conversion unit converts the coordinate system of the three-dimensional position data into the projector coordinate system, and then converts the coordinate system into a world coordinate system based on a horizontal plane.
The projector according to claim 2 or 3,
A second look-up table showing a change in angle of view representing the projection range of the image in angle according to the zoom position and the lens shift position;
With reference to the second look-up table, a current angle of view is obtained from a current zoom position of the zoom mechanism and a current lens shift position of the lens shift mechanism, and the projection is performed based on the angle of view. An angle-of-view symmetrizing unit that sets a projection center axis that represents the central ray of the projection light beam from the lens and outputs angle-of-view symmetrization information including a projection direction vector indicating the direction of the projection center axis and the current angle of view. When,
A projector data generation unit for generating projector data, including projector position coordinates indicating coordinates of an origin of the projector coordinate system, and the projection direction vector and the current angle of view indicated by the angle-of-view symmetrization information; The projector further comprising:
The projector according to claim 4, wherein
The projector data generation unit converts the direction of the projection vector into a world coordinate system with a horizontal plane as a reference.
The projector according to claim 4 or 5,
The data generation unit generates two-dimensional data by perspectively projecting the ridge line on a projection surface based on the projector data.
The projector according to any one of claims 4 to 6,
The angle-of-view symmetrization unit outputs projection plane tilt information indicating a tilt of a plane perpendicular to the projection center axis with respect to a plane perpendicular to the optical axis of the projection lens;
Based on the projection surface tilt information, a distortion correction coefficient calculation unit that calculates a distortion correction coefficient for correcting distortion of an image projected on the projection surface;
A distortion correction unit that performs distortion correction based on the distortion correction coefficient for the input video signal,
A projector in which an image based on the video signal subjected to the distortion correction is formed on the image forming surface.
The projector according to any one of claims 1 to 7,
The data generation unit creates a two-dimensional data file generated by generating the two-dimensional data in a file format of a predetermined format.
The projector according to claim 8, wherein
A projector further comprising output means for outputting the two-dimensional data file.
The projector according to any one of claims 1 to 7,
The data generation unit is a projector that generates three-dimensional data by forming each surface of the three-dimensional object into a polygon mesh.
The projector according to any one of claims 1 to 7, wherein
The data generation unit creates a three-dimensional data file in which the three-dimensional data is generated in a file format of a predetermined format.
The projector according to claim 11, wherein
A projector further comprising output means for outputting the three-dimensional data file.
The projector according to any one of claims 4 to 7,
The data generation unit creates a three-dimensional data file in which the three-dimensional data is generated in a file format of a predetermined format,
The projector data generation unit creates a projector data file that generates the projector data in a predetermined format file format,
A projector further comprising output means for outputting the three-dimensional data file and the projector data file.
The projector according to any one of claims 9, 12, and 13,
The projector is a communication unit capable of mutual communication with an external information processing apparatus or a removable storage unit.
A display element having an image forming surface composed of a plurality of pixels; and a projection lens that projects an image formed on the image forming surface, and the image is projected from the projection lens toward a three-dimensional object. A mapping data creation method performed by a projector,
A mapping data generation step for generating mapping data for creating a projection mapping video in response to the data creation request;
The mapping data generation step includes:
Three-dimensional measurement of the three-dimensional object, to obtain three-dimensional position data representing the position and shape of each surface on which the image on the three-dimensional object is projected in three-dimensional coordinates,
The coordinate system of the three-dimensional position data is converted into a projector coordinate system in which the projection area of the image is expressed in three-dimensional coordinates with a point where a principal ray from a pixel located diagonally on the image forming surface intersects as an origin. ,
Based on the coordinate-converted three-dimensional position data, for each of the surfaces of the three-dimensional object, a position of the surface and a ridge line indicating the outer shape of the surface are acquired, and the mapping is performed based on the position and the ridge line. Generating data for mapping, including generating data.
A display element having an image forming surface composed of a plurality of pixels; and a projection lens that projects an image formed on the image forming surface, and the image is projected from the projection lens toward a three-dimensional object. A program for causing a projector computer to execute mapping data generation processing for generating mapping data for generating a projection mapping video in response to a data generation request,
The mapping data generation process includes:
Processing to measure the three-dimensional object three-dimensionally and obtain three-dimensional position data representing the position and shape of each surface on which the image on the three-dimensional object is projected in three-dimensional coordinates;
The coordinate system of the three-dimensional position data is converted into a projector coordinate system in which the projection area of the image is expressed in three-dimensional coordinates with a point where a principal ray from a pixel located on the diagonal of the image forming surface intersects as an origin. Processing,
Based on the coordinate-converted three-dimensional position data, for each of the surfaces of the three-dimensional object, a position of the surface and a ridge line indicating the outer shape of the surface are acquired, and the mapping is performed based on the position and the ridge line. And a program for generating data.
A projector according to any one of claims 1 to 14,
An image processing device capable of mutual communication with the projector,
The video processing apparatus is a projection mapping system that creates a projection mapping video based on data for a projection mapping video creation tool generated by the projector.