JP2017026652A - Screen, position specification device, video processing device, pattern generation device, and method for manufacturing screen - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a screen capable of executing specification of a position of the screen at high speed.SOLUTION: A screen is provided with: a recursive reflective sheet 13; and a cut sheet 11 which covers the recursive reflective sheet, and in which a plurality of through holes 12 generated on the basis of an address code showing a position at which a grating pattern is formed, each corresponding to a plurality of points constituting the grating pattern, are formed. The grating pattern is arranged at a position in the cut sheet 11 so that respective points of the grating pattern are shifted from corresponding points of a reference pattern to be constituted of a plurality of points, a value obtained by digitizing a direction of shift from a corresponding point of the reference pattern to a corresponding point constituting the pattern is considered as a point code, and the address code is a number sequence constituted of point codes of the plurality of points constituting the grating pattern.SELECTED DRAWING: Figure 8

Description

  The present embodiment relates to a screen, a position specifying device, a video processing device, a pattern generation device, and a screen manufacturing method.

  There is a technique for projecting an image on a screen having a three-dimensional shape such as a building, an object, or a space. This technique is called, for example, projection mapping. When the screen is fixed, known screen shape information can be used to project the image without distortion. Further, as a method of projecting onto a screen having an unknown shape, there is a method of grasping the shape of the screen by projecting a marker, a pattern or the like onto the screen.

  In addition, Patent Document 1 below discloses a video projection system that projects an image on a stereoscopic screen provided on a moving body as well as a fixed object such as a building. In the technique described in Patent Document 1, a plurality of markers are provided on a projection object to be projected onto a stereoscopic screen, and an image including the markers is captured. Then, the position and direction information of the stereoscopic screen is acquired using one coordinate of the marker in the captured video.

JP 2011-254411 A

  However, when the screen moves, the conventional method of projecting a marker or coordinate system onto the screen projects the marker on the screen, grasps the position and shape of the screen based on the projected image, and then the image Will be projected. For this reason, when the moving speed of the screen is high, the processing time becomes slower than the moving speed, and there is a problem that the processing may not be in time.

  Further, the technique described in Patent Document 1 assumes that a moving body having a certain shape is known, and uses a plurality of markers to match projected video data with a screen. The technique described in Patent Document 1 assumes that a certain shape moves as it is, and has a problem that it cannot cope with a local shape change. Further, there is a problem that the position of each marker cannot be obtained when an undetectable marker occurs.

  This invention is made in view of the above, Comprising: Obtaining the screen which can identify the position of a screen at high speed, a position specification apparatus, a video processing apparatus, a pattern generation apparatus, and the manufacturing method of a screen Objective.

  In order to solve the above-described problems and achieve the object, a screen according to the present invention is formed on the basis of a reflecting material and an address code that covers the reflecting material and indicates a position where the pattern is formed. A plurality of through-holes corresponding to the plurality of points, and the pattern has a reference pattern composed of a plurality of points. A value obtained by quantifying the direction of deviation of the reference pattern corresponding to the reference pattern corresponding to the point constituting the pattern, which is arranged at a position within the light shielding material that is shifted from the corresponding point, is a point code, and the address code is The numerical sequence is composed of the point codes of the plurality of points constituting the pattern.

  According to the present invention, it is possible to specify the position of the screen at high speed.

FIG. 1 is a diagram showing a configuration example of a screen system according to the present invention. FIG. 2 is a diagram illustrating an application example of the screen system according to the embodiment. FIG. 3 is a diagram illustrating a configuration example of a computer system that functions as a video processing apparatus. FIG. 4 is a diagram illustrating a functional configuration example of the video processing apparatus. FIG. 5 is a diagram illustrating a configuration example of the pattern generation apparatus. FIG. 6 is a diagram illustrating an example of a reference lattice pattern. FIG. 7 is a diagram showing an example of a lattice pattern given an address code. FIG. 8 is a diagram illustrating a configuration example of the screen. FIG. 9 is a diagram illustrating an example of a method for manufacturing a screen. FIG. 10 is a diagram illustrating an example in which an image following the screen is projected for each screen. FIG. 11 is a diagram illustrating an example of address code correspondence data. FIG. 12 is a diagram illustrating an example in which a lattice pattern in which different address codes are embedded for each screen 1 is formed. FIG. 13 is a diagram showing the movement, rotation, and inclination of the screen. FIG. 14 is a diagram illustrating a configuration example of the screen device. FIG. 15 is a diagram illustrating an example of a lattice pattern formed on each screen in the screen device. FIG. 16 is a flowchart illustrating an example of the position specifying process and the video projecting process.

  Hereinafter, a screen, a position specifying device, a video processing device, a pattern generation device, and a screen manufacturing method according to embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments.

  FIG. 1 is a diagram showing a configuration example of a screen system according to the present invention. As shown in FIG. 1, the screen system of the present embodiment includes a screen 1 (member), a video processing device 2, a projection device 3, and a photographing device 4.

  FIG. 2 is a diagram showing an application example of the screen system of the present embodiment. In FIG. 2, one screen 1 is shown, but a plurality of screens 1 may be provided. Moreover, you may use the screen apparatus which has the some screen 1 so that it may mention later. The video processing apparatus 2 is a computer system such as a personal computer. The projection device 3 is called a projector or the like, and is a device that magnifies and projects an image. The photographing device 4 is, for example, a digital camera or the like, and is a device that can output a photographed video as video data. There are no restrictions on the method of the projection device 3 and the photographing device 4, and any method may be used. In addition, a light source (not shown) may be arranged separately from the photographing device 4 so as to face the screen 1.

  FIG. 3 is a diagram illustrating a configuration example of a computer system that functions as the video processing apparatus 2 according to the present embodiment. This computer system includes, for example, a control unit 101, an input unit 102, a storage unit 103, a display unit 104, and a communication unit 105, which are connected via a system bus 106.

  The control unit 101 executes a position specifying program and a video projection program according to the present invention. The input unit 102 includes, for example, a keyboard and a mouse, and is used by a computer system user to input various information. The storage unit 103 includes various memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory) and a storage device such as a hard disk. The storage unit 103 is a program to be executed by the control unit 101 and necessary information obtained in the process. Memorize data, etc. The storage unit 103 is also used as a temporary storage area that is temporarily used by the program. The display unit 104 is composed of a CRT, LCD (liquid crystal display panel), etc., and displays various screens for the computer system user.

  Here, an example of the operation of the computer system until the position specifying program and the video projection program according to the present invention are executable will be described. In the computer system having the above-described configuration, for example, a position specifying program and a video projection program are installed in the storage unit 103 from a CD-ROM set in a CD-ROM drive (not shown). Then, when the position specifying program and the video projection program are executed, the position specifying program and the video projection program read from the storage unit 103 are stored in predetermined locations in the storage unit 103. In this state, the control unit 101 executes position specifying processing and video projection processing according to a program stored in the storage unit 103.

  Here, the above-described program is provided using a CD-ROM as a recording medium. However, the present invention is not limited to this. For example, a floppy (registered trademark) disk may be used depending on the configuration of the computer system, the capacity of the provided program, and the like. It is also possible to use a recording medium such as a magnetic disk, a magneto-optical disk, or a magnetic tape. Moreover, it is good also as using the program provided by transmission media, such as an email and the internet.

  FIG. 4 is a diagram illustrating a functional configuration example of the video processing device 2 according to the present embodiment. The video processing device 2 includes a coordinate calculation unit 21, a lattice pattern estimation unit 22, an address determination unit 23, a coordinate conversion matrix determination unit 24, and a video data conversion unit 25. The coordinate calculation unit 21, the lattice pattern estimation unit 22, and the address determination unit 23 are processing units generated in the control unit 101 by the position specifying program. The coordinate transformation matrix determination unit 24 and the video data conversion unit 25 are processing units generated in the control unit 101 by the video projection program. Further, the video processing device 2 holds the address code correspondence data 26 and the video data 27 in the storage unit 103 of FIG.

  In the present embodiment, the video processing device 2 has a function as a position specifying device that executes the position specifying program and a function as a video projection program execution device that executes the video projection program. The specific program and the video projection program may be executed by different apparatuses. When the position specifying program and the video projection program are executed by different apparatuses, the position specifying apparatus that executes the position specifying program includes a coordinate calculation unit 21, a lattice pattern estimation unit 22, and an address determination unit 23, and includes address code correspondence data 26. Hold. The video projection program execution device includes a coordinate transformation matrix determination unit 24 and a video data conversion unit 25 and holds video data 27.

  FIG. 5 is a diagram illustrating a configuration example of the pattern generation device 6 according to the present embodiment. The pattern generation device 6 according to the present embodiment is a device that executes a lattice pattern generation program, and is a device that generates data of a lattice pattern (described later) formed on the screen 1. The pattern generation device 6 may be a separate device independent of the screen system of the present embodiment, or may be a component of the screen system. The pattern generation device 6 is mounted on a computer system similar to the computer system shown in FIG. 3 on which the video processing device 2 is mounted. For example, one computer system may have a function as the video processing device 2 and a function as the pattern generation device 6. An example of the operation of the computer system until the lattice pattern generation program is executable is the same as the above-described position specifying program.

  The pattern generation device 6 according to the present embodiment includes an address code arrangement unit 61 and a formation data generation unit 62. The address code arrangement unit 61 and the formation data generation unit 62 are processing units generated in the control unit 101 by a lattice pattern generation program. The pattern generation device 6 holds the address code correspondence data 63 in the storage unit 103. The address code correspondence data 63 includes an address code and information corresponding to the address code. The address code correspondence data 63 is data indicating correspondence between an address code and information indicating video data to be projected on the screen 1 in which the address code is embedded, for example, depending on the use of the screen 1.

  Next, the operation of the present embodiment will be described. In the present embodiment, a lattice pattern in which an address code is embedded is formed on the screen 1. The address code is information (sequence) indicating the position of the lattice pattern in the reference coordinate system as will be described later. In the present embodiment, one grid pattern is formed for one screen 1. The screen 1 is identified by the address code (several sequence) of each lattice pattern. Any method may be used for forming the lattice pattern on the screen 1, for example, a method of applying a substance that becomes a point of the lattice pattern to the screen 1 by spraying, printing, other than the point of the lattice pattern A method of exposing the points of the lattice pattern by masking can be used. Details of the method of forming the lattice pattern on the screen 1 will be described later.

  First, the address code arrangement unit 61 of the pattern generation device 6 determines an address code to be embedded in a lattice pattern formed on the screen 1 for each screen 1, that is, for each video data projected on the screen 1. In the present embodiment, one video data is associated with one screen 1. The address code arrangement unit 61 holds the correspondence between the information indicating the video data (file name and the like) and the address code corresponding to the video data as the address code correspondence data 63. By determining an address code for each screen 1, a lattice pattern in which an address code is embedded for each video data indicates a lattice pattern in which address information is included in the position of the lattice pattern itself. Here, the lattice pattern in which the address code of the present embodiment is embedded will be described. The reference coordinate system is a coordinate system on the screen 1 when the screen 1 is placed in an initial state without movement or distortion.

  FIG. 6 is a diagram illustrating an example of a reference lattice pattern (reference pattern). Here, an example will be described in which a 3 × 3 square lattice pattern is used as a lattice pattern in which an address code is embedded. For nine 3 × 3 points in FIG. 6, the index k is set as shown in FIG. 6 (the numbers shown at the upper right of each point in FIG. 6 are the values of k at each point).

The coordinates of the point corresponding to each index k are [u (k), v (k)] ^T. The address code is a numerical sequence indicating a code set for each of 9 points of 3 × 3. The code (point code) p (k) corresponding to each point is an integer from 0 to 3. At this time, the coordinates [u ′ (k), v ′ (k)] ^T of the point shifted in accordance with the code are given by the following equation (1).

Here, D is a width for shifting the position and is a constant. Here, D is a width of 1/10 of the lattice point interval. 1/10 is an example, D is not limited to 1/10, and may be a numerical value smaller than half of the lattice point interval. For example, when [0, 1, 2, 3, 0, 1, 2, 3, 0] is given as the address code p, the lattice pattern is as shown in FIG. FIG. 7 is a diagram showing an example of a lattice pattern given an address code. In FIG. 7, black circles indicate the coordinates ([u (k), v (k)] ^T ) of each point of the reference pattern shown in FIG. 6, and × is a lattice in which each point is shifted according to the address code. The coordinates of the point ([u ′ (k), v ′ (k)] ^T ). This x is a lattice pattern in which an address code is embedded.

  In the present embodiment, by generating the address code for each screen 1, if the address code of the lattice pattern can be decoded, the position of the screen 1, that is, the change from the initial state can be known. As an example of the shape of the screen 1, a rectangular shape as shown as the shape 200 in FIGS. 6 and 7 can be used. However, the shape of the screen 1 is not limited to the example of FIGS. Such a shape may be used.

  The formation data generation unit 62 generates a lattice pattern according to the above-described equation (1) based on the address code determined by the address code arrangement unit 61 for each screen 1. The formation data generation unit 62 outputs the generated lattice pattern as formation data. The output format of the formation data may be any format and may be electronic data indicating the arrangement of the lattice pattern on the screen 1, or the arrangement of the lattice pattern on the screen 1 may be printed on paper or the like. May be. If the formation data generation unit 62 has a function of forming a lattice pattern on the screen 1, the formation data generation unit 62 does not output formation data but forms formation data on the screen 1. May be. For example, when forming a grid pattern by printing, the formation data generation unit 62 may have a printing function and form the grid pattern on the screen 1 by printing.

  When the formation data generation unit 62 outputs the formation data, the formation data is input to the forming apparatus that forms the lattice pattern on the screen 1 through a communication path or via a medium or the like. The forming apparatus forms a lattice pattern on the screen 1 based on the formation data. The formation data may be output from the formation data generation unit 62 using paper, and a lattice pattern may be formed on the screen 1 by manual or automatic machining based on the paper.

  The screen 1 is composed of, for example, a base material on a flat plate and a cloth that is a pattern forming member on which a lattice pattern is formed. As the substrate, for example, a cloth, a resin sheet, a resin plate, or the like can be used, but not limited thereto, any material may be used. The pattern forming unit is, for example, a cloth subjected to coding or the like, and each point constituting the lattice pattern is formed on the cloth. Any method may be used as a method for forming a lattice pattern on the screen 1. For example, a material that reflects only light in a specific direction or light other than visible light (for example, infrared light) is emitted or reflected. A method of printing the material or the like to be used as a lattice pattern on the fabric, that is, applying the lattice pattern to the fabric is conceivable. For example, when a material that reflects only light in a specific direction is used, a light source that emits light in a direction that coincides with the line-of-sight direction of the image capturing device 4 is used, and the lattice pattern is arranged to be captured from the image capturing device 4. As a material that reflects only light in a specific direction, for example, a retroreflecting material can be used. In addition, when a material that emits or reflects light other than visible light (for example, infrared light) is used as the lattice pattern, any material may be used, but the material constituting the lattice pattern, other regions (clothes) The wavelength at which a sufficient difference in sensitivity is obtained in the photographing device 4 is applied.

  As described above, as a method for applying a lattice pattern, for example, there are the following methods. First, each point of the lattice pattern, that is, the marker portion is cut out from the mask sheet by a cutting plotter or the like to form a hole corresponding to the marker portion, thereby generating a perforated sheet. And this perforated sheet | seat is bonded on a pattern formation member. Then, ink is applied from above the perforated sheet. Thereby, ink is applied to the marker portion. Since the pattern forming member other than the marker portion is covered with the perforated sheet, ink is not applied. After the ink application, the perforated sheet is peeled off. Then, the pattern formation member which apply | coated the ink is bonded to a base material, and it is set as the screen 1. FIG.

  Also, using a material in which the retroreflective material is formed on the entire surface as a fabric, each point constituting the lattice pattern, that is, an area other than the marker part is applied with another printing paint, and the retroreflective material other than the marker part is hidden ( For example, a method of applying a white paint with a titanium oxide paint is also conceivable. Thus, when concealing other than the marker portion, specifically, for example, the following method can be considered. White ink is applied to a region other than the marker portion on the cloth having the retroreflective material formed on the entire surface by an inkjet printer or the like. That is, the white ink is printed so that the marker portion is left on the cloth on which the retroreflective material is formed on the entire surface. Then, the fabric coated with the white ink is bonded to the base material to obtain the screen 1.

  As described above, various methods can be considered as the method for forming the screen 1. In this embodiment, however, the difference in luminance due to the reflection between the points constituting the lattice pattern, that is, the markers and the other areas is increased. The following forming method will be described as an example of a forming method that can be used. In the method described below, since the difference in luminance between the points constituting the grid pattern and the other regions can be increased, the photographing apparatus 4 can reduce the size of the points compared to the other methods described above. The visibility of the point can also be improved. In the ink coating method using the perforated sheet described above, at present, the reflection characteristics of the ink of the retroreflective material itself are low, and it is difficult to increase the difference in luminance between the points constituting the lattice pattern and the other regions. . In addition, in the method in which the above-described retroreflective material is formed on the entire surface and the region other than the marker portion is applied with a white paint or the like, the white paint or the like itself is reflected, so the reflection characteristics of the retroreflective material to be used Depending on the situation, it may be difficult to increase the difference in reflected luminance between the points constituting the lattice pattern and the other areas.

  FIG. 8 is a diagram illustrating a configuration example of the screen 1 according to the present embodiment. As shown in FIG. 8, the screen 1 of this embodiment is a cut sheet 11 that is a light shielding material provided with a through-hole 12 formed by cutting out marker portions constituting a lattice pattern, that is, a cutout portion, and a reflective material. What pasted the retroreflection sheet 13 is used as a pattern formation member, and the pattern formation member and substrate 14 are pasted. That is, the cut sheet 11 covers the retroreflective sheet 13 and is formed with a plurality of through holes corresponding to a plurality of points that are generated based on an address code indicating a position where the lattice pattern is formed and constitute the lattice pattern. ing. In the present embodiment, since one grid pattern is formed on one screen 1, the address code indicates the position of the screen 1. The retroreflective sheet 13 is a sheet on which a retroreflective material is formed. Any material can be used for the cut sheet 11 as long as it can be processed with a cutting plotter. By making the surface of the cut sheet 11 matt, for example, reflection on the surface of the cut sheet 11 becomes diffuse reflection, and as a result, It is assumed that the reflection component toward the photographing apparatus 4 is reduced. Therefore, the difference between the luminance due to reflection from the retroreflective sheet 13 and the luminance reflected from the cut sheet 11 when photographing from the photographing device 4 is compared with a method of applying a region other than the marker portion with white paint or the like. It is conceivable that the ease of pattern recognition by the photographing apparatus 4 increases. The base material 14 is used for maintaining the flatness of the surface of the screen 1, but is not essential, and is a light shielding material provided with through holes 12 formed by cutting out marker portions constituting the lattice pattern. The cut sheet 11 and the retroreflective sheet 13 may constitute the screen 1. According to this method, since the retroreflective sheet used as the reflective material is not limited to the ink jet printer compatible sheet, there is an advantage that the degree of freedom in sheet selection is increased. In addition, if a difference in reflection luminance from the surface of the cut sheet, which is a light shielding material, can be obtained, a diffusion sheet (a sheet having a diffuse reflection material formed on the surface) can be used as the reflection material instead of the retroreflection sheet.

  FIG. 9 is a diagram showing an example of a method for manufacturing the screen 1 of the present embodiment. In FIG. 9, the cross section of each member is shown. As shown in FIG. 9A, a cut sheet 11 before the through hole 12 is formed by a cutting plotter is prepared. Next, as shown in FIG. 9B, the through hole 12 is formed by the cutting plotter at the marker position based on the formation data described above, and the cut sheet 11 having the through hole 12 is made the size of the base material 14. Cut together. Next, as shown in FIG.9 (c), the retroreflection sheet 13 cut according to the size of the base material 14 and the cut sheet 11 in which the through-hole 12 was formed are bonded. Next, as shown in FIG.9 (d), the base material 14 is bonded to the surface by which the cut sheet 11 of the retroreflection sheet 13 is not bonded.

  The procedure in FIG. 9 is an example, and the bonding in FIGS. 9C and 9D is performed after the retroreflective sheet 13 and the base material 14 are bonded, You may bond the cut sheet 11 in which the through-hole 12 was formed. Moreover, the process of cutting the cut sheet 11 and the retroreflective sheet 13 before the through hole 12 is formed in accordance with the size of the base material 14 may be performed at any stage. For example, after the retroreflective sheet 13 is cut in accordance with the size of the base material 14, FIG. 9A may be performed, or after performing the FIG. 9A, the retroreflective sheet 13 is used as the base material. 14 may be cut according to the size of 14, or the retroreflective sheet 13 may be cut along with the size of the base material 14 at the same time. Moreover, you may cut the retroreflection sheet 13 and the cut sheet 11 according to the size of the base material 14 after the bonding of FIG.

  In the above example, one grid pattern is formed on one screen 1, but a plurality of grid patterns may be formed on one screen 1. Also in this case, as described with reference to FIGS. 8 and 9, the screen 1 can be configured by the base material 14, the retroreflective sheet 13, and the cut sheet 11 in which the through holes 12 are formed.

  With the procedure described above, the screen 1 on which the lattice pattern in which the address code is embedded is formed. The screen 1 is photographed by the photographing device 4 to acquire an image, and an address code corresponding to the screen 1 can be obtained by analyzing the acquired image. Information such as the position and rotation angle of the screen 1 can be obtained by using the correspondence between the screen 1 and the address code thus obtained. Such information can be used for various purposes. For example, as a first application, an image that follows the screen 1 is projected for each screen 1. If an image to be projected onto the screen 1 is determined for each screen 1 and the image is projected onto the screen 1 using the position and rotation angle calculated using the address code of the screen 1, the screen 1 An image following the position and the rotation angle is projected.

  FIG. 10 is a diagram illustrating an example in which an image following the screen is projected for each screen. In FIG. 10, as an example of the first application, for example, an example of projecting a playing card image is shown. When projecting a card image, 54 types of video data including two types of jokers are generated as video data on the surface of the card, and one type of video data on the back side of the card is generated. These video data are stored in advance in the storage unit of the video processing device 2 (for example, the storage unit 103 in the configuration example of FIG. 3). In the case of the first application, the video data 27 of the video processing device 2 shown in FIG. 4 is a set of video data for each of these cards. In addition, 54 screens 1 are generated. The 54 screens 1 are formed with a lattice pattern in which different address codes are embedded for each screen 1. Then, the correspondence between the address code and the video data is held as address code correspondence data 63. FIG. 11 is a diagram illustrating an example of the address code correspondence data 63. As shown in FIG. 11, the address code corresponding data 63 includes an address code and information indicating video data. In the first application, A, B, and C in FIG. 11 indicate, for example, file names (data names) of video data. In the example of playing cards, A is video data of an ace card of spades, and B is , The video data of the second card of the spade, and so on, indicating one of the 54 types of video data of the playing card. FIG. 12 is a diagram illustrating an example in which a lattice pattern in which different address codes are embedded for each screen 1 is formed. FIG. 12 shows an example of a lattice pattern embedded in the screens 301 to 308 in order to project the images of the playing card shown in FIG. FIG. 12 corresponds to the state of the screens 301 to 308 in FIG. 10, the back surfaces of the screens 301, 303, and 304 face the imaging device 4 side, and the front surfaces of the screens 302, 305 to 308 are the imaging device. It faces the 4th side. A lattice pattern in which different address codes are embedded in each screen is formed on the front surface, and a lattice pattern in which the same address code is embedded is formed on the back surface.

  For example, the cut sheet 11 in which the retroreflective sheet 13 and the through holes 12 are formed is bonded to the back surface of the base material 14 of the screen 1 shown in FIG. 8, that is, the surface that is not bonded to the retroreflective sheet 13. Thus, a lattice pattern may be formed on the back surface. In the address code correspondence data 63, the address code embedded in the lattice pattern is associated with video data on the back side of the playing card. A screen having a lattice pattern formed on the back surface in this way is called a double-sided screen. Alternatively, instead of the double-sided screen, the two screens 1 may be bonded to each other so that the lattice pattern faces outward.

  After that, as shown in FIG. 2, one or more of the 54 screens 1 are photographed by the photographing device 4 whose viewing direction is substantially the same as that of the projection device 3, and an image is acquired. The video processing device 2 analyzes the image acquired by the projection device 3 to calculate an address code and a projective transformation matrix corresponding to the screen 1 for each screen 1 as described later. The video processing device 2 reads the information (file name, etc.) indicating the video data corresponding to the address code with reference to the address code corresponding data 63 for each screen 1, that is, for each calculated address code. Further, the video processing device 2 calculates the position of the screen 1 corresponding to the calculated address code. The position of the screen 1 is calculated, for example, as a coordinate value in a three-dimensional coordinate system with a predetermined point as the origin. This coordinate value is a coordinate value of a reference point defined in a coordinate system fixed to each screen 1. The reference point may be, for example, the coordinate value of the geometric center or may be the coordinate value of one predetermined point among the four corners, and there is no restriction on the position of the reference point. Next, the video processing device 2 reads video data corresponding to the information, performs conversion using the projective transformation matrix on the read video data, and projects the converted video data on the calculated position.

  For example, in the example of FIG. 10, eight screens 301 to 308 are photographed by the photographing device 4. Screens 301 to 308 are the double-sided forming screens of the above-described embodiment. Of the screens 301 to 308, the back surfaces of the screens 301, 303, and 304 are photographed by the photographing device 4, and the front surfaces of the screens 302, 305 to 308 are photographed by the photographing device 4. FIG. 13 is a diagram illustrating movement, rotation, and inclination of the screen 301. FIG. 13 shows a cross section of the screen 301. Similarly to the screen 301, the screens 302 to 308 can be individually moved, rotated, and tilted.

  Based on the image photographed by the photographing device 4, the video processing device 2 calculates address codes corresponding to the screens 301 to 308, and calculates the positions of the screens 301 to 308, respectively. As for the screens 302 and 305 to 308, the address code corresponding to the screens 302 and 305 to 308 corresponds to the video data on the surface of the playing card. Then, the video data corresponding to each address code on the surface of the playing card is converted using a projective transformation matrix, and the converted video data is calculated by the projection device 3 on the screens 302, 305 to 308, respectively. Project to. As a result, as shown in FIG. 10, video data of the surface of the playing card is projected onto the screens 302, 305 to 308. Further, the video data corresponding to the screens 302 and 305 to 308 is subjected to conversion using projection transformation matrices corresponding to the screens 302 and 305 to 308, respectively. The video data corresponding to is projected.

  For the screens 301, 303, and 304, the address code corresponding to the screens 301, 303, and 304 corresponds to the video data on the back side of the playing card. Data is converted using a projective transformation matrix, and the converted video data is projected onto the calculated positions on the screens 301, 303, and 304 by the projection device 3. As a result, as shown in FIG. 10, the video data of the back side of the playing card is projected onto the screens 301, 303, and 304. In addition, the video data corresponding to the screens 301, 303, and 304 is subjected to conversion using the projective transformation matrix corresponding to each of the screens 301, 303, and 304, thereby rotating and tilting the screens 301, 303, and 304. The video data corresponding to is projected. In this example, the double-sided screen is described as an example. However, even when a lattice pattern is formed on one side shown in FIG. 8, video data corresponding to the rotation and inclination of the screen can be similarly projected. . In this case, in the example shown in FIG. 10, the image data on the front surface of the playing card is projected and the image data on the back surface of the playing card is not performed.

  Next, the 2nd use of the screen 1 of this Embodiment is demonstrated. In the second application, for example, a screen device including a plurality of screens 1 of the present embodiment is used. FIG. 14 is a diagram illustrating a configuration example of the screen device. The right diagram and the left diagram in FIG. 14 respectively show different states of the same screen device. As shown in FIG. 14, the screen device includes a plurality of screens 402 that are the screens 1 of the present embodiment, and a frame 401 is formed outside the plurality of screens 402. The plurality of screens 402 are connected by a connection member 403. The connecting member 403 connects the screen 402 to the frame so that the screen 402 can rotate in three dimensions, that is, in the X, Y, and Z directions. The connection member 403 is an elastic body, for example. When such a screen device is arranged along a solid body, for example, the screen 402 rotates along the surface of the solid body. Assume that each screen 402 is formed with a lattice pattern in which different address codes are embedded. The image capturing device 4 captures an image by capturing the screen device, and the video processing device 2 uses this image to calculate an address code in the same manner as in the first application, and the position of each screen 402 and projective conversion. By calculating the matrix and projecting the video data on each screen 402 using the calculated position and the projective transformation matrix, the video data corresponding to the movement and deformation of the solid can be projected.

  In the second application as well, as in the first application, video data to be projected for each address code is generated and stored in the video processing apparatus 2. By projecting the video data corresponding to each screen 402 if the entire video data for projecting this video data onto a plurality of screens 402 constituting the screen device is divided for each screen 402, the screen device In addition, the entire video data can be projected.

  FIG. 15 is a diagram illustrating an example of a lattice pattern formed on each screen in the screen device. FIG. 15 illustrates an example in which the screen device illustrated in FIG. 14 includes 48 screens 402. FIG. 15 shows an example in which 48 types of address codes 1 to 48 are assigned to 48 screens, and a lattice pattern corresponding to each address code is formed.

  Further, as another example of projecting video data in accordance with the movement and deformation of a solid, an example in which the screen 1 of the present embodiment is pasted at a plurality of locations of the solid can be considered. In this example, a lattice pattern in which different address codes are embedded for each screen 1 is formed, and in the same manner as in the example using the above screen device, the photographing device 4 acquires an image including a plurality of screens 1, The video processing apparatus 2 uses this image to calculate an address code as in the first application, calculates the position of each screen 1 and a projective transformation matrix, and uses the calculated position and the projective transformation matrix to generate a video. By projecting the data onto each screen 1, it is possible to project video data corresponding to the movement and deformation of the three-dimensional object.

  Next, position specifying processing and video projection processing using the screen 1 of the present embodiment will be described. FIG. 16 is a flowchart illustrating an example of the position specifying process and the video projecting process according to the present embodiment. The position specifying process is a process realized by the position specifying program, and the video projecting process is a process realized by the video projecting program, and is a process common to the first application and the second application described above. The video processing device 2 performs the following processing for each screen, that is, for each address code.

  As shown in FIG. 16, first, the coordinate calculation unit 21 acquires image data obtained by photographing the screen 1 (the screen 1 on which the lattice pattern in which the address code is embedded) is photographed from the photographing device 4 (step S1). In the present embodiment, as described above, a lattice pattern in which different address codes are embedded for each screen 1 is formed on each screen 1. The imaging device 4 acquires image data including these lattice patterns by imaging an area including one or more screens 1. In step S <b> 1, the coordinate calculation unit 21 acquires this image data from the imaging device 4.

  Next, the grid pattern estimation unit 22 calculates the coordinates of the grid point image data using the image data (step S2). Specifically, the coordinates of 9 grid points of 3 × 3 formed in the image data are obtained.

  Next, the lattice pattern estimation unit 22 estimates the address code of the nine lattice patterns based on the coordinates of a plurality of (here, nine lattice points) in the image data (step S3). Specifically, the address code is estimated as follows.

If the coordinates of lattice points in the image data are [u _cam (k), v _cam (k)] ^T , [u _cam (k), v _cam (k)] ^T is expressed by the following equation (2). Thus, it is observed that the projection transformation matrix H (bold) is applied to [u ′ (k), v ′ (k)] ^T. Note that s is a variable for determining the scale.

In this _{case, [u cam (k),} v cam (k)] T and [u'(k), v'( k)] If ^T set of four or more of, estimating the projective transformation matrix H (in bold) can do. However, [u ′ (k), v ′ (k)] ^T can be calculated only when the address code is known. Therefore, [u'(k), v' (k)] instead of ^{T [u (k), v} (k)] assuming large error does not occur even with a ^T, [u _cam (k) , V _cam (k)] ^T and [u (k), v (k)] ^T are used to estimate the projective transformation matrix H (bold) ′ by the least square method.

  Specifically, for example, the projective transformation matrix H (bold) ′ is estimated as follows. First, it is assumed that H (bold) ′ is expressed by the following formula (3). The ninth element of H (bold) ′ is 1 because the scale is arbitrary. As shown in Expression (3), the projective transformation matrix H (bold) ′ can be expressed by eight parameters.

N (N ≧ 4) sets of _{[u cam (k), v} cam (k)] T and [u'(k), v'( k)] combination of ^{T (k = 1,2, ...,} N And N ≧ 4), 2N equations are obtained from the above equations (2) and (3) for k = 1, 2,..., N (N ≧ 4). When these equations are expressed by matrix multiplication, the following equation (4) is obtained.

At this time, h (bold) whose elements are eight parameters (eight elements) of the projective transformation matrix H (bold) ′ can be obtained by the least square method using the following equation (5). B (bold) ⁺ indicates a pseudo inverse matrix of B (bold).

  Projection transformation matrix H (bold) 'can be estimated by shaping the element of h (bold) obtained by the above equation (5) as in the above equation (3). In addition, the estimation method of the projection transformation matrix H (bold) 'described above is an example, and a specific processing procedure is not limited to the above example.

Using the estimated projective transformation matrix H (bold) ′, the coordinates [u ″ of the original lattice pattern from [u (k), v (k)] ^T as shown in the following equation (6). (K), v ″ (k)] ^T is estimated.

Finally, [u ″ (k), v ″ (k)] ^T is deduced from [u (k), v (k)] ^{T in} which direction the address code is obtained by obtaining for each lattice point. presume. That is, the estimated address code p _est (k) is given by the following equation (7).

However, θ = tan ⁻¹ ((v ″ (k) −v (k)) / (u ″ (k) −u (k))) is [u (k), v (k)] ^T The angle of [u ″ (k), v ″ (k)] ^T viewed from the point of view.

In the address code estimation method described above, the tendency of the projection transformation estimation error varies depending on the address code. _Therefore , in the estimation of the estimated address code p _est (k), an error-prone code string and an error-prone code string are difficult. Exists. 6 is used for estimating the projective transformation matrix H (bold) ′, compared to the case where all nine points constituting the lattice pattern are used in the case of using the reference pattern of FIG. It has been confirmed by simulation that the estimation accuracy of the address code is higher when the points excluding the center point of 6 are used. As described above, by appropriately selecting the number of grid points used at the time of estimation based on the grid pattern, it is possible to improve the accuracy of address code estimation.

  Returning to the description of FIG. 16, after the process of step S <b> 3, the address determination unit 23 selects video data to be projected based on the estimated address code and the stored address code corresponding data 26 (step S <b> 4). The address code correspondence data 26 is the same data as the address code correspondence data 63 and is prestored in the video processing device 2.

  Then, the coordinate conversion matrix determination unit 24 converts the selected video data using the coordinate conversion matrix estimated at the time of estimating the address code, and generates the converted video data as projection data. It outputs to the projection apparatus 3 (step S5). Thereafter, the projection device 3 projects the projection data onto the screen 1.

  Further, the numbers of the lattice patterns shown in FIG. 15 are determined according to a score indicating the estimation accuracy of the address code. As described above, the estimation accuracy of the address code depends on the value of the address code when the address code is set (how to shift from the reference pattern).

In the following, a description will be given of a method for selecting an address code that has a high estimation accuracy of the address code, that is, robust to projective transformation. Robust address code to projective transformation, for all the address code (4 ^nine), obtaining a score respectively the robustness index for projective transformation. Here, a projection transformation is applied by simulation to change the facing plane into a plane rotated about the X-axis and Y-axis (roll and pitch) by 5 ° in a range of ± 80 by ± 80. Is estimated. If all nine address codes are correct, the score is incremented by one. Therefore, the upper limit of the score is 33 × 33 = 1089 corresponding to the case where all the address codes are correct at all angles around the X axis and the Y axis.

  In this way, a score is obtained for each address code, and a 3 × 3 lattice pattern is arranged on the scanning line in descending order of score to generate a lattice pattern. In FIG. 15, the grid pattern number indicates the score corresponding to the grid pattern. Thus, in FIG. 15, the lattice patterns are arranged in descending order of score. Although FIG. 9 shows an example in which lattice patterns are arranged in descending order of score, the arrangement on the screen 1 is not limited to this. However, it is desirable to arrange a grid pattern having a high score in a place where distortion at the edge of the screen 1 is expected to increase.

  With the above processing, the position and rotation state of the screen 1 (projection transformation with respect to the reference coordinate system on the screen) can be estimated at high speed.

  Although an example in which a 3 × 3 square lattice pattern is used as an example of an address code embedding pattern has been described here, a rectangular pattern of n × m (n ≠ m) points may be used. Here, an example has been described in which the interval between the vertical and horizontal points is equal as the reference pattern, but one or both of the vertical and horizontal directions may not be equal. Furthermore, the reference pattern may be a circle or a polygon instead of a square or rectangle, and there is no restriction on the shape as long as it is a pattern composed of five or more points. However, in order to reduce errors in position calculation, it is desirable that each point in the reference pattern is arranged so as not to be biased in the reference pattern as much as possible. Even if the lattice patterns overlap each other, the position can be specified by the position specifying method of the present embodiment.

  By using the lattice pattern in which the above-described address code is embedded, it is possible to estimate the position on the captured camera image, the rotation angle of the two-dimensional coordinate, the position of the three-dimensional plane, and the inclination of the three-dimensional plane. These pieces of information are information used for geometric correction of the projected image in the projector / camera system. On the other hand, by using these pieces of information, a lattice pattern in which an address code is embedded can be used as an AR (Augmented Reality) marker. AR is a technology for adding and displaying information to a camera image when an image taken by a camera is displayed on a display. For example, character information is added to an image, or an object that does not actually exist is added using a CG (Computer Graphics) model. In this AR, an AR marker is often used as an auxiliary. For example, the Hiro marker is famous. The AR marker is printed on paper, cloth, flat plate, etc., taken with a camera, and the position and inclination of the CG model are determined in accordance with the position and inclination of the AR marker. Information (such as a CG model) can be combined with a camera image without a sense of incongruity.

  The grid pattern with the embedded address code can estimate the position on the camera image, the rotation angle of the two-dimensional coordinates, the position of the three-dimensional plane, and the inclination of the three-dimensional plane from the camera image. It can be used to determine the position / posture for combining information. In addition, since the shape of the surface (plane or gentle curved surface) on which the lattice pattern is formed can be estimated from the camera, virtual information that is not actually formed (images, patterns, characters, etc.) appears on the plane. It is also possible to add information for composition to an image as if it were formed. That is, the lattice pattern can be used as an AR marker by generating an address code based on information for synthesizing virtual information (CG model, planar image, pattern, character, etc.) with a camera image. The method for mounting the lattice pattern is not limited to the formation, and for example, a method of applying a substance to be a point of the lattice pattern to the screen 1 by spraying or the like may be used.

  In the lattice pattern in which the address code is embedded, the absolute coordinates on the forming surface can be defined by the address code, so that different information can be added for each address code. At this time, even if information corresponding to the address code cannot be added to the image due to the temporary disappearance of the address code as the camera moves, the address code can be seen again from the camera. The information corresponding to the address code can be displayed again.

  In general, when it is desired to add different information for each marker using AR markers that have been used so far, many types of AR markers are required. On the other hand, when a lattice pattern in which an address code is embedded is used as an AR marker, various information can be easily added by changing the address code.

  AR markers that have been used so far have a unique code or pattern in a rectangular frame, such as the Hiro marker, in order to robustly determine the position and orientation. On the other hand, since the lattice pattern in which the address code is embedded can express information by the coordinates of the point, the formed AR marker can be made inconspicuous. In addition, it is conceivable that a material that reflects only light in a specific direction or a material that emits or reflects light other than visible light (for example, infrared light) is formed on the screen as a lattice pattern. By using such a material, it is possible to realize an AR marker whose lattice pattern is not visible to the observer.

  As described above, the screen, the position specifying device, the image processing device, the pattern generating device, and the screen manufacturing method according to the present invention are useful for a screen system that projects an image, and are particularly suitable for a system in which the screen moves. Yes.

  1, 301 to 308, 402 screen, 2 image processing device, 3 projection device, 4 photographing device, 6 pattern generation device, 11 cut sheet, 12 through hole, 13 retroreflective sheet, 14 base material, 21 coordinate calculation unit, 22 grid pattern estimation unit, 23 address determination unit, 24 coordinate transformation matrix determination unit, 25 video data conversion unit, 26 address code correspondence data, 27 video data, 61 address code arrangement unit, 62 formation data generation unit 62 address code correspondence data , 101 control unit, 102 input unit, 103 storage unit, 104 display unit, 105 communication unit, 106 system bus, 401 frame, 403 connection member.

Claims (10)

A reflective material;
A light shielding material that covers the reflective material and is formed based on an address code indicating a position where a pattern is formed and has a plurality of through holes corresponding to a plurality of points constituting the pattern;
With
The pattern is arranged at a position in the light shielding material that is deviated from a corresponding point of the reference pattern for each point of the pattern with respect to a reference pattern composed of a plurality of points.
A value obtained by quantifying the direction of deviation from the corresponding point of the reference pattern corresponding to the point constituting the pattern is a point code, and the address code is composed of the point codes of the plurality of points constituting the pattern A screen characterized in that it is a sequence of numbers.   The screen according to claim 1, wherein the reflective material is a retroreflective material or a diffuse reflective material. A base material disposed on the back surface of the surface on which the light shielding material of the reflective material is disposed;
The screen according to claim 1, further comprising:   The screen according to claim 1, wherein the reference pattern is a square lattice.   The screen according to claim 4, wherein the reference pattern is a 3 × 3 square lattice.   4. The screen according to claim 1, wherein n is an integer, and m is an integer different from n, the reference pattern is a rectangle of n × m points. 5. A coordinate calculation unit that calculates a coordinate value in the image of the point in the image based on an image obtained by photographing a member in which a pattern including a plurality of points is arranged;
A lattice pattern estimation unit that estimates an address code that is information indicating the position of the pattern based on the coordinate value;
With
The pattern is arranged at a position in the member that is deviated from a corresponding point of the reference pattern for each point of the pattern with respect to a reference pattern composed of a plurality of points.
A value obtained by quantifying the direction of deviation from the corresponding point of the reference pattern corresponding to the point constituting the pattern is a point code, and the address code is composed of the point codes of the plurality of points constituting the pattern A position specifying device characterized in that the position specifying device is a number sequence. A coordinate calculation unit that calculates a coordinate value in the image of the point in the image based on an image obtained by photographing a member in which a pattern including a plurality of points is arranged;
A lattice pattern estimation unit that estimates an address code that is information indicating the position of the pattern based on the coordinate value;
Based on the address code, a projective transformation matrix corresponding to the shape of the member is calculated, and video data to be projected onto the member is converted into a video data conversion unit using the projective transformation matrix,
With
The pattern is arranged at a position in the member that is deviated from a corresponding point of the reference pattern for each point of the pattern with respect to a reference pattern composed of a plurality of points.
A value obtained by quantifying the direction of deviation from the corresponding point of the reference pattern corresponding to the point constituting the pattern is a point code, and the address code is composed of the point codes of the plurality of points constituting the pattern A video processing apparatus characterized by being a sequence of numbers. A lattice pattern generation unit that generates a pattern composed of a plurality of points;
An arrangement unit that determines the arrangement of the pattern in the member based on an address code that is information indicating the position of the pattern;
With
The pattern is arranged at a position in the member that is deviated from a corresponding point of the reference pattern for each point of the pattern with respect to a reference pattern composed of a plurality of points.
A value obtained by quantifying the direction of deviation from the corresponding point of the reference pattern corresponding to the point constituting the pattern is a point code, and the address code is composed of the point codes of the plurality of points constituting the pattern A pattern generation apparatus characterized by being a numerical sequence to be processed. A pattern forming step in which a plurality of through holes corresponding to a plurality of points that are generated based on an address code indicating a position where a pattern is formed on the light shielding material and that constitute the pattern are formed; and
A bonding step of bonding the light-shielding material and the reflective material;
With
The pattern is arranged at a position in the light shielding material that is deviated from a corresponding point of the reference pattern for each point of the pattern with respect to a reference pattern composed of a plurality of points.
A value obtained by quantifying the direction of deviation from the corresponding point of the reference pattern corresponding to the point constituting the pattern is a point code, and the address code is composed of the point codes of the plurality of points constituting the pattern A method for manufacturing a screen, wherein