KR101408295B1 - Projection device on nonflat screen and method therefor - Google Patents

Abstract

The present invention applies different colors or images on a nonflat screen, which is made of particles to be transformable, according to the height of the screen using a beam projector. According to the present invention, a depth camera reads information on an image on the transformable nonflat screen and information on an X, Y, and Z-axes of each pixel of the image. An image processing part changes the image into a color value or an image corresponding to the information on a Z-axis of each pixel measured by the depth camera. The changed color value or image is outputted to a position corresponding to the screen by the beam projector. Therefore, an image outputted by the beam projector changes when the nonflat screen made of particles is transformed by an external force.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a projection apparatus and method using a non-planar screen,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a projection apparatus and method using a non-planar screen, and more particularly, to a projection apparatus and method using a non-planar screen which is configured to respond to a shape change of a non-planar screen while outputting a different color or image, And more particularly, to a projection apparatus and method using the same.

Generally, when an image is displayed using a beam projector, an image output from the beam projector is displayed on the screen. The image displayed in this way can be said to be simply two-dimensional. The image displayed by the beam projector is an image implemented on a flat screen because the screen itself is planar.

However, if the screen itself has a constant height value (Z value) with respect to the XY plane, it may be considered to perform a projection such that the projector has different color values depending on the difference in height. When the non-planar screen is implemented with particles such as sand, for example, the shape of the screen formed by such particles is easily deformed by external force. In order to cope with the real time height change value of the screen, .

SUMMARY OF THE INVENTION It is a primary object of the present invention to provide a projection apparatus capable of outputting an image that can match a height (Z value) with respect to a non-planar screen having a height value.

It is another object of the present invention to provide a projection apparatus capable of changing the color of a height in real time while responding to a change in height value of a non-planar screen.

A projection apparatus using a non-planar screen according to the present invention includes: an image of a non-planar screen capable of changing a shape; a depth camera for reading XYZ-axis information about each pixel of an image; An image processor for transforming the color value or the image corresponding to the magnitude of the Z-axis information for each pixel measured by the depth camera; A beam projector for outputting the color value or the image converted by the image processing unit at a position corresponding to the image of the screen; And a control unit controlling the image processing unit to output the beam projector. At this time, according to the change of the shape of the non-planar screen, the image outputted by the beam projector can also be changed in real time.

According to another embodiment, the control unit may further include an input unit for receiving a signal to be applied to the image processing unit under the control of the control unit.

According to another embodiment of the present invention, the apparatus further comprises screen correcting means for matching the frame of the image output by the beam projector with the screen.

According to another embodiment of the present invention, it is possible to further comprise conversion means for matching the image obtained by the depth camera with the output image of the beam projector corrected by the correction means.

According to another embodiment, based on the XYZ axis information obtained by the depth camera, the control unit judges the object on the non-planar screen and controls the beam projector to output the image corresponding to the object.

The method according to the present invention includes the steps of receiving XYZ value information of each pixel forming an image and an image of a non-planar screen by a depth camera; Converting the Z value information from the XYZ value information into a corresponding color value; And controlling the beam projector to output a different color corresponding to the Z value information with respect to the position corresponding to the XY value.

According to an embodiment of the present invention, there is further provided a screen correcting step of correcting the image frame outputted by the beam projector and formed on the screen to coincide with the screen.

According to a specific embodiment of the screen correction process, the screen image frames sa1, sa2, sa3, and sa4 after rectifying the rectangle are inputted on the screen and are calculated by the corrected projector coordinate systems np1, np2, np3, (P1, p2, p3, p4) of the projector coordinate system to a projector coordinate system after correction using a perspective transform (PT) matrix.

According to another embodiment of the present invention, the method further comprises a matching process of matching the corrected projected coordinate system with a camera coordinate system input from the depth camera.

An example of such a matching process is a process of matching Z values based on an image of a beam projector projected on a screen and a process of projecting image frames c1, c2, c3, c4 of a beam project projected by a depth camera, (P1, p2, p3, p4) in the projector coordinate system using a perspective transform matrix (PT) and then transforming it into a pre-correction image frame To the corrected image frames np1, np2, np3, and np4.

As described above, according to the present invention, a non-planar screen, which is formed of powder and changes its shape by an external force, has a technical effect that can output a specific color or image corresponding to height information or a screen- Lt; / RTI > The present invention makes it possible to display a more efficient image and color on a screen including a substantially three-dimensional screen and being transformed in real time.

1 is a schematic view of a projection apparatus according to the present invention;
2 is an exemplary block diagram of a projection apparatus of the present invention.
3 is an illustration of an image frame output on a screen;
4 is an exemplary view of an output image frame viewed from a projector coordinate system.
5 is an exemplary view of an image before and after correction based on a projector coordinate system.
Fig. 6 is an illustration of an image before and after correction based on a screen; Fig.
7 is an exemplary view showing the difference between the image of the depth camera and the image of the projector.
8 is an exemplary diagram including a vector for obtaining a new coordinate system;
9 is a diagram illustrating a process for matching an image acquired by a depth camera with an image of a beam projector.
Figure 10 is an illustration of an example non-planar screen implemented in accordance with the present invention;

Hereinafter, the present invention will be described in more detail based on the embodiments shown in the drawings.

FIG. 1 is a block diagram illustrating a basic configuration of a display apparatus according to an embodiment of the present invention. Referring to FIG. As shown, the screen 10 according to the present invention has a height value that can be expressed as a Z-axis on a plane consisting of an X-axis and a Y-axis.

For example, the screen 10 may be configured such that the upper surface thereof is opened and the inside thereof is filled with a material such as sand or a particle. When the operator touches the sand or the like, the shape of the sand changes corresponding to the touch of the screen 10. That is, it can be said that the Z value as the height value can be changed with respect to a certain point having a predetermined XY coordinate value. Such a screen 10 may be referred to as a non-planar screen or a nonplanar screen.

In the present invention, the projector 30 is used to provide a display device that is matched to a three-dimensional image of the screen by outputting different colors corresponding to the respective height values of the non-planar screen. Here, outputting another color corresponding to the Z value in a non-planar (three-dimensional) screen means outputting a color or an image with saturation and / or brightness that can be visually distinctly distinguished by combining RGB in the beam projector it means. And control for such display is performed in the computer 100.

2, the depth camera 20 photographs the non-planar screen 10, and extracts XYZ-axis information corresponding to each of the pixels of the image to be photographed have. That is, in addition to the XY value of the image, it is possible to measure the Z value of each pixel constituting the image using infrared rays, and it is already commercialized. The data obtained by the depth camera 20 is transmitted to the image processing unit 140 inside the computer.

The image processing unit 140 converts the Z value corresponding to each pixel from the data received from the depth camera 20 into data having specific color information or data having a specific image. Here, the color or image corresponding to the Z value, that is, the height of each point on the non-planar screen, is stored in advance in the form of a table, and is converted into color information extracted from the table value. Alternatively, the table values may be stored in the image processing unit 140 or stored in the storage unit 120, and may be selected and used under the control of the control unit 110. Accordingly, under the control of the control unit 110, the image processor 140 converts the Z value of the data obtained by the depth camera 20 into specific color information, and outputs the color information to the beam projector 30 as an XY value thereof do.

Where the Z values may be grouped. That is, the data of the Z value among the data obtained by the depth camera 20 may be grouped so as to have the same color or image up to a certain range around a certain reference point (the highest point, the lowest point, or the center point on the screen). It may be performed by information or commands input by the user through the input unit 130 or may be designed to be embedded in the image processing unit 140 itself.

Accordingly, the image output by the beam projector 30 has a corresponding color according to the height of each part in the non-planar screen 10. [ For example, if the shape implemented on the non-planar screen 10 has a continuous shape of mountains and oceans, a portion having an XY value corresponding to a high-height mountain is output so as to realize green, The part having XY value corresponding to the sea may be outputted so as to realize blue color. By adjusting the ratio of RGB to each part, it is possible to irradiate the peak of the mountain to the deepest green color and the deepest part of the sea to the deepest blue color.

When the non-planar screen made of particles is deformed by external force, the Z value for each XY coordinate value changes. The Z value thus changed is input to the image processing unit 140 through the depth camera, and the color corresponding to the changed Z value under the control of the control unit 110 can be output to the screen. That is, according to the present invention, it can be seen that the color outputted from the projector is changed in real time corresponding to the change of Z value (height value) in the non-planar screen.

Figure 10 illustrates a non-planar display projected by a beam projector in accordance with the present invention. Next, a description will be made of an embodiment that can be modified in the projection apparatus of a non-planar screen according to the present invention as described above.

First, the image processing unit 140 may determine the terrain information of the non-planar screen based on the information obtained from the depth camera. That is, if the terrain having a constant width in the XY plane is continuous within a certain height (Z value) range, it can be recognized as a road. It is needless to say that the algorithm or condition for judging such conditions must be stored in the image processing unit 140, and it is needless to say that it is necessary for the control unit to determine the algorithm or condition based on the algorithm or condition.

As another example, a tree can be defined as having a vertex when viewed on the Z axis, and having a lower height gradually within a certain range of XY values around this vertex. Therefore, the information obtained from the depth camera is calculated, and if it has such a condition value, it can be judged that the object is a tree.

As described above, for an object determined to be a road or a tree, a corresponding image should be output by the beam projector. That is, when the condition of the various objects is embedded in the storage unit or the image processing unit 140 and the condition value in the image processing unit is determined to correspond to the specific object as a result of the determination by the control unit, May be output on the corresponding XY plane.

Based on the XYZ value information acquired by the depth camera, an object on the non-planar screen can be determined, and an image corresponding to the object can be output to the corresponding portion. Here, the image may include a specific image stored in the storage unit or the image processing unit 140, in addition to the color as described above.

The beam projector output under the control of the control unit may further output a specific image stored in the image processing unit or the storage unit. For example, when the control unit determines that a certain region is the sea on the non-planar screen, it is possible to output an image of a fish shape, and in particular, it is also possible to output a form in which a fish shape image continuously moves. Alternatively, if it is recognized that a certain area on the non-planar screen is a plane with no elevation difference, it is also possible to output an image of a tree or the like within such area.

Of course, in the output of such a beam projector, it is needless to say that, in the output of the beam projector, it is judged by the image processing section based on the XYZ information of the image information photographed by the depth camera or the control section judges this according to the condition stored in the image processing section, The control unit must control the beam projector.

Here, the beam projector 30 preferably outputs an image at a position perpendicular to the reference plane of the non-planar screen 10. That is, when a flat surface forming the bottom of the non-planar screen 10 is referred to as a reference surface, the beam projector 30 should be installed at right angles to the standard surface (reference screen surface) So that the occurrence of distortion can be suppressed. However, as can be seen from FIG. 1, it is physically impossible to install both the beam projector 30 and the depth camera 20 at the correct positions.

For example, when the beam projector 30 is installed in a slightly tilted state on one side and output to the screen, an output image frame 50 having black edges is displayed as shown in FIG. The substantially distorted image frame 50 will be illuminated on the screen 10 and if so output may not exactly match the actual height (Z value) on the non-planar screen 10. [ It is a matter of course that the larger the size of the screen 10, the more distorted the image is displayed.

Hereinafter, the image of the beam projector irradiated on the non-planar screen 10 is made to coincide with the model material (an object such as an acid formed by particles such as sand or the like) implemented on the actual nonplanar screen 10 An example will be described. First, a process of accurately matching an image output from the beam projector 30 to an actual non-planar screen 10 will be described.

3, the image screen frame 50 shown on the screen 10 is formed by points s1, s2, s3 and s4, and the beam projector 30 is installed in a tilted state and output, Shape. This type of frame 50 should be output on a screen so that it can have a corrected screen frame 60 consisting of an exact rectangle, i.e., an accurate rectangle without distortion, sa1, sa2, sa3, sa4.

Here, the image screen frame 50 displayed on the screen 10 has a distorted quadrangle, but this is only due to the relative relationship between the beam projector 30 and the screen 10, As shown in Fig. 4, a rectangular shape having an angle of 90 degrees is output. In other words, in the projector coordinate system centering on the beam projector 30, a rectangular shape as shown in FIG. 4 is once formed.

That is, when outputting from the beam projector, the corner has a rectangular frame 7 having a value of 90 degrees. However, since the space between the beam projector 30 and the screen 10 does not correspond exactly vertically, 10, a distorted quadrangle and quadrangle image screen frame 50 as shown in Fig. 3 (screen frame before correction). And in the image screen frame 50 in Fig. 3, the correction frame 60 should be displayed on the screen 10 as described above.

Hereinafter, the image frame output by the beam projector 30, that is, the image frame shown in FIG. 4 will be referred to as a projector coordinate system, and the coordinate system displayed on the screen will be referred to as a screen coordinate system. Therefore, the image screen frame 50 shown in FIG. 3 is substantially based on the screen coordinate system, that is, before the correction was performed (screen frame before correction), and the image frame shown in FIG. 4 is based on the projector coordinate system (Pre-compensation projector frame).

Therefore, it can be seen that the frame of the image outputted from the beam projector 30 appears as a distorted rectangle in relation to the rectangle or the screen 10. In contrast, if the image is output as a frame having a distorted quadrangle in the beam projector 30, that is, as a distorted quadrangle based on the projector coordinate system, a rectangular image frame can be obtained in the screen coordinate system. This point is focused on.

3, the pre-correction screen frame 50 appears as a distorted quadrangle when the screen coordinate system is taken as a reference. With respect to the black coordinate point marked on the corner, with respect to the projector coordinate system, (P1, p2, p3, p4) indicated by black in FIG. That is, in the projector coordinate system, the point represented by the rectangles p1, p2, p3, and p4 (pre-correction projector frame) is displayed as a distorted rectangle in the screen coordinate system.

The pre-correction projector frame having the black points p1, p2, p3, and p4 is already recognized by the image processing unit 140. [ That is, when setting the image processing unit 140, the image processing unit 140 recognizes the information about the point by setting each corner of the image frame output by the beam projector, Are recognized by the image processing unit 140.

In order to correct the image frame displayed on the screen by the beam projector 30, a rectangle (screen frame after correction) to be displayed in the screen coordinate system should be designated, as indicated by the red dot in FIG. In FIG. 3, the four points indicated in red are input points through the input unit 130 by the user. Such a screen frame after the correction indicated by red dots can be regarded as an output frame which can be viewed as a rectangle as shown in the screen coordinate system and visually without distortion.

Next, Fig. 5 is a projector coordinate system corresponding to the screen coordinate system of Fig. In FIG. 5, the points indicated by black are the same as the points (p1, p2, p3, p4) in FIG. 4 but can be referred to as a pre-correction projector frame viewed based on the projector coordinate system. The red dots np1, np2, np3, and np4 are red dots that are input through the screen 10 as square corners of a frame desired to be displayed by the user. (Projector frame after correction).

Next, the frame (pre-correction projector frame) by the points p1, p2, p3, and p4 is divided by the points np1, np2, np3, and np4 as shown in Fig. Into a frame to be formed (corrected projector frame). Since the points (np1, np2, np3, np4) indicated by red are determined by the user, the image processing unit 140 detects the points np1, np2, np3 and np4 through the input unit 130 and performs the following processing .

In order to convert a point in the projector coordinate system to a point in the user selected area, an operation is performed using a PT (Perspective Transform) matrix. These PT matrices match the coordinates of the rectangle and the quadrangle, and this operation itself is known.

For example, each point (np1, np2, np3, np4) can be matched using a PT matrix, which can be calculated as a product of the PT matrix as follows.

np1 = PT x p1,

np2 = PT x p2,

np3 = PT x p3,

np4 = PT x p4.

The pixels inside the points (np1, np2, np3, np4) can also be calculated and have predetermined coordinate values when calculating the values of the points. That is, when a point is designated in a coordinate system with respect to a designated point, the 3D SDK (Software Development Kit, Software Development Kit; OpenGL or DirectX) used in the present invention calculates coordinate values corresponding to the pixels inside Function. Accordingly, the pixels inside the frame formed by the points (np1, np2, np3, np4) can also obtain corrected coordinate values.

The values of the obtained points (np1, np2, np3, np4) are based on the projector coordinate system, and substantially the beam projector outputs the frames formed by the points (np1, np2, np3, np4). However, since the beam projector 30 and the screen 10 are relatively inclined as described above, when an image is output in this state, an image of a frame as shown in Fig. 6 is displayed on the screen.

Fig. 6 shows the screen coordinate system as a reference. A frame displayed in blue is a rectangular frame output as a frame by the points np1, np2, np3, and np4 (corrected screen frame). That is, a frame by a screen coordinate system formed by the points np1, np2, np3, and np4. And the gray frame may be displayed on the screen in a substantially distorted state, which is illustratively a frame by the projector coordinate system output from the beam projector (screen frame before correction).

When the above process is performed, the image output from the beam projector has an image frame on the corrected screen as shown in blue in FIG. 6, which can be said to be a rectangle. In the process up to now, an embodiment has been described in which the output value by the beam projector projected at a certain angle can be outputted as a frame having a rectangular shape on the screen.

Next, an embodiment in which the output of the beam projector and the recognition coordinates of the camera are perfectly matched will be described.

7, reference numeral 70 denotes an image frame outputted by the beam projector 30 and formed on the screen 10, and reference numeral 80 denotes an image frame which is captured by the depth camera 20 on the basic plane formed by the screen 10 It is an image frame. Then, the projector coordinate system is viewed in two-dimensional coordinates made up of XY, and the depth camera 20 acquires three-dimensional coordinate information of XYZ in the process of recognizing the projected image.

When the image of the projector is photographed by the depth camera 20, the Z value of the projected image plane is not constant when it is not perpendicular to the screen 10. Therefore, all Z values must be converted equally. As shown in Fig. 8, a new coordinate system for the plane recognized by the camera has to be created, which can be created by the following procedure.

If the normal vector of the plane is obtained and (1, 0, 0) is orthogonal to the normal vector of the plane, the Y-axis of the new coordinate system can be obtained as a result. Then, we can obtain the new X axis as a result of externalizing the normal vector of the new Y axis and the plane, and set the size of all the newly obtained axes as 1.

The point P (Xp, Yp, Zp) on the plane in the camera coordinate system is converted into a point NP (Xnp, Ynp, Znp) in the new coordinate system by obtaining a new coordinate system in this way and letting the new XYZ axes be NX, .

Here, if point P is inner product of the axis of the new coordinate system, NP can be obtained as follows.

X np = N x P

Ynp = NY · P

Znp = NZ · P

In addition, the matrix that converts P to NP is called NA.

NP = NA x P

The Z values of all the points on the same plane after the conversion are thus equal.

In FIG. 9, a depth camera formed by the points c1, c2, c2 and c4 appears as a perspective image frame of the projector taken by the camera (a camera image pickup frame). If such an image frame is made to correspond to an image frame made up of red dots as shown in Fig. 5 and project frames (np1, np2, np3, np4) corrected with porosity, the camera coordinate system of the depth camera, The projector's coordinate system coincides with that of the projector. Therefore, the coordinate system of the pixels on both sides can be matched exactly by matching the image frame shown in FIG. 9 (a) to that of the projector image through the following process.

As described above, the PT matrix is a matrix capable of substantially transforming a quadrangle into a rectangle or vice versa. This PT matrix can convert the quadrangle and the rectangular image into each other. Therefore, in order to correct the imaging frames c1, c2, c3 and c43 taken by the depth camera to the corrected project frames np1, np2, np3 and np4 You have to go through the following two steps.

The image shown in Fig. 9A is first converted into a rectangular image frame as shown in Fig. 9B, that is, the pre-correction projector frame p1, p2, p3, p4. This pre-correction projector frame is the same as the image frames p1, p2, p3 and p4 shown in Fig. 4, and is a rectangle formed when the beam projector first outputs it, and is stored in the image processing unit 140 . The PT matrix described above is also used by matching the coordinate system (c1, c2, c3, c4) of the above depth camera to the coordinate system (p1, p2, p3, p4) of the original beam projector.

Next, the coordinate system converted into (b) should be matched with the coordinate system of the finally outputted beam projector. The coordinate system of the finally outputted beam projector is rectangular frames (np1, np2, np3, np4) indicated by red dots in Figs. 5 and 9C. In this process, the PT matrix described above is used. That is, the coordinate system (p1, p2, p3, p4) of the image shown in (b) is matched with the coordinate system (np1, np2, np3, np4) .

In this case, the image information substantially obtained from the depth camera substantially coincides with the coordinate value of the image information substantially output from the beam projector. This coincidence means that the information actually possessed by the image obtained by the depth camera is completely identical to the information output by the beam projector. Thus, the beam projector will be able to output a specific color to the non-planar screen without error based on the coordinate values obtained by the depth camera.

The image thus output through the correction process to the non-planar screen 10 will be completely consistent with the height value of the shape of the non-planar screen 10. If the shape of the non-planar screen 10 is deformed by the external force as described above, it is photographed by the depth camera, and the image processing unit 140 performs the process of the above-described process under the control of the control unit, Will be output. Therefore, the image corresponding to the changed shape can be output from the beam projector while responding to the shape change on the non-planar screen in real time.

According to the present invention as described above, the image having the corresponding color is projected on the non-planar screen according to the height of the shape realized on the non-planar screen, based on the information about the XYZ axis obtained from the depth camera. It can be seen that it is technological thought.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims. Of course.

10 ..... Non-flat screen
20 ..... depth camera
30 ..... beam projector
50 ..... Screen image frame before correction
60 ..... Screen image frame after correction
70 ..... Pre-compensation projector image frame

Claims (10)

A depth camera for reading the image of the non-planar screen capable of changing the shape and the XYZ axis information for each pixel of the image;
An image processor for converting the Z-axis information of each pixel measured by the depth camera into a color value or an image corresponding to the Z-axis information;
A beam projector for outputting the color value or the image converted by the image processing unit at a position corresponding to the image of the screen; And
And a control unit controlling the image processing unit to output the beam projector;
Wherein the image outputted by the beam projector is changed in accordance with the change of the shape of the non-planar screen.
The projection apparatus according to claim 1, further comprising an input unit for receiving a signal to be applied to the image processing unit under the control of the control unit.
3. The projection apparatus according to claim 1 or 2, further comprising screen correcting means for matching the frame of the image output by the beam projector with the screen.
4. The projection apparatus according to claim 3, further comprising conversion means for matching the image obtained by the depth camera with the output image of the beam projector corrected by the correction means.
3. The apparatus according to claim 1 or 2, wherein, based on the XYZ-axis information obtained by the depth camera, the control unit judges the object on the non-planar screen and controls the non-planar surface to control the beam projector to output the image corresponding to the object Projection device using screen.
Receiving a XYZ value information of each pixel forming an image and an image of a non-planar screen capable of changing a shape by a depth camera;
Converting the Z value information among the XYZ value information into a corresponding color value or image; And
And controlling the beam projector to output a different color or image corresponding to the Z value information with respect to the position corresponding to the XY value;
Wherein the output of the color or image changes as the shape of the non-planar screen changes.
7. The projection method according to claim 6, further comprising a screen correcting step of correcting the image frame output by the beam projector and formed on the screen to coincide with the screen.
8. The method of claim 7,
After the screen image frames sa1, sa2, sa3 and sa4 after rectification on the screen are received and computed by the corrected projector coordinate systems np1, np2, np3 and np4, the pre-correction image frame p1 in the projector coordinate system , p2, p3, p4) are transformed into a projector coordinate system using a perspective transform (PT) matrix and then transformed into a projector coordinate system.
9. The method of claim 8,
Further comprising a matching step of matching the corrected projected coordinate system with a camera coordinate system input from the depth camera.
10. The method of claim 9,
A process of matching Z values based on an image of a beam projector projected on a screen,
(P1, p2, p3, p4) in the projector coordinate system by using a perspective transform (PT) matrix, and receives projection image frames c1, c2, c3 and c4 of the beam project projected by the depth camera And transforming the transformed image frame into a corrected image frame (np1, np2, np3, np4) in a projector coordinate system using a perspective transform matrix (PT).

Images