JP6412685B2 - Video projection device - Google Patents

Description

  The present invention relates to a technique for superimposing and displaying an image on a real object.

  There is a projection device that projects visual information (hereinafter referred to as “visual information”) such as graphics, characters, still images, and images on a wall-mounted screen or white wall. Further, the projection apparatus can be connected to a public network such as the Internet or an in-house LAN, etc., and visual information can be transmitted from the remote to the projection apparatus for communication, remote conference, and remote presentation. The features of the projection device are that it can be displayed on a large screen relatively easily, and that visual information can be directly projected and superimposed on the actual object.

  As one of the usage scenes taking advantage of the latter feature, there is a scene where veterans give guidance to workers on site from a distance. In such a case, it is important for the worker on site to work while looking at the actual product. Therefore, if information is superimposed on a real object and presented, it is considered that the operator is intuitive and easy to understand, and the work becomes easier.

  By the way, depending on the work site, there may be an obstacle between the projection surface and the projection apparatus. According to the following Patent Document 1, a camera that captures the state of a projection plane is provided in a projection apparatus, and when there is a difference between an image captured by the camera and an image to be projected, an obstacle (shield) There has been proposed an apparatus that can determine that the presence of the. When such an obstacle is detected, it is also possible to change the layout of the projected image and reconstruct and present the image avoiding the obstacle.

JP 2013-148868 A

  By the way, in an actual work site, the projection surface is not a flat surface but a three-dimensional surface that is uneven, or the projection image may be reflected by a metal material of the projection surface. Further, visual information presented in such work instructions includes information in which the display position itself such as an instruction pointer is important, and general information that is not directly related to the display position such as a drawing or a figure.

  However, in Patent Document 1, an obstacle in front of the projection plane can be detected and the layout can be changed accordingly. However, the standing position of the user who is viewing the projected image is not considered, and the projection is not performed. Depending on the condition of the surface, it may not function sufficiently. Specifically, such a state is a state in which there is difficulty in viewing due to reflection on the projection surface, distortion of the image due to projections and depressions on the projection surface, and the like.

  The present invention has been made in view of the above problems, and in a projection apparatus capable of projecting visual information onto a real object, a projection position suitable for the standing position of the user who is viewing the visual information. It aims to be.

  The present invention is based on the position of the user and the depth information to the subject including the projection plane, depending on the projection positional relationship of the reflection relative to the standing position of the user or the degree of unevenness of the projection plane. The projection position is changed. Further, when the content of visual information to be projected is visual information that cannot be permitted to change the projection position, it is possible to project to the position as it is without changing the projection position.

  According to an aspect of the present invention, in a projection device that projects an image on a subject including a projection plane, a distance calculation unit that calculates a distance from the projection device to the subject, and a three-dimensional view of a user who uses the projection device Based on the user position detection unit for detecting the position, the distance to the subject calculated by the distance calculation unit, and the position of the user detected by the user position detection unit, the projection device projects the projection plane. A normal projection propriety determination unit that determines whether or not it is possible to normally project an image toward the projector, an object input unit that inputs information of an object to be projected onto the projection device, and the normal projection propriety determination unit And a drawing unit for projecting by changing the projection position according to the determination result.

  According to the present invention, in a projection device capable of projecting visual information onto a real object, a suitable projection position can be set according to the standing position of the user who is viewing the visual information.

  According to the present invention, in a projection device capable of projecting visual information onto a real object, a suitable projection position can be set according to the standing position of the user who is viewing the visual information.

1 is an external view of a projection apparatus according to a first embodiment of the present invention, and is a diagram schematically showing a use scene of the projection apparatus. FIG. It is a functional block diagram which shows the example of 1 structure of the projection apparatus by 1st Embodiment. It is a figure which shows the principle of the method of calculating a distance value from parallax. It is a figure which shows a parallax image, a user area | region mask, and a three-dimensional projection process. It is a figure explaining the principle of the reflection determination which judges whether a projection image | video is reflected by a user. It is a figure which shows the parallax image and picked-up image in the utilization scene of FIG. It is a figure which shows the three-dimensional position information for calculating the normal vector of a projection surface. It is a figure explaining object information and link information. It is a flowchart figure which shows the flow of the process by this Embodiment. It is the figure which showed typically the external appearance of the projection apparatus by the 2nd Embodiment of this invention, and the utilization scene of this apparatus. It is a figure explaining the object information of 2nd Embodiment. It is a flowchart figure which shows the flow of a process of 2nd Embodiment. It is the figure which showed typically the external appearance of the projection apparatus by the 3rd Embodiment of this invention, and the utilization scene of this apparatus. It is a functional block diagram which shows the example of 1 structure of the projection apparatus regarding 3rd Embodiment. It is a functional block diagram which shows the example of 1 structure of the remote terminal regarding 3rd Embodiment. It is a flowchart figure which shows the flow of a process of 3rd Embodiment. It is a flowchart figure which shows the flow of a process of a remote terminal.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. It should be noted that the expressions in the drawings are exaggerated as far as they are acceptable for easy understanding, and may be different from actual ones.
Moreover, in the following description, the structure which attached | subjected the same code | symbol also in different drawing is suppose that it is the same structure, and suppose that the description is abbreviate | omitted.

(First embodiment)
First, a first embodiment of the present invention will be described with reference to FIGS.
<Appearance of the device>
FIG. 1 shows the appearance of a projection apparatus according to the first embodiment of the present invention that can superimpose and display an image on an actual object, and displays the actual object using this projection apparatus, so that the user can work It is the figure which showed the mode that it is performing typically.

  Projection apparatus 107 according to the present embodiment includes projector 102, multi-camera 103, and control unit 104 (for example, a general-purpose personal computer (PC)). In FIG. 1, the projector 102, the multi-camera 103, and the control unit 104 are independent from each other. However, a configuration in which these functions are included in one housing may be used.

  Visual information projected from the projector 102 is projected onto the real object 101. The user 105 is working while viewing the projection information 106 projected on the real object 101. Here, the projection information 106 is information such as a drawing and text, and is schematically shown as an example representing information whose display position is not so important.

<Block configuration>
Next, FIG. 2 is a functional block diagram showing a configuration example of the projection device 107 in the present embodiment.
As shown in FIG. 2, the projection device 107 includes a multi-camera 103, a projector 102, a control unit 104, and a projection information input device 206 not shown in FIG. Here, the multi-camera 103 further includes a camera 1 (103-1) and a camera 2 (103-2). The control unit 104 further includes a distance calculation unit 201, a user standing position calculation unit 202, a normal projection availability determination unit 203, a drawing unit 204, and an object input unit 205.

<Content of processing>
The multi-camera 103 is a device that acquires an image of the shooting range including the real object 101 illustrated in FIG. 1, and can shoot subjects in the shooting range from different positions using, for example, two cameras. ing. The video acquired by the multi-camera 103 is input to the distance calculation unit 201 in the control unit 104. Here, the multi-camera 103 is not limited to the multi-camera as illustrated, and any device that can directly calculate the parallax calculated by the distance calculation unit 201 described later or the distance to the subject may be used. It can be replaced with a general-purpose ranging device. Specifically, general-purpose ranging devices are active parallax calculation devices using infrared light projection patterns, and TOF (Time of Flight) that measures distance based on the reflection time of infrared light to the subject. Various devices such as a distance measuring device can be applied. In the case of a distance measuring device capable of calculating parallax, the multi-camera 103 may be simply replaced. In the case of a distance measuring device capable of directly calculating a distance, the distance calculating unit 201 may be replaced.

  In the control unit 104, the video acquired by the multi camera 103 is taken into the distance calculation unit 201. The distance calculation unit 201 receives the images acquired by the camera (1) 103-1 and the camera (2) 103-2, and calculates a parallax image between these images. At the same time, based on the positional relationship between the camera (1) 103-1 and the camera (2) 103-2, the parallax image between the images is converted into a distance value. The distance calculation unit 201 outputs the calculated parallax image and the distance value to the normal projection availability determination unit 203 and the user standing position calculation unit 202. A method for calculating the parallax image and the distance value will be described later.

  In the control unit 104, the video acquired by the camera (2) 103-2 is taken into the user standing position calculation unit 202. The user standing position calculation unit 202 receives the video acquired by the camera (2) 103-2, the parallax image calculated by the distance calculation unit 201, and the distance value, and calculates the user standing position. The user standing position calculation unit 202 outputs the calculated user standing position to the normal projection availability determination unit 203. A method for calculating the user's standing position will be described later.

  The normal projection availability determination unit 203 can project the object normally by using the parallax image calculated by the distance calculation unit 201 and the distance value and the user standing position calculated by the user standing position calculation unit 202 as inputs. And outputs the result to the drawing unit 204. The normal projection propriety determination method will be described later.

  An object input unit 205 inputs object information and link information, which are information related to an object to be projected, from a projection information input device 206 (not shown in FIG. 1) outside the control unit 104 and outputs the information to the drawing unit 204 as it is. . Here, the projection information input device 206 only needs to be a device that can input object information and link information. You may input from an apparatus. The object information and link information will be described later.

The drawing unit 204 receives the determination result calculated by the normal projection propriety determination unit 203 and the object information and the link information from the object input unit 205, rearranges the objects appropriately if necessary, and converts the drawing data to the external Output to the projector 102.
Upon receiving the drawing data from the drawing unit 204, the projector 102 performs projection according to the drawing data.

<Details of each processing content>
A method of calculating a parallax image and a distance value performed by the distance calculation unit 201 will be described in detail with reference to FIG.

  First, a method for calculating a parallax image will be described. The parallax indicates that the corresponding positions of the subject are different in the two images taken at different positions, and the deviation. A parallax image representing parallax as an image is shown in FIG. 4A. The parallax image represents the above-described shift in terms of pixel values, and is normalized so as to be 0 to 255. A larger value represents a foreground (value 255, white), and a smaller value represents a distant view (value 0, black). In this specification, the distance to the foreground and the distance to the farthest background are limited, and description will be made assuming that the parallax does not exceed 255. When normalization is performed with the parallax exceeding 255, the parallax before normalization may be recalculated using the parallax amount between the foreground and the distant view when calculating the distance value shown below.

  It is assumed that the camera (2) 103-2 is located on the left side and the camera (1) 103-1 is located on the right side in the direction of the subject. FIG. 3 is a diagram showing the situation from directly above. In FIG. 3, a point 301 indicates one point where the subject is located, and further, a camera (1) 103-1 and a camera (2) 103-2 are shown. Here, of the two cameras, the left camera (2) 103-2 is used as a reference (reference camera), and the coordinate system of this camera is referred to as a reference coordinate system (hereinafter referred to as “reference coordinate system”). To do. Further, it is assumed that the two cameras have the same characteristics and are installed completely horizontally. The correction method when the characteristics of the two cameras are different or when they are not installed horizontally can be dealt with using camera geometry, but detailed description thereof is omitted. Further, there is no particular problem even if the left and right positional relationships of the camera (1) 103-1 and the camera (2) 103-2 are reversed. The parallax is obtained by extracting a block corresponding to a local block of a predetermined size selected from images captured by the reference camera from the other camera image (extracting by local block matching) and calculating the amount of deviation. be able to.

Here, the input image of the camera (1) 103-1 is I _R (x, y), the input image of the camera (2) 103-2 is I _L (x, y), and the local block size is 15 × 15. In this case, the formula for calculating the parallax M (x, y) is as follows. I (x, y) represents the luminance value at the pixel (x, y).

  As described above, the parallax image can be calculated by obtaining the shift amount that minimizes the sum of absolute values (SAD: Sum of Absolute Difference) in the local region at each pixel position in the image. However, the method for calculating the parallax image is not limited to the above method, and any method may be used as long as it can calculate the parallax images of the cameras installed at different positions.

Next, a method for calculating the distance value from the parallax image will be described. In order to calculate the distance value from the parallax image, a camera parameter indicating the characteristics of the photographed camera is necessary. The camera parameters include internal parameters and external parameters, which are calculated by using the method described in Chapter 6 “Camera Calibration” of “Computer Vision-Geometry of Vision” (Corona), for example. Is possible.
Of the calculated camera parameters, the distance value can be calculated as follows using the focal length f (unit m) and the distance d (unit m) between the cameras.

  In FIG. 3, one point 301 of the subject is shown (imaged) as an image 302 on the imaging surface of the camera (2) 103-2 and an image 303 on the imaging surface of the camera (1) 103-1. ). A position 304 is a point where the image 302 of the camera (2) 103-2 is shown on the projection plane of the camera (1) 103-1. Therefore, the pixel shift M between the image 303 and the position 304 is parallax. Further, a focal length f related to the camera parameter is denoted by reference numeral 305, and an inter-camera distance d is denoted by reference numeral 306.

  The relationship between the distance z (unit m) to the subject and the parallax M can be generally obtained as follows according to the principle of triangulation.

Here, q is the length (unit: m) per pixel of the image, and is a value determined by the image sensor employed in the camera. By the product of M and q, the amount of pixel shift can be converted into a real distance parallax.
As described above, the parallax image and the distance value to the subject can be calculated using images acquired by the two cameras arranged at different positions.

  Next, a method for calculating the user standing position performed by the user standing position calculating unit 202 will be described in detail with reference to FIG. The standing position of the user is a position (three-dimensional coordinates) in the reference coordinates. As a method for obtaining a user's standing position (hereinafter referred to as “user coordinates”), a general-purpose 3 that calculates by using information that generates ultrasonic waves or magnetism on the user's head or the like is used. Although a dimension position measurement device may be used, a method of using a parallax image and a distance value obtained by the distance calculation unit 201 will be described here.

  FIG. 4A is a diagram schematically illustrating a parallax image calculated by the distance calculation unit 201. It has been described that the parallax value can be converted into a distance value by the above equation (2). The distance value at this time is the z coordinate in the aforementioned reference coordinate system. Furthermore, if the horizontal position (x coordinate) and the vertical position (y coordinate) in the reference coordinates are known, the three-dimensional position of the user can be specified. The position in the horizontal direction and the vertical direction can be obtained by back-projecting in the three-dimensional space using the coordinates on the image corresponding to the user and the distance value converted from the parallax value in the parallax image. Hereinafter, a method of extracting a region corresponding to a user position in a parallax image, and back projection into a three-dimensional space using the parallax value of the region, the user's horizontal position (x coordinate) and vertical direction A method for acquiring the position (y coordinate) will be described.

  First, a method for obtaining an area corresponding to a user position on a parallax image and its coordinates will be described. Specifically, an area where the user exists is obtained by using the video of the reference camera (2) 103-2 and taking the difference between the image of only the background where the user is not shown and the image where the user is shown. To extract. The result is acquired as a user area mask (hereinafter referred to as “user area mask image”). A background image without a user can be obtained, for example, by taking a picture without the user when the system is activated. The captured background image may be held in a storage area (memory) (not shown) in the control unit 104. Such a method is a general method called background difference in image processing.

Here, n is the total number of pixels whose mask value is 1 in the user mask image. (X _i , Y _i ) indicates the coordinates of the pixel whose i-th mask value is 1. Here, since the camera used for the background difference and the camera used as the reference for the parallax image are the same, the user area mask image and the parallax image have a common coordinate axis. As described above, the region (coordinates) corresponding to the user position in the parallax image can be specified.

  Next, a method of back projecting to a three-dimensional space using a parallax value and acquiring a user's three-dimensional coordinates will be described with reference to FIG. The position of the center of gravity (user position) calculated earlier is the coordinate position 403 in FIG. Since the coordinate position 403 on the two-dimensional image and its parallax value can be known, the distance Z to the user can be calculated from the above equation (2). Since ΔOPA and ΔOP′A ′ have a similar relationship, the three-dimensional position P ′ (X, Y, Z) of the user can be calculated as follows.

  As described above, the user's standing position (user coordinates) in the reference coordinate system can be calculated. Here, the user position in the parallax image is calculated based on the background difference, but the present invention is not limited to this method. For example, a face area may be detected from the image using a general-purpose face detection process, and the same process as described above may be performed.

  Next, the determination process performed by the normal projection availability determination unit 203 will be described in detail with reference to FIGS.

  FIG. 5 is a view of FIG. 1 taken from directly above, and shows a projector 102, a multi-camera 103, a user 105, and a projection plane 101 that are a part of the projection device 107. Here, the description of the control unit 104 is omitted. A position 502 indicates a point at which visual information is projected by the projection device 107 (hereinafter referred to as “real object drawing position”). A broken line 501 indicates a normal line of the projection plane 101 at the projection point. The normal projection propriety determination unit 203 determines whether normal projection is possible according to the state of the surface of the on-object drawing position 502. Specifically, it is determined whether or not the user 105 is positioned in the direction of regular reflection from the projector 102 toward the on-object drawing position 502 (hereinafter referred to as “reflection determination”), and on-object drawing. Two types of determination (hereinafter, referred to as “concave / convex determination”) are performed as to whether or not the projection surface around the position 502 is uneven more than a predetermined condition. The normal projection availability determination unit 203 determines whether projection is possible based on these two types of determination results.

  FIG. 6 is a diagram schematically showing a parallax image (FIG. 6A) and a captured image of the reference camera (FIG. 6B) in the positional relationship as described in FIG. 5. A point 603 (hereinafter referred to as “point T”) in the parallax image (FIG. 6A) and a point 602 in the captured image (FIG. 6B) indicate the same position. This corresponds to the real object drawing position 502. Also, reference numerals 601, 605, and 606 in FIG. 7B correspond to 101, 105, and 106 in FIG.

Here, T _ref is a parameter for reflection determination, and is a positive constant. For example, it is set to 15 degrees. Subsequently, the unevenness determination will be described. The unevenness determination is performed using a parallax value in the vicinity including a distance 603 corresponding to the drawing position 502 on the real object (which may be a distance value but is not described). The determination formula for the unevenness determination is as follows.

Here, T _bump is a parameter for determining unevenness and is a positive integer. The setting can be determined according to how far the parallax of one pixel corresponds. For example, a parallax amount with a distance of about 10 cm may be set.
As described above, when the reflection determination result JudgeF _ref (T) and the unevenness determination result JudgeF _bump (T) are obtained, the normal projection propriety determination unit 203 performs the determination as follows.

  The normal projection availability determination unit 203 generates a determination result as described above and outputs the result. In the above formula (10), the final determination is performed using both the reflection determination and the unevenness determination, but the final determination may be performed using only one of them.

  Next, a method for controlling the projection position performed by the drawing unit 204 will be described in detail. The drawing unit 204 controls the projection position when “normal projection is impossible” is determined in the normal projection propriety determination 203 (when JudgeF (T) is 1 or TRUE). When the normal projection propriety determination unit 203 determines “normal projection is possible” (when JudgeF (T) is 0 or FALSE), the normal projection position is projected as it is. Hereinafter, the case of “normal projection impossible” will be described.

  The drawing unit 204 receives object information and link information from the object input unit 205. Here, object information and link information will be described with reference to FIG. The object information 800 is information including an object ID 801, an object type 802a, an object argument length 802b, an object argument 803c, an object position 804, and a link ID 805, and is information indicating the drawing contents of the object. On the other hand, the link information 810 is information composed of a link ID 811, a link length 812a, a link-object ID (s) 813b, and a movement shift amount 814, and indicates link information between objects. Hereinafter, individual items will be described.

  First, the items of the object information 800 will be described. The object ID 801 is a unique ID set for each drawing object. The object type 802a is information indicating the type of object, and is an item indicating the type of visual information such as a straight line, a curve, a point, a character, a rectangle, an image, and a moving image. The object argument length 802b is an item for designating the length of an argument that varies depending on the type of object, and determines the storage size of 803c. The object argument 803c is an item for storing an argument specific to the object type 802a. For example, the object color, line thickness, start point and end point, curvature, character code, string code, image size, image / video data storage destination, etc. are necessary arguments for drawing the object. The object position 804 is information indicating the drawing position of the object. A link ID 805 is an item for storing link information of related objects, and stores the link ID of the link information 810. The objects linked to each other are assumed to make a common change (change of drawing position) when changing the layout. The link information is not essential information, but layout changes of related objects can be suppressed when a layout change described later is performed. Subsequently, items of the link information 810 will be described. The link ID 811 is a unique ID given for each piece of link information. The link length 812a is an item that determines the storage size of 813b that stores the IDs of related objects. The link-object ID (s) 813b is an item for storing the ID of a related object (an object group that is moved in the same manner when the layout is changed). The movement shift amount 814 is an item for storing the movement amount of the entire linked object. The movement shift amount 814 of the link information 810 is set to 0 at the time of input. A movement amount is set when the layout is changed, which will be described later, and all linked objects are shifted and drawn by the same movement amount. Thereby, it is possible to draw without breaking the layout between linked objects. Regarding the change of the object layout, the object is moved under a predetermined condition, and the above-described projection propriety determination is repeated at the movement destination, and if there is no problem, the object may be moved there. The predetermined condition is to control the movement amount to be as small as possible from the original projection position. As a specific method, the distance from the original drawing position to the new destination may be controlled to be as small as possible. Further, a layout changing method performed by the layout changing unit described in Patent Document 1 may be used. As the evaluation value at this time, the determination result of the normal projection availability determination unit 203 may be used for a plurality of movement destinations. Further, a mechanism may be used in which a user instruction is input with respect to the movement destination described in Patent Document 1 and the movement destination is determined based on the result.

<Flowchart>
Next, the processing flow of the control unit 104 will be described with reference to FIGS. 9 and 2.
(Step S101)
The control unit 104 captures video captured by the multi-camera 103 connected to the outside and inputs the video to the distance calculation unit 201. The distance calculation unit 201 inputs a video and calculates a parallax image by the method described above. Thereafter, the process proceeds to step S102.

(Step S102)
The distance calculation unit 201 calculates a distance value based on the calculated parallax image and camera parameters, and outputs the result to the user standing position calculation unit 202 and the normal projection availability determination unit 203. Thereafter, the process proceeds to step S103.

(Step S103)
The user standing position calculation unit 202 inputs the parallax image and the distance value from the distance calculation unit 201, and calculates the user standing position by the method described above. The user standing position calculation unit 202 outputs the calculated result to the normal projection availability determination unit 203. Thereafter, the process proceeds to step S104.

(Step S104)
The normal projection availability determination unit 203 inputs the user's standing position from the user standing position calculation unit 202, and further receives the parallax image and the distance value from the distance calculation unit 201, and performs the above-described reflection determination (object position). (It is determined at a position obtained by adding a movement shift amount 814 to 804). If the normal projection propriety determination unit 203 determines that normal projection is possible by the reflection determination, the process proceeds to step S105. When the normal projection propriety determination unit 203 determines that normal projection is not possible by the reflection determination, the process proceeds to step S106.

(Step S105)
Subsequently, the normal projection availability determination unit 203 performs the above-described unevenness determination. If the normal projection propriety determination unit 203 determines that normal projection is possible by the unevenness determination, the process proceeds to step S107. If the normal projection propriety determination unit 203 determines that normal projection is not possible through the unevenness determination, the process proceeds to step S106.

(Step S106)
When the normal projection propriety determination unit 203 determines that normal projection is not possible, the drawing unit 204 changes the projection position described above, returns to step S104, and repeats the above-described processing.

(Step S107)
The drawing unit 204 performs drawing when the normal projection availability determination unit 106 determines that normal projection is possible.

  With the above processing, when it is predicted that the projection content is difficult to see according to the standing position of the user who is viewing the visual information, the projection position can be changed and the drawing can be performed.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to FIGS.

<Appearance example of the device>
FIG. 10 shows an appearance of a projection apparatus according to the second embodiment of the present invention that can display an image superimposed on an actual object, and displays the actual object using the apparatus, so that the user can perform work. It is the figure which showed the state currently performed typically. This embodiment is different from the first embodiment in that the visual information projected on the real object 101 includes pointer (1001) information whose projection position is important. The user 105 is working while viewing information (projection information 106 and pointer information 1001) projected onto the real object 101. As described above, FIG. 10 is a diagram showing an example of a scene using the projection apparatus of the present invention.

<Block configuration example>
The functional block diagram and processing contents of the projection apparatus in the present embodiment are the same as those in the first embodiment (see FIG. 2). The difference from the first embodiment is that an “object movement availability” item is added to the object information 800 (FIG. 8) of the first embodiment. Object information used in the present embodiment will be described with reference to FIG.

The object information 1100 is information including an object ID 1101, an object type 1102 a, an object argument length 1102 b, an object argument 1103 c, an object position 1104, an object movement availability 1105, and a link ID 1106, and is information indicating the drawing contents of the object. Yes. On the other hand, the link information is the same as in the first embodiment.
Hereinafter, each item of the object information 1100 will be described.

The object ID 1101 is a unique ID set for each drawing object.
The object type 1102a is information indicating the type of object, and is an item indicating the type of visual information such as, for example, a straight line, a curve, a point, a character, a rectangle, an image, and a moving image. The object argument length 1102b is an item that specifies the length of an argument that varies depending on the type of object, and determines the storage size of 1103c. The object argument 1103c is an item for storing an argument specific to the object type 1102a. For example, the object color, line thickness, start point and end point, curvature, character code, string code, image size, image / video data storage destination, etc. are necessary arguments for drawing the object. The object position 1104 is information indicating the drawing position of the object.

  The object movement permission / prohibition 1105 is a flag that controls whether or not to permit the layout change in the item added in the present embodiment. When this flag is set (“unmovable”), the projection position is not changed even if the normal projection availability determination unit 203 determines that normal projection is impossible. Details will be described later. The link ID 1106 is an item for storing the link information of the related object, and stores the link ID of the link information. The objects linked to each other are assumed to make a common change (change of drawing position) when changing the layout. The link information is not essential information, but layout changes of related objects can be suppressed when a layout change described later is performed.

<Flowchart>
The processing flow of the control unit 104 will be described with reference to FIGS.
(Step S201)
The control unit 104 captures video captured by the multi-camera 103 connected to the outside and inputs the video to the distance calculation unit 201. The distance calculation unit 201 inputs a video and calculates a parallax image by the method described above. Thereafter, the process proceeds to step S202.

(Step S202)
The distance calculation unit 201 calculates a distance value based on the calculated parallax image and camera parameters, and outputs the result to the user standing position calculation unit 202 and the normal projection availability determination unit 203. Thereafter, the process proceeds to step S203.

(Step S203)
The user standing position calculation unit 202 inputs the parallax image and the distance value from the distance calculation unit 201, and calculates the user standing position by the method described above. The user standing position calculation unit 202 outputs the calculated result to the normal projection availability determination unit 203. Thereafter, the process proceeds to step S204.

(Step S204)
The normal projection propriety determination unit 203 determines whether or not to perform normal projection propriety according to the object movement propriety item 1105 stored in the object information among the object information and the link information input from the object input unit 205. The normal projection propriety determination unit 203 advances the process to step S208 because the drawing position cannot be changed when the value of the object movement propriety item 1105 is “impossible to move”. If the value of the object movement permission / prohibition item 1105 is “movable”, the normal projection propriety determination unit 203 advances the process to step S205.

(Step S205)
The normal projection availability determination unit 203 inputs the user's standing position from the user standing position calculation unit 202, and further receives the parallax image and the distance value from the distance calculation unit 201, and performs the above-described reflection determination (object position). 1104 is determined at a position obtained by adding a movement shift amount 814). If the normal projection propriety determination unit 203 determines that normal projection is possible by the reflection determination, the process proceeds to step S206. When the normal projection propriety determination unit 203 determines that normal projection is impossible by the reflection determination, the process proceeds to step S207.

(Step S206)
Subsequently, the normal projection propriety determination unit 203 performs the above-described unevenness determination (determination is performed at a position obtained by adding the movement shift amount 814 to the object position 1104). If the normal projection propriety determination unit 203 determines that normal projection is possible by the unevenness determination, the process proceeds to step S208. If the normal projection propriety determination unit 203 determines that normal projection is not possible by the unevenness determination, the process proceeds to step S207.

(Step S207)
When the normal projection propriety determination unit 203 determines that normal projection is not possible, the drawing unit 204 changes the projection position described above (sets the movement amount to the movement shift amount 814), returns to step S205, and repeats the processing.

(Step S208)
When the normal projection propriety determination unit 203 determines that normal projection is possible, the drawing unit 204 performs drawing.

  As described above, according to the standing position of the user who is viewing the visual information, the projection position can be changed when it is predicted that the projection content is difficult to see. At the same time, in the case of visual information that has a problem in changing the projection position, it can be controlled to display it as it is.

(Third embodiment)
Next, a third embodiment of the present invention will be described with reference to FIGS.

<Appearance example of the device>
FIG. 13 shows an example of the appearance of a projection apparatus according to a third embodiment of the present invention that can superimpose and display an image on a real object, and a terminal that controls projection contents from a remote location, and the projection apparatus. It is the figure which showed typically a mode that it displayed on a real object using and the user is working.

A difference from the embodiments described above is that there is a terminal 1301 (hereinafter referred to as a “remote terminal”) that can control projection contents from a remote location.
The remote terminal 1301 has a function and apparatus for performing wireless communication such as Wi-Fi (Wireless Fidelity), and can transmit visual information to the projection apparatus 107 from a remote place. The display surface 1302 of the remote terminal 1301 is a touch panel type display (capacitance type, pressurization type or any other type), and can be used to manipulate visual information by touching the screen, The pointer shown can be manipulated. On the display surface 1302, an image of a work site where the projection device 107 is located is displayed. The real object 101 is displayed as 1305, the user 105 is displayed as 1306, the pointer 1001 is displayed as 1303, and the projection information 106 is displayed as 1304. It is a figure which shows the scene where the veteran is giving the instruction | indication from the remote site using the remote terminal 1301. FIG.

<Block configuration example>
A functional block diagram of the present embodiment will be described with reference to FIG. 14 and FIG. FIG. 14 is a block diagram relating to the projection device 107, and FIG. 15 is a block diagram relating to the remote terminal 1301.

  First, the projection device 107 will be described. The multi-camera 103 and the projector 102 connected to the control unit 1401 are the same as those in the first embodiment. A portion 1402 indicated by a dotted line inside the control unit 1401 is the same as that of the first embodiment, and thus the description thereof is omitted. The blocks added to the control unit 1401 are an encoding unit 1403, a transmission unit 1404, a decoding unit 1405, and a reception unit 1406. The transmission unit 1404 and the reception unit 1406 are connected to an external network (Internet, in-house LAN, etc.) (the connection method may be either wireless or wired), and data transmission / reception with a remote terminal 1301 described later It can be performed.

  Next, the remote terminal 1301 will be described. As shown in FIG. 15, the remote terminal 1301 has a remote terminal control unit 1501 and a remote terminal input / output unit 1502. The remote terminal control unit 1501 includes a receiving unit 1503, a decoding unit 1504, a control unit 1505, an encoding unit 1506, and a transmitting unit 1507. The remote terminal input / output unit 1502 includes a display 1507 and an input unit 1508. The display 1507 and the input unit 1508 in the remote terminal input / output unit 1502 may be integrally formed, or may be a so-called touch panel display.

<Content of processing>
Subsequently, regarding the contents of processing in each block, only the parts different from the first embodiment will be described. First, the projection device 107 will be described, and then the remote terminal 1301 will be described.

  The control unit 1401 of the projection device 107 captures the video captured by the multi-camera (2) 103-2 into the encoding unit 1403. The encoding unit 1403 compresses the input video to a code with a smaller data amount (for example, the international standard video compression standard H.264) and outputs the encoded data to the transmission unit 1404. The transmission unit 1404 receives the encoded data from the encoding unit 1403, and is encoded to the outside (for example, the remote terminal 1301) according to the communication standard (TCP / IP, UDP, SIP, etc.) through the network. Send data. The control unit 1401 takes in data transmitted from the outside (for example, the remote terminal 1301) into the reception unit 1406. The data received at this time is data obtained by encoding object information and link information. When receiving the encoded data sent thereto, the receiving unit 1406 outputs the data to the decoding unit 1405. The decoding unit 1405 receives the encoded data from the receiving unit 1406, decodes the object information and the link information (processing opposite to the compression processing performed by the encoding unit 1506 of the remote terminal 1301 described later), and outputs the result. The data is output to the object input unit 205. In 1402, the control unit 1401 inputs the multi-camera video and the object information, and generates a projection video according to the above-described method (the first embodiment or the second embodiment). The projector 102 projects the generated projection video.

  Next, the remote terminal 1301 will be described. The data sent from the external network is taken into the receiving unit 1503 inside the remote terminal control unit 1501, and the data corresponding to the encoded data generated by the encoding unit 1403 is extracted from the data and extracted. The encoded data is output to the decoding unit 1504. The decoding unit 1504 receives the encoded data from the receiving unit 1503, performs a process reverse to the compression process performed by the above-described encoding unit 1403, and decodes the video data. The decoding unit 1504 outputs the decoded video data to the display 1507 inside the remote terminal input / output unit 1502. The display 1507 displays the video input from the decoding unit 1504. The input unit 1508 accepts an external touch input operation and outputs the coordinate information to the control unit 1505. The control unit 1505 receives information touched by the input unit 1508 and receives object information and link information from a device (for example, the above-described keyboard, mouse, or external storage device) that inputs object information (not shown). . The control unit 1505 changes the content of the object information (for example, object type, position information, object argument, etc.) based on the information input from the input unit 1508, and outputs the result to the encoding unit 1506. The encoding unit 1506 receives the object information and the link information, and compresses (for example, ZIP compression) into a code with a smaller data amount. The encoding unit 1506 outputs the compressed encoded data to the transmission unit 1507. The transmission unit 1507 receives the encoded data from the encoding unit 1506, and transmits the encoded data to the outside (for example, the control unit 1401 of the projection apparatus) on the communication standard (TCP / IP, SIP, etc.) via the network.

<Flowchart>
In the third embodiment, a processing flow of the projection device 107 and the remote terminal 1301 will be described with reference to FIGS. 16 and 17. First, the processing flow of the projection apparatus 107 will be described with reference to FIG.

(Step S301)
The projection device 107 starts drawing processing, reception processing, and transmission processing in parallel, and proceeds to step S302, step S304, and step S306.

(Step S302)
The projection apparatus 107 performs processing corresponding to steps S101 to S107 in the flowchart in the first embodiment described above, or processing corresponding to steps S201 to S208 in the flowchart in the second embodiment. Thereafter, the process proceeds to step S303.

(Step S303)
The projection device 107 determines whether or not to end the drawing process by an overall control unit that controls the entire device (not shown). When terminating the drawing process, the overall control unit also terminates other reception processes and transmission processes. The overall control unit repeats the return process in step S302 when the drawing process is not terminated.

(Step S304)
The projection device 107 receives data sent from the outside by a receiving unit 1406 inside the projection device 107. Upon receiving the data, the receiving unit 1406 takes out data obtained by encoding the object information and the link information, and outputs the data to the decoding unit 1405. When receiving the encoded data from the receiving unit 1406, the decoding unit 1405 decodes the data and separates the object information and the link information. The decryption unit 1405 outputs the object information and link information to the object input unit 205. Thereafter, the process proceeds to step S305.

(Step S305)
The projection device 107 determines whether or not to end the reception process by an overall control unit that controls the entire device (not shown). When terminating the reception process, the overall control unit also terminates other drawing processes and transmission processes. If the overall control unit does not terminate the reception process, it repeats the return process of step S304.

(Step S306)
The projection device 107 takes in the video of the camera (2) 103-2 and inputs it to the encoding unit 1403. The encoding unit 1403 compresses and encodes the input camera video. The encoding unit 1403 outputs the encoded data to the transmission unit 1404. A transmission unit 1404 receives encoded data from the encoding unit 1403, processes the data into communicable data, and transmits the data to the outside of the projection device 107. Thereafter, the process proceeds to step S307.

(Step S307)
The projection device 107 determines whether or not to end the transmission process by an overall control unit that controls the entire device (not shown). When terminating the transmission process, the overall control unit also terminates other drawing processes and reception processes. If the overall control unit does not end the transmission process, the overall control unit repeats the return process of step S306.

Next, the processing flow of the remote terminal 1301 will be described with reference to FIG.
(Step S401)
The remote terminal 1301 starts reception / drawing processing and touch input / transmission processing in parallel, and proceeds to step S402 and step S405.
(Step S402)
The remote terminal 1301 receives data sent from the outside by an internal receiving unit 1503. When receiving the data, the receiving unit 1503 takes out the encoded video data and outputs it to the decoding unit 1504. When receiving the encoded video data from the receiving unit 1503, the decoding unit 1504 decodes the data and outputs the result to the display 1507. Thereafter, the process proceeds to step S403.

(Step S403)
Upon receiving the video data from the decoding unit 1504, the display 1507 displays the video on the display. Thereafter, the process proceeds to step S404.

(Step S404)
The remote terminal 1301 determines whether or not to end the reception / drawing process by an overall control unit that controls the entire terminal (not shown). When terminating the reception / drawing process, the overall control unit also terminates other touch input / transmission processes. If the reception / drawing process is not terminated, the overall control unit returns to step S402 and repeats the process.

(Step S405)
The input unit 1508 of the remote terminal 1301 accepts touch input from the outside and outputs the information to the control unit 1505. The control unit 1505 changes the contents of the object information (for example, object type, position information, object argument, etc.), and outputs the result to the encoding unit 1506. Thereafter, the process proceeds to step S406.

(Step S406)
The encoding unit 1506 receives object information and link information from the control unit 1505, and compresses the code into a code with a smaller data amount. The encoding unit 1506 outputs the compressed encoded data to the transmission unit 1507. The transmission unit 1507 receives the encoded data from the encoding unit 1506, and transmits the encoded data to the outside (for example, the control unit 1401 of the projection apparatus) on the communication standard via the network. Thereafter, the process proceeds to step S407.

(Step S407)
The remote terminal 1301 determines whether or not to end the touch input / transmission process by an overall control unit that controls the entire terminal (not shown). When terminating the touch input / transmission process, the overall control unit also terminates other reception / drawing processes. If the touch input / transmission process is not terminated, the overall control unit repeats the return process in step S405.

According to the present embodiment, in a projection device that can project visual information onto a real object, it is expected that the projection content is difficult to see depending on the standing position of the user who is viewing the visual information. It is possible to change the projection position.
Further, in the case of visual information having a problem in changing the projection position, control can be performed so that the display is performed at the position as it is.

<About the first to third embodiments>
In each of the above-described embodiments, the configuration and the like illustrated in the accompanying drawings are merely examples, and are not limited thereto, and can be appropriately changed within a range in which the effect of the present invention is exhibited. is there. In addition, various modifications can be made without departing from the scope of the object of the present invention. For example, a projection processing and control method in the projection apparatus, a program for causing a computer to execute these methods, a computer-readable recording medium for recording the program, and the like, and the projection processing and the control method are processed by hardware. An integrated circuit or the like may be used.

  In the description of each of the above embodiments, each component for realizing the function is described as being a different part, but it should actually have a part that can be clearly separated and recognized in this way. It doesn't have to be. The image projection apparatus that implements the functions of the above embodiments may configure each component for realizing the function using, for example, different parts, or may configure all the components. It may be mounted on one LSI. That is, what kind of mounting form should just have each component as a function.

In addition, a program for realizing the functions described in the above embodiments is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. Processing may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices.
Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the above-described functions, or may be a program that can realize the above-described functions in combination with a program already recorded in a computer system.

  The present invention can be used as a video projection device.

(Appendix) The present invention includes the following disclosure.
(1)
In a projection device that projects an image on a subject including a projection surface,
A distance calculation unit for calculating a distance from the projection device to the subject;
A user position detector for detecting a three-dimensional position of a user who uses the projection device;
Based on the distance to the subject calculated by the distance calculation unit and the position of the user detected by the user position detection unit, it is possible to normally project an image from the projection device toward the projection plane. A normal projection propriety determining unit that determines whether or not it is possible;
An object input unit for inputting information of an object to be projected onto the projection device;
A projection apparatus comprising: a drawing unit configured to change a projection position according to a determination result of the normal projection availability determination unit.
According to the present invention, in a projection device capable of projecting visual information onto a real object, a suitable projection position can be set according to the standing position of the user who is viewing the visual information.
(2)
The projection apparatus according to (1), wherein the object information input to the object input unit includes an item for designating whether movement is possible or not.
According to the present invention, it is possible to change the projection position according to the standing position of the user who is viewing the visual information when the projection content is expected to be difficult to see. At the same time, in the case of visual information that has a problem in changing the projection position, it can be controlled to display it as it is.
(3)
When changing the projection position according to the determination result of the normal projection enable / disable determining unit, the display position of the object is controlled according to an item for specifying whether or not to move that is included in the object information. The projection apparatus according to 2).
Whether or not the object can be moved is a flag for controlling whether or not to permit the change of the layout. Depending on the standing position of the user who is viewing the visual information, the projection position can be changed when it is expected that the projection content is difficult to see.
(4)
The projection according to (2) or (3), wherein, in the item for designating whether movement is possible or not, included in the object information, whether to move the pointer indicating the indicated position is set for the object to be drawn. apparatus.
When this flag is set (“unmovable”), it is possible to prevent the projection position from being changed even if the normal projection propriety determination unit determines that normal projection is impossible.
(5)
The determination performed by the normal projection propriety determination unit is:
The projection apparatus according to any one of (1) to (4), wherein either one of reflection determination or unevenness determination, or both is performed.
In the case of visual information that has a problem in changing the projection position, it can be controlled to display at the position as it is.
(6)
In a projection processing method using a projection device that projects an image on a subject including a projection surface,
A distance calculating step of calculating a distance from the projection device to the subject;
A user position detecting step for detecting a three-dimensional position of a user who uses the projection device;
Based on the distance to the subject calculated by the distance calculating step and the position of the user detected by the user position detecting step, the image is normally projected from the projection device toward the projection plane. Normal projection propriety determination step for determining whether or not it is possible;
An object input step for inputting information of an object to be projected onto the projection device;
A projection processing method comprising: a drawing step of projecting by changing a projection position according to a determination result of the normal projection propriety determination step.
(7)
A program for causing a computer to execute the projection processing method according to (6) above.
(8)
The computer-readable recording medium which records the program as described in said (7).

DESCRIPTION OF SYMBOLS 101 ... Real object, 102 ... Projector, 103 ... Multi camera, 104 ... Control part, 105 ... User, 106 ... Projection information, 107 ... Projection apparatus, 201 ... Distance calculation part, 202 ... User standing position calculation part, 203 ... normal projection propriety determination unit, 204 ... drawing unit, 205 ... object input unit, 206 ... projection input device.

Claims (6)

A projection device that projects an object onto a projection plane,
A ranging device,
A user position detection unit for detecting a three-dimensional position of the user;
Judgment whether or not the projection of the object to a certain position on the projection plane causes regular reflection in the direction toward the three-dimensional position of the user based on the measurement result of the distance measuring device And
A controller that controls a projection position of the object on the projection plane so as to avoid a projection position that the determination unit determines to cause regular reflection in a direction toward the three-dimensional position of the user. A projection apparatus.   The determination unit performs regular reflection in a direction in which the inclination of the projection surface at the certain position is directed to the three-dimensional position of the user when projecting to the certain position, based on the measurement result of the distance measuring device. The projection apparatus according to claim 1, wherein it is determined whether or not it is generated. An input unit for inputting object information including object position information for specifying the position of the object;
When the determination unit determines that the projection of the object to the position specified by the object position information causes specular reflection in a direction toward the three-dimensional position of the user, the control unit The projection apparatus according to claim 1, wherein a change process is performed to change the projection position to a position different from a position specified by the object position information. The object information further includes object movement availability information for designating whether the object can be moved,
The projection apparatus according to claim 3, wherein the control unit prohibits the change process when the object movement availability information specifies immovability.   5. The projection apparatus according to claim 4, wherein when the object is a pointer representing a designated position on the projection plane, the object movement availability information is designated as immovable. 6. A projection device that projects an object onto a projection plane,
A ranging device,
A normal projection propriety determination unit that determines whether or not the object can be normally projected to a certain position on the projection plane based on the measurement result of the distance measuring device;
An input unit for inputting object information including object position information for specifying the position of the object;
When the normal projection propriety determining unit determines that the object cannot be normally projected onto the position specified by the object position information, the projection position of the object is different from the position specified by the object position information. And a control unit that performs a change process for changing the object to a position where the object can be normally projected .
The object information further includes object movement availability information for designating whether the object can be moved,
The control unit prohibits the change processing when the object movement availability information specifies immovability.

Images