JP6237028B2 - Projection apparatus, projection method, and information processing system - Google Patents

Description

  The present invention relates to a projection apparatus, a projection method, and an information processing system.

  2. Description of the Related Art Conventionally, a technique for operating a projection apparatus by moving a pointing position using a pointing device such as an electronic pen with respect to a projection image projected on a projection surface such as a screen by a projection apparatus such as a projector is known. ing. On the other hand, there is a technique for operating the projection apparatus without using a pointing device. For example, a gesture that is a predetermined operation of the user is recognized by a camera, and an operation corresponding to the gesture is executed on the projection apparatus. Furthermore, when an operation for operating an operation target such as a button included in the projection image is performed by the user, this operation is recognized by the camera, and processing corresponding to the operation of the operation target is executed on the projection apparatus. There are also things to let you. In addition, it is also possible to realize an operation according to the above-described gesture even for a technique called multi-projection in which a plurality of projectors are arranged side by side and a projection image is projected onto a projection surface that has a large screen as a whole.

  In these techniques, since the operation of the projection apparatus is executed in accordance with the pointing position, it is desirable that the current pointing position can be detected with high accuracy. As a technique for detecting the pointing position with high accuracy, for example, an image obtained by marking a predetermined portion of the projection surface is retained, and a pointing is obtained by taking a difference between the marked image and the currently projected image. Some detect the position. With this technique, the current pointing position can be detected without always displaying the marker.

  However, in the above-described prior art, in a situation where the projection apparatus is operated by recognizing the user's operation by the camera, when the imaging area representing the area that can be captured by the camera is smaller than the projection image, the operation on the projection apparatus is It may not be detected. A case where the imaging region is smaller than the projected image occurs when the distance between the projection plane and the camera is short. That is, the above-described problem occurs when a sufficient distance between the projection surface and the camera cannot be secured. This is a problem that also occurs when a projection image is projected onto a projection screen of a large screen by multi-projection.

  The present invention has been made in view of the above, and a projection apparatus, a projection method, and information capable of detecting a plurality of instruction operations on one projection plane projected by one projection apparatus in multi-projection An object is to provide a processing system.

  In order to solve the above-described problems and achieve the object, a projection apparatus according to the present invention projects a projection image onto a projection plane in a projection apparatus connected to at least one other projection apparatus via a network. An imaging unit that captures an imaging region included in the projection plane, a detection unit that detects an instruction operation on the projection image by a user based on a captured image captured by the imaging unit, and the target to be instructed Based on the instruction target information stored in the instruction target information storage unit and the instruction target information stored in the instruction target information storage unit, there is an instruction target that cannot be included in the imaging area. When the determination unit determines that there is an instruction target that cannot be included in the imaging region, the image is captured to the other projection device. And a request unit that makes a request.

  According to one aspect of the present invention, it is possible to recognize more user gestures for a plurality of operation objects.

FIG. 1 is a diagram illustrating an example of an application image of the projection system according to the first embodiment. FIG. 2 is a diagram illustrating a hardware configuration example of the projector according to the first embodiment. FIG. 3 is a functional block diagram illustrating a device configuration example according to the first embodiment. FIG. 4 is a diagram for explaining an example of multi-projection according to the first embodiment. FIG. 5 is a diagram for explaining a calculation example of the coordinate transformation matrix. FIG. 6 is a sequence diagram illustrating an example of the flow of coordinate conversion processing according to the first embodiment. FIG. 7 is a sequence diagram illustrating an example of a flow of button position detection processing according to the first embodiment. FIG. 8 is a diagram for explaining an example of processing for capturing the entire content image (projected image) according to the first embodiment. FIG. 9 is a sequence diagram illustrating an example of the flow of the support request process according to the first embodiment. FIG. 10 is a flowchart showing an example of a flow of support request determination processing according to the first embodiment. FIG. 11 is a flowchart illustrating an example of a flow of determining a captured button image according to the first embodiment. FIG. 12 is a diagram for explaining an example of coordinate conversion when requesting support from an adjacent projector according to the first embodiment. FIG. 13 is a flowchart illustrating an example of the flow of the position determination process of the movement destination of the camera operation according to the first embodiment. FIG. 14 is a diagram for explaining a specific example of the movement destination of the camera operation according to the first embodiment. FIG. 15 is a flowchart illustrating an example of a process of determining a response to a support request according to the first embodiment. FIG. 16 is a diagram for explaining an example of capturing a button image when a response indicating suspension is generated in response to a support request. FIG. 17 is a flowchart illustrating an example of the flow of recursive support request processing. FIG. 18 is a diagram for explaining an example of capturing a button image when requesting support recursively in response to a support request. FIG. 19 is a diagram illustrating an example of capturing a button image when a supportable range is set for a support request. FIG. 20 is a diagram for explaining an example of capturing a button image when the position of the camera operation is redetermined in response to the support request. FIG. 21 is a diagram for explaining an example of capturing a button image when the camera is broken at the time of requesting assistance. FIG. 22 is a diagram for explaining an example of capturing a button image when all button images cannot be captured in response to a support request. FIG. 23 is a diagram illustrating an example in which the priority is determined according to the area of the button image. FIG. 24 is a flowchart illustrating an example of the flow of position determination processing for the movement destination of the camera operation when the button image is larger than the camera angle of view. FIG. 25 is a diagram illustrating an example in which a range in which a pointing operation can be performed is fed back. FIG. 26 is a diagram illustrating an example in which a button image that can be pointed is fed back.

  Embodiments of a projection apparatus, a projection method, and an information processing system according to the present invention will be described below with reference to the accompanying drawings. In addition, this invention is not limited by the following embodiment. Moreover, each embodiment can be combined suitably as long as the content is not contradicted.

(Embodiment 1)
[Application image of projection system]
An application image of the projection system according to Embodiment 1 will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of an application image of the projection system according to the first embodiment. The projection system is an example of an information processing system.

  As shown in FIG. 1, the projection system according to the first embodiment is a large-scale projector that includes a plurality of projection surfaces such as a projection surface A and a projection surface B by multi-projection using a plurality of projectors such as a projector A and a projector B. A projected image is projected onto a screen. That is, the projector A projects the projection image on the projection plane A, and the projector B projects the projection image on the projection plane B. Each projector is equipped with a camera. A camera mounted on each projector is used to recognize a user's gesture. In the present embodiment, since the distance between the projection plane and the projector is short, the camera mounted on each projector can capture only a part of the projection plane. In the example of FIG. 1, two button images are arranged on the projection image of the projection plane A. The button image is an object that is a target of gesture operation by the user.

  In the configuration described above, the projector A detects the button image arranged in the projection image projected onto the projection plane A by using the camera mounted on the projector A (see (1) in FIG. 1). The button image detection process is performed by each projector. Then, since the projector A cannot capture two buttons only with the camera mounted on the projector A, it assists the adjacent projector B to capture the button image arranged in the projection image projected onto the projection plane A. (See (2) in FIG. 1).

  When the projector B receives the support request from the projector A, the projector B determines whether or not the support can be performed (see (3) in FIG. 1). The determination as to whether or not the support can be performed is made based on, for example, whether or not the user has captured a button image, or whether or not the button image is included in the projection image projected by the user. In the example of FIG. 1, since the button image is not arranged in the projection image of the projection plane B projected by the projector B, the projector B determines that the support can be performed. Then, the projector B gives an answer to the effect that support can be provided to the projector A (see (4) in FIG. 1).

  Projector A, which has received an answer from projector B that support can be provided, notifies projector B of the coordinates of the button image to be supported (see (5) in FIG. 1). The coordinates of the button image to be supported are notified after being converted into the coordinates when viewed from the projector B side. Thereby, the projector B controls the orientation of the camera mounted on the projector B according to the instructed coordinates, and performs support for capturing the button image arranged in the projection image projected on the projection plane A (FIG. 1). (See (6)).

  In other words, in this embodiment, when multiple projectors are used for multi-projection, more operations can be performed while coordinating between the multiple projectors even if the projection plane and the projector are close to each other. Capture the subject. Thus, it is possible to recognize more user gestures for a plurality of operation targets.

[Hardware Configuration of Projector according to Embodiment 1]
Next, the hardware configuration of the projector according to the first embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating a hardware configuration example of the projector according to the first embodiment.

  As shown in FIG. 2, the projector 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a photographing device 14, a projection device 15, an I / I F16.

  Among these, the CPU 11 performs overall control of the projector 10. The ROM 12 stores programs and various data used in processing executed under the control of the CPU 11. The RAM 12 temporarily stores data used in processing executed under the control of the CPU 11. The photographing device 14 photographs a projection plane and a user's gesture and records them in the RAM 13. The imaging device 14 is a camera, for example. The projection device 15 projects the content image developed on the RAM 13. The I / F 16 exchanges various data with an external device or the like. The I / F 16 is an interface for, for example, infrared communication, USB (Universal Serial Bus), LAN (Local Area Network), IEEE 1394, and the like.

[Apparatus Configuration According to Embodiment 1]
Next, the apparatus configuration according to the first embodiment will be described with reference to FIG. FIG. 3 is a functional block diagram illustrating a device configuration example according to the first embodiment.

  As shown in FIG. 3, the projector 10 includes an imaging module 110, a projection module 120, a pointing operation detection module 130, and a cooperative operation control module 140. Among these, the imaging module 110 captures a projection surface or a user gesture. The projection plane imaged by the imaging module 110 is not limited to the projection plane projected by the projector 10 itself. The projection module 120 projects a content image (projected image) on the projection surface.

  The pointing operation detection module 130 analyzes the content image (projected image) projected by the projector 10, controls the orientation of the camera, detects a pointing operation from the camera image, and outputs a command corresponding to the operation. The pointing operation detection module 130 includes a camera control unit 131, a camera image storage unit 132, an imaging region position calculation unit 133, a button image recognition unit 134, a button image position calculation unit 135, and a camera operation position determination unit. 136 and a pointing operation detection unit 137.

  The camera control unit 131 controls the photographing module 110 and controls the direction of the camera. The camera image storage unit 132 stores a camera image representing an image photographed by the camera. The imaging region position calculation unit 133 calculates the four corner coordinates of the imaging region (camera angle of view) representing the region captured by the camera based on the camera image stored by the camera image storage unit 132. The camera angle of view is, for example, a square, and the imaging region position calculation unit 133 calculates the coordinates of the four corners that are the vertices of the square. If the camera angle of view is not a square, a minimum rectangle including the shape of the camera angle of view at that time may be obtained, and the four corner coordinates of the obtained square may be calculated. The calculated coordinates are coordinates with respect to the projection plane on which the projector 10 projects the content image (projected image).

  The button image recognition unit 134 searches for a button image that matches the button image stored in advance from the camera images stored by the camera image storage unit 132. The button image position calculation unit 135 calculates the four corner coordinates of the button image searched by the button image recognition unit 134. The calculated coordinates are coordinates with respect to the projection plane on which the projector 10 projects the content image (projected image). The camera operation position determination unit 136 determines the orientation of the camera from the four corner coordinates of the camera field angle calculated by the imaging region position calculation unit 133 and the four corner coordinates of the button image calculated by the button image position calculation unit 135. The destination position of the camera operation for control is determined. The pointing operation detection unit 137 detects a user gesture (pointing operation) from the camera image stored in the camera image storage unit 132 and outputs a command corresponding to the gesture to the projection module 120.

  From the analysis result of the content image (projected image) by the pointing operation detection module 130, the cooperative operation control module 140 requests other projectors adjacent to the projector 10 to assist in capturing the button image that is the user's operation target. In addition, the cooperative operation control module 140 receives a support request from an adjacent projector. Further, the cooperative operation control module 140 converts the difference in the coordinate system between adjacent projectors. The cooperative operation control module 140 includes a determination unit 141, a button list storage unit 142, a coordinate conversion unit 143, a support request unit 144, and a support reception unit 145.

  The determination unit 141 determines whether or not to request support for capturing a button image from an adjacent projector according to the number of button images detected by the pointing operation detection module 130. The determination unit 141 determines whether or not support is possible when a support request is received from an adjacent projector. In addition, the determination unit 141 determines which button on the projection plane is captured by which of the adjacent projectors. The button list storage unit 142 stores the coordinates of the button image calculated by the pointing operation detection module 130, the number of detected button images, and the like. In addition, the button list storage unit 142 stores the coordinates of the button image when supporting an adjacent projector.

  The coordinate conversion unit 143 converts its own coordinate system into the coordinate system of the adjacent projector when requesting support for capturing the button image from the adjacent projector. The support request unit 144 requests the adjacent projector to assist in capturing the button image. The support receiving unit 145 receives support for capturing a button image from an adjacent projector. Note that data transfer between adjacent projectors by the support request unit 144 and the support reception unit 145 is a daisy chain. That is, data transfer is performed in a bucket relay manner.

[Multi projection]
Next, multi-projection according to the first embodiment will be described with reference to FIG. FIG. 4 is a diagram for explaining an example of multi-projection according to the first embodiment. Multi-projection is a mechanism for projecting a large screen image as a whole by superimposing projected images projected by adjacent projectors little by little. In general, when multi-projection is realized, image alignment, hue adjustment, and the like are performed as preprocessing.

  In the example shown in FIG. 4, a case where multi-projection is performed by three projectors A, B, and C is shown. Projector A projects a projection image onto a projection plane (frame A) represented by a broken line. Projector B projects a projection image onto a projection plane (frame B) represented by a solid line. Projector C projects a projection image onto a projection plane (frame C) represented by a one-dot chain line. As a result, the projection of a large screen image shown in the upper part of FIG. 4 is realized. As shown in the upper part of FIG. 4, overlapping areas exist on the projection surfaces (frame A and frame B) of the projector A and the projector B. Similarly, overlapping areas exist on the projection planes (frame B, frame C) of projector B and projector C. The lower part of FIG. 4 represents each of the projection images projected on the projection surfaces of the frame A, the frame B, and the frame C. That is, it is a projection image projected by each projector.

  In this embodiment, a coordinate system is used to control the direction of its own camera toward the position of the button image that cannot be captured by other projectors in order to assist in capturing a button image that cannot be captured by an adjacent projector. Is converted. The coordinate system conversion will be described below.

[Conversion of coordinate system]
Next, calculation of the coordinate transformation matrix will be described with reference to FIG. FIG. 5 is a diagram for explaining a calculation example of the coordinate transformation matrix.

  FIG. 5 illustrates a case where a conversion matrix is calculated between adjacent projectors A and B. In order to calculate the transformation matrix, four or more sets of corresponding points between the projector A and the projector B are used. In order to obtain the corresponding points, images with markers are displayed at the four corners of the region where the projection surfaces of the projector A and the projector B overlap.

For example, the coordinate system of the projector A is expressed as xy. “X” is the horizontal coordinate of projector A, and “y” is the vertical coordinate of projector B. Similarly, the coordinate system of the projector B is expressed as pq. “P” is the horizontal coordinate of projector B, and “q” is the vertical coordinate of projector B. Then, the coordinates of the marker “marker 1” in the coordinate system of the projector A are (x ₁ , y ₁ ), and the coordinates of the marker “marker 1” in the coordinate system of the projector B are (p ₁ , q ₁ ). The transformation matrix “H ₂₁ ” is calculated by the mathematical formula shown below. Using four or more sets of such corresponding points (“marker 1”, “marker 2”, “marker 3”, “marker 4”), a transformation matrix is calculated.

[Coordinate transformation processing sequence]
Next, the coordinate conversion process according to the first embodiment will be described with reference to FIG. FIG. 6 is a sequence diagram illustrating an example of the flow of coordinate conversion processing according to the first embodiment.

  As shown in FIG. 6, the camera control unit 131 controls the photographing module 110 to photograph the entire content image (projected image) projected by the projector 10 (step S101). And the camera control part 131 preserve | saves the camera image obtained by imaging | photography in the camera image storage part 132 (step S102). The coordinate conversion unit 143 acquires the camera image stored in the camera image storage unit 132, and calculates a conversion matrix using the coordinates of each marker (step S103).

[Button position detection processing sequence]
Next, the button position detection process according to the first embodiment will be described with reference to FIG. FIG. 7 is a sequence diagram illustrating an example of a flow of button position detection processing according to the first embodiment.

  As shown in FIG. 7, the camera control unit 131 controls the photographing module 110 to photograph the entire content image (projected image) projected by the projector 10 (step S201). And the camera control part 131 preserve | saves the camera image obtained by imaging | photography in the camera image memory | storage part 132 (step S202). The button image recognition unit 134 acquires a camera image stored in the camera image storage unit 132 (steps S203 and S204).

  Then, the button image recognition unit 134 searches for an image that matches the previously stored button image in the acquired camera image (step S205). Note that the button image registered by the user may be acquired from the outside and used without using the button image stored in advance. The button image position calculation unit 135 calculates the four corner coordinates for the entire content image (projected image) for the button image included in the camera image searched by the button image recognition unit 134 (step S206). Then, the button image position calculation unit 135 registers the calculated four corner coordinates of the button image in the button list storage unit 142 (step S207). When a plurality of button images are searched, the button image position calculation unit 135 calculates four corner coordinates for each button image and registers them in the button list storage unit 142.

[Shooting the entire content image (projected image)]
Next, processing for capturing the entire content image (projected image) according to Embodiment 1 will be described with reference to FIG. FIG. 8 is a diagram for explaining an example of processing for capturing the entire content image (projected image) according to the first embodiment. The entire content image (projected image) is taken when the projector 10 is turned on or when the projected content image (projected image) is switched. Further, the left side of the upper part of FIG. 8 represents a default state in which the camera direction is not particularly controlled. As shown on the left in the upper part of FIG. 8, the imaging region 3 (camera angle of view) is smaller than the projected image 1 (content image) on which the button image 2 is arranged.

  The camera control unit 131 controls the orientation of the camera by controlling the photographing module 110 so as to move the imaging region 3 in the upper left direction of the projection image 1. The imaging module 110 detects the top left vertex coordinate of the projection image 1. Then, the camera control unit 131 controls the imaging module 110 so that the upper left vertex of the imaging region 3 overlaps the upper left vertex coordinate of the detected projection image 1. The imaging module 110 moves the imaging area 3 so that the upper left vertex of the imaging area 3 overlaps the upper left vertex coordinates of the projection image 1 as shown in the upper right part of FIG. 3 is photographed to generate a camera image. The generated camera image is stored in the camera image storage unit 132.

  Thereafter, the camera control unit 131 sets the upper left corner of the new imaging area 3 at the vertex coordinates on the upper right of the imaging area 3 as shown in the right side of the middle stage of FIG. 3, the left side of the middle stage of FIG. The imaging module 110 is controlled so that the vertex coordinates overlap each other, and each time the imaging area 3 is captured, a camera image is generated. Here again, the generated camera image is stored in the camera image storage unit 132.

  The camera control unit 131 repeatedly executes the above processing until all the vertex coordinates of the projection image 1 appear in the camera image. Then, the projection image 1 is shot in the order of the arrows shown on the left in the lower part of FIG. When shooting of the entire projection image 1 is completed, a camera image that covers the entire projection image 1 is stored in the camera image storage unit 132. Further, as shown on the right side in the lower part of FIG. 8, the camera control unit 131 controls the imaging module 110 to return the position of the imaging region 3 to the default state when the imaging of the entire projection image 1 is completed.

[Support request processing sequence]
Next, the support request process according to the first embodiment will be described with reference to FIG. FIG. 9 is a sequence diagram illustrating an example of the flow of the support request process according to the first embodiment. In FIG. 9, a case where projector A makes a support request to projector B is taken as an example.

  As shown in FIG. 9, in the projector A, the determination unit 141 refers to the button list storage unit 142 (step S301), acquires information on the button list, and grasps the number of button images (step S302). Then, the determination unit 141 determines whether to request support from the adjacent projector B based on the number of button images (step S303). Here, it is assumed that it is determined that the adjacent projector B requests support, and the determination unit 141 requests the support request unit 144 to request support (step S304). Accordingly, the support request unit 144 inquires of the adjacent projector B whether support is possible (step S305).

  In the projector B, the support reception unit 145 receives the support request from the projector A, and returns a response to the support request to the projector A (step S306). Here, it is assumed that a response indicating that support is possible is obtained, and in projector A, support request unit 144 determines that the response indicating that support is possible is received from projector B to determination unit 141. Notification is made (step S307). The determination unit 141 determines which button image is to be captured by which camera because support is possible (step S308).

  Then, the determination unit 141 notifies the support request unit 144 of the projector number (identification information) of the adjacent projector B and the coordinates of the button image to be captured (step S309). The support request unit 144 notifies the coordinate conversion unit 143 of the projector number of the projector B and the coordinates of the button image, and requests coordinate conversion (step S310). The coordinate conversion unit 143 performs coordinate conversion and notifies the support request unit 144 of the coordinates of the converted button image (step S311). The support request unit 144 transmits the converted button image coordinates to the projector B to the projector B (step S312). Further, the determination unit 141 notifies the camera operation position determination unit 136 of the coordinates of the button image for which no support has been requested, and the camera operation position determination unit 136 controls the camera toward the coordinates of the button image (step) S313). Also in the projector B, the capture of the button image with assistance is realized by controlling the camera based on the coordinates of the converted button image.

[Support Request Judgment Flow]
Next, with reference to FIG. 10, the flow of the support request determination process according to the first embodiment will be described. FIG. 10 is a flowchart showing an example of a flow of support request determination processing according to the first embodiment. In addition, the determination process shown in FIG. 10 represents the detail of the process in step S303 shown in FIG.

  As shown in FIG. 10, when a plurality of button images are detected (step S401: Yes), the determination unit 141 determines whether there are a plurality of adjacent projectors (step S402). At this time, if there are a plurality of adjacent projectors (step S402: Yes), the determination unit 141 determines to make a support request to the plurality of adjacent projectors (step S403).

  On the other hand, when there is only one adjacent projector (step S402: No), the determination unit 141 determines to make a support request to the adjacent projector (step S404). That is, in multi-projection, at least one projector is adjacent to the projector. In addition, when only one or one button image is detected (step S401: No), the determination unit 141 determines that no support request is to be made because it can be captured by its own projector (step S405).

  In the example shown in FIG. 10, the case where one button image fits within the camera angle of view has been described, but the present embodiment is applied even when a plurality of button images fit within the camera angle of view. Can do. For example, since the coordinates of the button image are obtained by photographing (scanning) the entire content image, the button within the camera angle of view is added to the processing in step S401 based on the coordinates and the size of the camera angle of view. It may be determined whether or not the image fits. When a plurality of button images are included in the camera angle of view and there are a plurality of button images except for the number of button images that are included in the camera angle of view, a support request may be made to an adjacent projector. On the other hand, when a plurality of button images are accommodated within the camera angle of view and no other button images exist, it is not necessary to make a support request to an adjacent projector.

[Determination flow of button image to be captured]
Next, the flow of the determination process of the button image to be captured according to the first embodiment will be described with reference to FIG. FIG. 11 is a flowchart illustrating an example of a flow of determining a captured button image according to the first embodiment. The determination process shown in FIG. 11 represents details of the process in step S308 shown in FIG. Hereinafter, a case where the number of detected button images is two will be described as an example.

  As illustrated in FIG. 11, the determination unit 141 determines whether or not an answer from a projector adjacent in the left direction can be supported with respect to its own projector (step S501). Then, when the answer from the projector adjacent in the left direction cannot be supported (step S501: No), the determination unit 141 determines whether the answer from the projector adjacent in the right direction can be supported (step S501: No). Step S502). At this time, when the answer from the projector adjacent in the right direction cannot be supported (step S502: No), the determination unit 141 determines the button image that the self captures according to the predetermined priority (step S503). ).

  In addition, when the answer from the projector adjacent in the right direction can be supported (step S502: Yes), the determination unit 141 captures the right button image in the projector adjacent in the right direction in the arrangement in the content image. And determine to capture the left button image by itself (step S504). Further, when the answer from the projector adjacent in the left direction can be supported (step S501: Yes), the determination unit 141 captures the left button image in the projector adjacent in the left direction in the arrangement of the content image. And determine to capture the right button image by itself (step S505).

[Example of coordinate transformation]
Next, with reference to FIG. 12, coordinate conversion when requesting assistance from an adjacent projector according to the first embodiment will be described. FIG. 12 is a diagram for explaining an example of coordinate conversion when requesting support from an adjacent projector according to the first embodiment. FIG. 12 shows an example of a situation in which multi-projection is performed by two projectors and two button images are arranged on the projection surface of projector B, so that projector A assists in capturing “button 1”. To

  As shown in FIG. 12, the solid line represents the projection plane of projector A, and the broken line represents the projection plane of projector B. In the coordinate system of the projector B, the coordinates of “button 1” are (30, 25). A conversion matrix is used to convert the coordinates of the “button 1” into the coordinate system of the projector A. As a result of the conversion, the coordinates of “button 1” in the coordinate system of projector A are (50, 25). Thereby, the projector A captures “button 1” by controlling the direction of the camera toward the coordinates (50, 25).

  When the projector B supports the button image arranged on the projection surface of the projector A, the x coordinate of the supporting button image may be expressed as negative in the projector B coordinate system. In the above example, for the sake of simplicity, the case where the coordinates of the center position (center of gravity) of the button image is converted has been described, but actually, the conversion processing is performed for each of the four corner coordinates of the button image.

[Camera movement destination position determination flow]
Next, with reference to FIG. 13, the position determination process of the movement destination of the camera operation according to the first embodiment will be described. FIG. 13 is a flowchart illustrating an example of the flow of the position determination process of the movement destination of the camera operation according to the first embodiment. The position determination process illustrated in FIG. 13 is started when the button image position calculation unit 135 calculates the four corner coordinates of the button image and the imaging region position calculation unit 133 calculates the four corner coordinates of the camera field angle. The

  As illustrated in FIG. 13, the camera operation position determination unit 136 acquires the four corner coordinates of the button image (step S601). In addition, the camera operation position determination unit 136 acquires the four corner coordinates of the camera angle of view (step S602). Then, the camera operation position determination unit 136 determines whether or not all the four corner coordinates of the button image are within the four corner coordinates of the camera angle of view (step S603).

  At this time, when all the four corner coordinates of the button image are not within the four corner coordinates of the camera angle of view (step S603: No), the camera operation position determination unit 136 determines the center point coordinates (center of gravity) of the button image coordinates. Is calculated (step S604). Further, the camera operation position determination unit 136 calculates the midpoint coordinates (center of gravity) of the coordinates of the camera angle of view (step S605). Then, the camera operation position determination unit 136 calculates the distance between the midpoint coordinates of the button image and the midpoint coordinates of the camera angle of view (step S606). The distance calculated here is the amount of movement of the imaging area by the camera. As a result, the camera operation position determination unit 136 determines the calculated distance as the position of the movement destination of the camera operation (step S607). On the other hand, when the four corner coordinates of the button image are all within the four corner coordinates of the camera angle of view (step S603: Yes), the camera operation position determination unit 136 does not have to perform the camera operation. Is determined not to move (step S608).

  Here, an example of the process in step S603 illustrated in FIG. 13 will be described. What is necessary is just to compare about each vertex coordinate as follows.

Comparison of the lower left vertex coordinates.
If “the x coordinate of the lower left vertex of the button image <the x coordinate of the lower left vertex of the camera angle of view”, the button image is not included in the current camera angle of view.
If “x coordinate of lower left vertex of camera angle of view <= x coordinate of lower left vertex of button image” cannot be determined, the following comparison is performed.
If “y coordinate of the lower left vertex of the button image <y coordinate of the lower left vertex of the camera angle of view”, the button image is not included in the current camera angle of view.
If “y coordinate of the lower left vertex of the camera angle of view <= y coordinate of the lower left vertex of the button image” cannot be determined, the following comparison is performed.

Top left vertex coordinate comparison.
If “the x coordinate of the upper left vertex of the button image <the x coordinate of the upper left vertex of the camera angle of view”, the button image is not included in the current camera angle of view.
If “x coordinate of upper left vertex of camera angle of view <= x coordinate of upper left vertex of button image” cannot be determined, the following comparison is performed.
If “y coordinate of upper left vertex of camera view angle <y coordinate of upper left vertex of button image”, the button image is not included in the current camera view angle.
If “y coordinate of the upper left vertex of the button image <= y coordinate of the upper left vertex of the camera field angle” cannot be determined, the following comparison is performed.

Comparison of the lower right vertex coordinates.
If “the x coordinate of the lower right vertex of the camera field angle <the x coordinate of the lower right vertex of the button image”, the button image is not included in the current camera field angle.
If “the x coordinate of the lower right vertex of the button image <= the x coordinate of the lower right vertex of the camera field angle” cannot be determined, the following comparison is performed.
If “y coordinate of the lower right vertex of the button image <y coordinate of the lower right vertex of the camera angle of view”, the button image is not included in the current camera angle of view.
If “y coordinate of the lower right vertex of the camera angle of view <= y coordinate of the lower right vertex of the button image” cannot be determined, the following comparison is performed.

Top right vertex coordinate comparison.
If “the x coordinate of the upper right vertex of the camera view angle <the x coordinate of the upper right vertex of the button image”, the button image is not included in the current camera view angle.
If “x coordinate of upper right vertex of button image <= x coordinate of upper right vertex of camera angle of view” cannot be determined, the following comparison is performed.
If “y coordinate of upper right vertex of camera view angle <y coordinate of upper right vertex of button image”, the button image is not included in the current camera view angle.
If “y coordinate of the upper right vertex of the button image <= y coordinate of the upper right vertex of the camera angle of view”, the button image is included in the current camera angle of view.

  Here, an example of processing in steps S604 to S607 will be described. As described below, the midpoint coordinates serving as the center of gravity are obtained by obtaining the midpoints in the x direction and the y direction from the respective vertex coordinates, and a value for moving between the obtained midpoint coordinates is calculated.

The midpoint coordinates of the button image.
"X coordinate =
(X coordinate of lower left vertex of button image + x coordinate of upper right vertex of button image) / 2
"Y coordinate =
(Y coordinate of the lower left vertex of the button image + y coordinate of the upper right vertex of the button image) / 2 ”

The midpoint coordinates of the current camera angle of view.
"X coordinate =
(X coordinate of the lower left vertex of the camera angle of view + x coordinate of the upper right vertex of the camera angle of view) / 2
"Y coordinate =
(Y coordinate of the lower left vertex of the camera angle of view + y coordinate of the upper right vertex of the camera angle of view) ÷ 2 ”

Calculated distance (movement amount).
"X coordinate =
X coordinate of midpoint coordinates of button image-x coordinate of midpoint coordinates of current camera angle of view "
"Y coordinate =
Button image midpoint coordinate y coordinate-current camera angle of view midpoint coordinate y "

[Specific examples of camera movement destinations]
Next, the movement destination of the camera operation according to the first embodiment will be described with reference to FIG. FIG. 14 is a diagram for explaining a specific example of the movement destination of the camera operation according to the first embodiment.

  In the example shown in the upper part of FIG. 14, the vertex coordinates of the button image 2 are (1, 9), (1, 11), (5, 9), and (5, 11). Similarly, the vertex coordinates of the imaging region 3 (camera field angle) are (6, 4), (6, 8), (12, 4), and (12, 8). The projection image 1 (content image) includes the button image 2 and the imaging region 3 having the above-described vertices.

  In the state described above, the camera operation position determination unit 136 calculates the midpoint coordinates (9, 6) that are the center of gravity of the current imaging region 3 from the vertex coordinates of the imaging region 3. In FIG. 14, the midpoint coordinates of the imaging region 3 are represented by black squares. In addition, the camera operation position determination unit 136 calculates the midpoint coordinates (3, 10) serving as the center of gravity of the button image 2 from the vertex coordinates of the button image 2. In FIG. 14, the midpoint coordinates of the button image 2 are represented by black circles.

  Then, the camera operation position determination unit 136 calculates the amount of movement from the midpoint coordinates (9, 6) of the current imaging region 3 to the midpoint coordinates (3, 10) of the button image 2. In the calculation of the movement amount, since “x direction = 3-9 = −6” and “y direction = 10−6 = + 4”, the current imaging region 3 is moved “−6” in the x direction, and y A movement amount “(−6, +4)” to be moved “+4” in the direction is obtained. The camera operation position determination unit 136 determines a movement amount “(−6, +4)” for moving the current imaging region 3 “−6” in the x direction and “+4” in the y direction. As a result, as shown in the lower part of FIG. 14, a movement result is obtained in which the coordinates that are the midpoint coordinates of the image area 3 overlap the coordinates that are the midpoint coordinates of the button image 2.

[Judgment flow for response to support request]
Next, with reference to FIG. 15, a determination process for a response to a support request according to the first embodiment will be described. FIG. 15 is a flowchart illustrating an example of a process of determining a response to a support request according to the first embodiment.

  As illustrated in FIG. 15, the support receiving unit 145 receives a support request for capturing a button image from an adjacent projector (step S701). Here, the support receiving unit 145 determines whether information is stored in the button list storage unit 142 (step S702).

  If no information is stored in the button list storage unit 142 (step S702: No), the support reception unit 145 determines whether a support request has already been received from another adjacent projector (step S703). . At this time, if the support reception unit 145 has not received a support request from another adjacent projector (step S703: No), there is no button image to be captured on its own projection plane, and the support reception unit 145 supports other adjacent projectors. Since it is not performed, a response indicating that the support is possible is returned to the projector as the support request transmission source (step S704).

  On the other hand, when receiving a support request from another adjacent projector (step S703: Yes), the support receiving unit 145 replies to the projector that transmitted the support request that the support request is suspended (step S703: Yes). Step S705). In other words, if a support request is received from another adjacent projector and the coordinate after conversion is awaited, the support request is put on hold further. If there is no actual request for support from the transmission source projector after the suspension, it may be answered that support can be provided to other adjacent projectors. In addition, when information is stored in the button list storage unit 142 (step S702: Yes), the support reception unit 145 indicates that support is impossible because the button image to be captured has already been determined. A response is made to the projector that is the source of the support request (step S706).

[Specific examples of responses to support requests (1)]
Next, with reference to FIG. 16, the capture of the button image in the case where an answer indicating that the support request is suspended is generated. FIG. 16 is a diagram for explaining an example of capturing a button image when a response indicating suspension is generated in response to a support request.

  FIG. 16 shows an example in which multi-projection is realized by five projectors A to E. It is assumed that two button images are arranged on the projection surface A of the projector A, two button images are arranged on the projection surface C of the projector C, and one button image is arranged on the projection surface E of the projector E. In the lower table of FIG. 16, the vertical axis represents the flow of time, and the horizontal axis represents the cooperative relationship between the projectors.

  First, since a plurality of button images are detected on the projection surface A, the projector A requests the adjacent projector B for assistance. Further, since a plurality of button images are detected on the projection surface C, the projector C requests the adjacent projectors B and D for assistance.

  Next, although the projector B has received the request for assistance from the projector A and the projector C, it cannot determine which projector should be supported at this time, so the projector A and the projector C are put on hold. To answer. Since the support request is not made at exactly the same timing, even when a support request is received from a certain projector, the response is made after waiting for a certain time. Further, since the projector D has received a support request only from the projector C, the projector D replies that it can support the projector C.

  Subsequently, since it has been found that the projector C can receive support from the projector D, the support request for the projector B is canceled (canceled). Further, since the support request from the projector C is canceled and only the support request from the projector A is received, the projector B replies to the projector A that support can be provided. The projector A makes a request for assistance again to the projector B, but stops the assistance request when a predetermined period has elapsed or according to the number of trials of the assistance request.

[Recursive support request processing flow]
Next, the flow of recursive support request processing will be described with reference to FIG. FIG. 17 is a flowchart illustrating an example of the flow of recursive support request processing. Note that the recursive support request processing means, for example, that repetitive support requests are made recursively when capturing of all button images is not realized by support by a single projector.

  As shown in FIG. 17, the support receiving unit 145 receives a support request from n projectors (step S801). Here, n is a natural number satisfying “n ≧ 1”. Then, the support receiving unit 145 determines whether information is stored in the button list storage unit 142 (step S802).

  If no information is stored in the button list storage unit 142 (step S802: No), the support reception unit 145 determines whether n = 1 (step S803). Note that the fact that no information is stored in the button list storage unit 142 means that there is no button image to be captured at present. At this time, when n = 1 is not satisfied (step S803: No), the support receiving unit 145 cannot capture all button images only with its own support. A request is made (step S804).

  When the support reception unit 145 has not received a response indicating that support is possible from an adjacent projector (step S805: No), it cannot capture all the button images, but the number obtained by the response + 1 A response to the projector as the support request source is returned to the effect that support of (+1 is own support) is possible (step S806). Further, when n = 1 (step S803: Yes), the support reception unit 145 can capture the button image only with its own support. A response is made to the projector (step S807). Further, when the support reception unit 145 receives an answer indicating that support is possible from an adjacent projector (step S805: Yes), all the button images can be captured, so that support is possible. Then, a response is made to the projector that requested the support (step S807).

  Further, when the information is stored in the button list storage unit 142 (step S802: Yes), the support reception unit 145 requests the n projectors for support for the n projectors because it cannot support itself (step S808). ). When the support reception unit 145 has not received a response indicating that support is possible from an adjacent projector (step S809: No), the support reception unit 145 cannot capture all the button images, but the number of units obtained in the response ( The support request source projector is answered to the effect that it is possible to support (it cannot support itself) (step S810). Further, when the support reception unit 145 receives an answer indicating that support is possible from an adjacent projector (step S809: Yes), all the button images can be captured, so that support is possible. Then, a response is made to the projector that requested the support (step S807).

[Specific examples of responses to support requests (2)]
Next, with reference to FIG. 18, the capture of a button image when requesting support recursively in response to a support request will be described. FIG. 18 is a diagram for explaining an example of capturing a button image when requesting support recursively in response to a support request.

  FIG. 18 shows an example in which multi-projection is realized by three projectors A to C. Further, it is assumed that three button images are arranged on the projection surface A of the projector A. In the lower table of FIG. 18, the vertical axis represents the flow of time, and the horizontal axis represents the cooperative relationship between the projectors.

  First, since three button images are detected on the projection surface A, the projector A requests two projectors for assistance from the adjacent projector B. Since the button image is not arranged on the projection plane B, the projector B determines that one unit can be supported. However, since the projector B cannot respond to the support request for two units from the projector A, it further requests the projector C for one unit of support. Since the button image is not arranged on the projection surface C, the projector C determines that support for one unit is possible, and replies to the projector B that support is possible.

  Subsequently, the projector B determines that it is possible to support two units including the projector B based on an answer from the projector C adjacent to the right in the direction that it can support, and can support two units. A response is given to projector A. As a result, it has been found that the projector A can obtain support from the projector B and the projector C adjacent in the right direction, so that the projector A decides to capture the button A located on the leftmost side. Further, the projector A performs coordinate conversion for converting the button B and the button C into the coordinate system of the projector B, and requests the projector B for assistance. Since it is known that the projector B can obtain support from the projector C adjacent in the right direction, the projector B itself captures the button B and decides to have the button C captured by the projector C. Again, the projector B performs coordinate conversion for converting the button C into the coordinate system of the projector C, and requests the projector C for assistance.

[Specific examples of responses to support requests (3)]
Next, with reference to FIG. 19, the capture of a button image when a supportable range is set for a support request will be described. FIG. 19 is a diagram illustrating an example of capturing a button image when a supportable range is set for a support request.

  In FIG. 19, a case where multi-projection is realized by four projectors A to D is taken as an example. It is assumed that three button images are arranged on the projection surface A of the projector A and one button image is arranged on the projection surface C of the projector C. In the lower table of FIG. 19, the vertical axis represents the flow of time, and the horizontal axis represents the cooperative relationship between the projectors. In the following, a supportable range is set as a parameter common to all projectors. The supportable range represents how many units can be supported in the connection relationship between adjacent projectors. Here, the supportable range is described as “2”.

  First, since three button images are detected on the projection surface A, the projector A requests two projectors for assistance from the adjacent projector B. Since the button image is not arranged on the projection plane B, the projector B determines that one unit can be supported. However, since the projector B cannot respond to the support request for two units from the projector A, it further requests the projector C for one unit of support.

  Subsequently, since the button image is arranged on the projection plane C, the projector C determines that it cannot respond to the support request from the projector B itself. Here, since the supportable range is “2”, the projector C responds to the projector B that the support cannot be performed without making a support request for one projector to the projector D.

  After that, the projector B determines from the response from the projector C that it can support one unit by its own support, and responds to the projector A that support for one unit is possible. Since the projector A can capture only two button images by itself and the support of the projector B, the button A to be captured is determined according to a preset priority. As a result, the projector A performs coordinate conversion of the button image to be supported by the projector B, and requests the projector B for support.

  In other words, by setting the supportable range, it is possible to prevent the request for support from propagating indefinitely, and to increase the camera control load caused by requesting the button image to be captured by a projector that is too far away. Can be prevented. For example, when the camera is pointed at the projection surface of four projectors ahead, it may be difficult to capture a preferable image because the image to be captured has a steep angle instead of a high load. .

[Specific examples of responses to support requests (4)]
Next, with reference to FIG. 20, the capture of the button image when the position of the camera operation is redetermined in response to the support request will be described. FIG. 20 is a diagram for explaining an example of capturing a button image when the position of the camera operation is redetermined in response to the support request.

  FIG. 20 shows an example in which multi-projection is realized by three projectors A to C. It is assumed that two button images are arranged on the projection plane A of the projector A and one button image is arranged on the projection plane B of the projector B. In the lower table of FIG. 20, the vertical axis represents the flow of time, and the horizontal axis represents the cooperative relationship between the projectors.

  First, since two button images are detected on the projection surface A, the projector A makes a support request for one projector to the adjacent projector B. Since the button image (button C) is arranged on the projection plane B, the projector B determines that the projector A cannot be supported, and further requests the adjacent projector C for one unit. The projector C replies to the projector B that the support is possible because the button image is not arranged on the projection surface C. In response to the response from projector C, projector B responds to projector A that support is possible.

  Subsequently, since it has been found that the projector A can obtain support for one projector from the projector adjacent in the right direction, the projector A decides to capture the button A located on the leftmost side. In addition, the projector A performs coordinate conversion of the button B and requests the projector B to support the button B.

  Here, since it is known that the projector B can support the projector C adjacent in the right direction, the button B including the button B arranged on the projection plane A and the button C arranged on the projection plane B is displayed. As a target, when capturing by itself (projector B) and projector C, it is determined which projector captures which button image. Since it is known that the projector B can receive support from the projector C adjacent in the right direction, the projector B captures the button B located on the left side, and asks the projector C to capture the button C. decide. Since the projector B receives the coordinates of the button B after coordinate conversion from the projector A, the projector B captures the button B based on the coordinates, performs the coordinate conversion of the button C for the projector C, and The projector C is requested for capture support. Accordingly, the camera control load can be reduced as a whole by resetting the camera orientation in the cooperative relationship of the projectors.

[Specific examples of responses to support requests (5)]
Next, with reference to FIG. 21, the capture of the button image when the camera is broken at the time of requesting assistance will be described. FIG. 21 is a diagram for explaining an example of capturing a button image when the camera is broken at the time of requesting assistance.

  FIG. 21 shows an example in which multi-projection is realized by three projectors A to C. Further, it is assumed that two button images are arranged on the projection surface A of the projector A. In the lower table of FIG. 21, the vertical axis represents the flow of time, and the horizontal axis represents the cooperative relationship between the projectors. In the following, an example is given of a case where various processes in advance such as shooting of the entire content image have been completed, and the camera of the projector B has failed due to some factor during operation.

  First, since two button images are detected on the projection surface A, the projector A makes a support request for one projector to the adjacent projector B. Although the button image is not arranged on the projection surface B, the projector B detects that the mounted camera is out of order and determines that the projector B cannot provide assistance. Therefore, the projector B makes a support request for one projector to the adjacent projector C.

  Subsequently, since the button image is not arranged on the projection plane C, the projector C replies to the projector B that support is possible. When the projector B receives an answer indicating that support is possible from the projector C, the projector B answers to the projector A that support for one unit is possible. Since it has been found that the projector A can obtain support from a projector adjacent in the right direction, the projector A decides to capture the button A located on the leftmost side. In addition, the projector A performs coordinate conversion of the button B and requests the projector B to support the button B. The projector B further performs coordinate conversion of the button B and requests the projector C to support the button B. Note that, as described above, the projector B has completed various types of pre-processing, and has only failed in the camera during operation. Therefore, various types of processing such as coordinate conversion can be performed.

  In other words, even when a camera mounted on a projector that has a cooperative relationship in multi-projection fails, it is possible to preferably capture a button image by recursively requesting assistance.

[Specific example of response to support request (6)]
Next, with reference to FIG. 22, the capture of the button image when not all the button images can be captured in response to the support request will be described. FIG. 22 is a diagram for explaining an example of capturing a button image when all button images cannot be captured in response to a support request.

  FIG. 22 shows an example in which multi-projection is realized by three projectors A to C. Further, it is assumed that four button images are arranged on the projection surface B of the projector B. In the table in the lower part of FIG. 22, the vertical axis represents the flow of time, and the horizontal axis represents the cooperative relationship between the projectors.

  First, since four button images are detected on the projection surface B, the projector B requests the adjacent projector A and projector C for assistance. The projector A replies to the projector B that the button image is not arranged on the projection surface A, and that one projector can be supported. Further, since the button image is not arranged on the projection surface C, the projector C replies to the projector B that one unit can be supported.

  Subsequently, the projector B recognizes from the responses from the projector A and the projector C that it can capture one button image by itself and two button images with assistance. However, since there are four button images arranged on the projection plane B, the projector B determines the buttons (three) to be captured according to the predetermined priority of the button image. Here, the priority is assumed to be higher in the order of button B, button D, button A, and button C. Thereby, the projector B determines to capture the three button images of the button B, the button D, and the button A.

  It is known that the projector B can obtain support from each of the projector A adjacent in the right direction and the projector C adjacent in the left direction. Therefore, the projector B causes the projector A to capture the leftmost button A based on the coordinates of the button B, button D, and button A to be captured, and causes the projector D to position the rightmost button D. Capture and decide to capture Button B by itself. Accordingly, the projector B requests the projector A to support the button A that has undergone the coordinate conversion, and requests the projector C to support the button D that has performed the coordinate conversion.

  In other words, when all the button images cannot be captured, the support is requested to the adjacent projector according to the position (coordinates) of the button image determined to be captured according to the priority, so that the button image is preferably captured. At the same time, the camera control load can be reduced.

[Effects of Embodiment 1]
The projector 10 requests an adjacent projector to assist the capturing of the button image based on the coordinates of the button image arranged on the projection surface projected by the projector 10 and the coordinates of the camera angle of view. Then, the projector 10 determines which button image is requested to be supported when an answer that can provide support is received. Subsequently, the projector 10 converts the coordinate system of the coordinates of the button image for which support is requested to the coordinate system of the adjacent projector, and requests support for capturing the button image together with the converted coordinates. As a result, the projector 10 can recognize more user gestures for a plurality of operation targets.

(Embodiment 2)
Although the embodiment of the projector 10 according to the present invention has been described so far, the present invention may be implemented in various different forms other than the above-described embodiment. Accordingly, different embodiments will be described with respect to (1) priority, (2) relationship between the button image and the camera angle of view, (3) feedback related to the pointing operation, and (4) configuration.

(1) Priority In the above embodiment, a case has been described where, when capturing a plurality of button images, which button image is to be captured by which projector is determined according to a predetermined priority. Such priority may be determined by the area (size) of the button image. FIG. 23 is a diagram illustrating an example in which the priority is determined according to the area of the button image. As shown in FIG. 23, when multi-projection is realized by the projector A and the projector B and a plurality of button images are arranged on the projection plane A, the area of each button image is determined from the four corner coordinates of each button image. Is calculated. Then, the priority of the button image is determined according to the calculated area. For example, the priority is assumed to be higher in descending order of the area of the button image. In the example of FIG. 23, the priority is set higher in the order of button B, button A, and button C in accordance with the area of the button image. That is, the priority is determined on the assumption that the button image having a larger area is more likely to be operated.

(2) Relationship between Button Image and Camera Angle of View In the above-described embodiment, each process when the button image fits within the camera angle of view has been described. Such a button image may not fit within the camera angle of view. FIG. 24 is a flowchart illustrating an example of the flow of position determination processing for the movement destination of the camera operation when the button image is larger than the camera angle of view.

  As shown in FIG. 24, the camera operation position determination unit 136 calculates the area of the button image and the area of the camera angle of view (step S901). Then, the camera operation position determination unit 136 determines whether the area of the button image is larger than the area of the camera view angle (step S902). At this time, when the area of the button image is larger than the area of the camera angle of view (step S902: Yes), the camera operation position determination unit 136 calculates a midpoint coordinate serving as the barycenter of the button image ( Step S903). Further, the camera operation position determination unit 136 calculates the midpoint coordinates that are the center of gravity of the camera angle of view (step S904).

  Then, the camera operation position determination unit 136 calculates the distance between the midpoint coordinates of the button image and the midpoint coordinates of the camera angle of view (step S905). The distance calculated here is the amount of movement of the imaging area by the camera. Accordingly, the camera operation position determination unit 136 determines the calculated distance as the position of the movement destination of the camera operation (step S906). On the other hand, when the area of the button image is smaller than the area of the camera angle of view (step S902: No), the camera operation destination position determination unit 136 executes the position determination process (normal process) shown in FIG. (Step S907). In other words, even if there is a button image that does not fit within the camera angle of view, it is possible by moving the imaging area so that the midpoint coordinates of the button image and the midpoint coordinates of the camera angle of view overlap. The user operability can be improved as much as possible.

(3) Feedback related to pointing operation In the above embodiment, when the area of the imaging region is smaller than the area of the button image, in order to facilitate the user operation, The case where the movement amount for overlapping the point coordinates is calculated has been described. In the present embodiment, the imaging area may be fed back in order to facilitate user operation.

  FIG. 25 is a diagram illustrating an example in which a range in which a pointing operation can be performed is fed back. In the example shown in FIG. 25, the projection image 1 includes a button image 2 and an imaging region 3. The example shown in FIG. 25 represents a state when the imaging region 3 is moved to the position of the button image 2.

  As shown in FIG. 25, when the imaging area 3 is moved, the imaging area 3 is displayed with a thick frame or the like so that the user can recognize the range in which the user can detect the pointing operation. In the embodiment described above, the pointing operation by the user is detected without making the user aware of the imaging region. However, when the area of the imaging area is smaller than the area of the button image, the pointing operation by the user cannot be detected in the entire area of the button image. For this reason, in the present embodiment, by displaying a projection image in which the imaging region is displayed, the user is made aware of the imaging region and the operability of the pointing operation by the user is improved. In addition, the user can recognize a button image that can / cannot be pointed. Note that any method may be used for displaying the imaging region as long as the current imaging region can be recognized by the user, such as blinking the imaging region, as well as displaying a frame.

  FIG. 26 is a diagram illustrating an example in which a button image that can be pointed is fed back. In the example shown in FIG. 26, the projected image 1 includes a button image 2a, a button image 2b, a button image 2c, and an imaging region 3. As shown in FIG. 26, when the imaging area 3 is moved, the button image 2c included in the imaging area 3 is changed to a predetermined display such as a color different from that of the other button images. .

  In the embodiment described above, the pointing operation by the user is detected without making the user aware of the current imaging region. However, when there are a plurality of button images in the projected image, it is difficult for the user to recognize which button image is an operable button image. For this reason, in the present embodiment, by projecting a projection image in which the color of the button image included in the current imaging region is changed, the user can recognize the operable button image, and the operability of the pointing operation by the user To improve. It should be noted that the change in the display of the button image is not limited to changing the color, but if the user can recognize the currently operable button image, such as blinking or transmitting the button image, Such a method may be adopted. The button image included in the current imaging region may include a plurality of button images depending on the arrangement of the button image or the imaging region. Indicates a button image.

(4) Configuration In addition, information including processing procedures, control procedures, specific names, various data, parameters, and the like shown in the document and drawings can be arbitrarily changed unless otherwise specified. Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution or integration of each device is not limited to the one shown in the figure, and all or a part thereof is functionally or physically distributed in arbitrary units according to various burdens or usage conditions. Or they can be integrated.

  For example, in the above-described embodiment, the case where the projector 10 includes the cooperative operation control module 140 has been described. However, the cooperative operation control module 140 may be included in a server device provided separately. Specifically, information such as the coordinates of the button image detected by the pointing operation detection module 130 is transmitted to the server device, and it is determined whether or not to request assistance from an adjacent projector on the server device side.

DESCRIPTION OF SYMBOLS 10 Projector 110 Image pick-up module 120 Projection module 130 Pointing operation detection module 131 Camera control part 132 Camera image memory | storage part 133 Image pick-up area position calculation part 134 Button image recognition part 135 Button image position calculation part 136 Camera movement position determination part 137 Pointing operation detection part 140 Cooperative Operation Control Module 141 Judgment Unit 142 Button List Storage Unit 143 Coordinate Conversion Unit 144 Support Request Unit 145 Support Acceptance Unit

JP 2008-003802 A

Claims (17)

In a projection device connected to at least one other projection device via a network,
A projection unit that projects a projection image on a projection surface;
A photographing unit for photographing a photographing region included in the projection plane;
A detection unit for detecting an instruction operation on the projection image by a user based on a photographed image photographed by the photographing unit;
An instruction target information storage unit that stores instruction target information regarding an instruction target included in the projection plane to be an instruction target;
A determination unit that determines presence / absence of an instruction target that cannot be included in the imaging region based on the instruction target information stored by the instruction target information storage unit;
A projection apparatus, comprising: a request unit configured to request imaging to the other projection apparatus when the determination unit determines that there is an instruction target that cannot be included in the imaging region. In response to a response to a shooting request from the other projection device, a shooting request determination unit that determines which shooting of the instruction target is requested;
A coordinate conversion unit that converts a coordinate value representing the position of the instruction target determined to request imaging into a coordinate value of a coordinate system of a projection plane corresponding to the other projection device that requests imaging;
The projection apparatus according to claim 1, further comprising: an imaging request unit that requests the other projection apparatus to perform imaging together with the coordinate value of the instruction target whose coordinate system has been converted. A request accepting unit that accepts a request for photographing the target object from the other projection device;
A response unit that responds to a request for shooting according to whether or not shooting of the instruction target is being performed by the shooting unit when a request for shooting is received;
A photographing request accepting unit that accepts a photographing request together with the coordinate value of the instruction target whose coordinate system has been converted;
The projection apparatus according to claim 2 , wherein the photographing unit photographs the instruction target having the received coordinate value.   4. The projection apparatus according to claim 2, wherein the imaging request determination unit determines which of the instruction targets is requested for imaging according to a position of the other projection apparatus.   The projection apparatus according to claim 3, wherein the answering unit replies to the effect that the request for photographing is suspended when a request for photographing is accepted from a plurality of the other projection apparatuses. The request unit, when it is impossible to shoot in response to a shooting request, further requests the other projection device to shoot,
The projection apparatus according to claim 3, wherein the answering unit answers to the projection requesting projection apparatus in response to an answer received from another projection apparatus.   The projection device according to claim 6, wherein the request unit requests the other projection device to perform photographing according to a photographing requestable range in which a range of the projection device that requests photographing is determined.   The said imaging | photography request determination part determines again which imaging | photography of the said instruction | indication object is requested according to the position of the said projection apparatus from which the reply which can image | photograph was obtained. The projection device according to any one of the above. The request unit, when the photographing unit is out of order, further requests the other projection device to shoot,
The projection apparatus according to claim 3, wherein the answering unit answers to the projection requesting projection apparatus in response to an answer received from another projection apparatus.   The projection apparatus according to claim 4, wherein the imaging request determination unit determines which of the instruction objects is requested for imaging according to the priority of the instruction object.   The said imaging | photography request determination part determines which imaging | photography of the said instruction | indication object is requested according to the said priority which is the instruction | indication priority order of the instruction | indication object instruction | indication. Projection device.   The projection apparatus according to claim 10, wherein the imaging request determination unit determines which of the instruction objects is to be requested according to the priority determined from the area of the instruction object. .   The projection apparatus according to claim 1, wherein the imaging unit performs imaging so that the center of gravity of the imaging region overlaps the center of gravity of the instruction target.   The projection device according to claim 1, wherein the projection unit projects the projection image on which the imaging region is displayed.   The projection apparatus according to claim 1, wherein the projection unit projects the projection image in which the display of the instruction target included in the current imaging region is changed to a predetermined display. A projection method performed by a projection device connected to at least one other projection device via a network, comprising:
The projector is
A projection unit that projects a projection image on a projection surface;
An imaging unit that captures an imaging area included in the projection plane;
A detection step of detecting an instruction operation on the projection image by a user based on a photographed image photographed by the photographing unit;
A determination step of determining the presence or absence of an instruction target that cannot be included in the imaging region, based on instruction target information regarding the instruction target included in the projection plane that is an instruction target;
And a requesting step of requesting imaging to the other projection device when it is determined that there is an instruction target that cannot be included in the imaging region. An information processing system having one or more information processing devices,
A projection unit that projects a projection image on a projection surface;
A photographing unit for photographing a photographing region included in the projection plane;
A detection unit for detecting an instruction operation on the projection image by a user based on a photographed image photographed by the photographing unit;
An instruction target information storage unit that stores instruction target information regarding an instruction target included in the projection plane to be an instruction target;
A determination unit that determines presence / absence of an instruction target that cannot be included in the imaging region based on the instruction target information stored by the instruction target information storage unit;
An information processing system comprising: a requesting unit that makes a request for photographing to another information processing apparatus when the determining unit determines that there is an instruction target that cannot be included in the shooting region.

Images