JP2014153769A - Projection system, control program, control device, and control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the detection accuracy of a user operation to a projection device.SOLUTION: A projection part projects a projection image onto a projection surface. An imaging part images an imaging area, to generate a captured image. An image detection part detects a coincident image indicating an image in the projection image which is coincident with an operation target image. A first coordinate calculation part calculates a first coordinate for specifying the position of the imaging area with respect to the projection image. A second coordinate calculation part calculates a second coordinate for specifying the position of the coincident image with respect to the projection image. A movement information calculation part calculates movement information for moving the imaging area, from the first coordinate and the second coordinate. An imaging control part controls imaging by the imaging part, so as to move the imaging area, according to the movement information. An operation detection part detects the user operation to the coincident image. A projection control part generates the projection image and controls projection by the projection part.

Description

  The present invention relates to a projection system, a control program, a control apparatus, and a control method.

  2. Description of the Related Art Conventionally, a technique for operating a projection apparatus by moving a pointing position using a pointing device such as an electronic pen with respect to a projection image projected on a projection surface such as a screen by a projection apparatus such as a projector is known. ing. On the other hand, there is a technique for operating the projection apparatus without using a pointing device. For example, a technique is known in which a user's predetermined gesture is recognized by a camera and an operation corresponding to the gesture is executed on the projection apparatus. In addition, when an operation for operating an operation target such as a button included in the projection image is performed by the user, this operation is recognized by the camera, and processing corresponding to the operation of the operation target is executed on the projection apparatus. Technology to make it known is also known. These techniques are assumed to be used, for example, for presentations such as conferences and for displaying contents to users in stores.

  In these techniques, since the operation of the projection apparatus is executed in accordance with the pointing position, it is desirable that the current pointing position can be detected with high accuracy. As a technique for detecting the pointing position with high accuracy, for example, an image obtained by marking a predetermined portion of the projection surface is retained, and a pointing is obtained by taking a difference between the marked image and the currently projected image. Some detect the position. With this technique, the current pointing position can be detected without always displaying the marker.

  However, the above-described conventional technique has a problem that it may not be possible to detect a user operation on the projection apparatus. Specifically, in a situation where the projection apparatus is operated by recognizing the user's operation by the camera, the operation on the projection apparatus is detected when the imaging area representing the area that can be captured by the camera is smaller than the projection image. May not be possible. A case where the imaging region is smaller than the projected image occurs when the distance between the projection plane and the camera is short. That is, the above-described problem occurs when a sufficient distance between the projection plane and the camera cannot be secured in applications such as presentation and content display.

  The present invention has been made in view of the above, and an object of the present invention is to provide a projection system, a control program, a control device, and a control method capable of improving the detection accuracy of a user operation on the projection device. .

  In order to solve the above-described problems and achieve the object, a projection system according to the present invention images a projection unit that projects a projection image on a projection plane and an imaging region that represents an imageable region, and generates a captured image. An imaging unit, an image detection unit that detects a matching image that represents an image in the projection image that matches an operation target image that represents an operation target image of a user, and a first position that specifies a position of the imaging region with respect to the projection image From the first coordinate calculation unit that calculates one coordinate, the second coordinate calculation unit that calculates the second coordinate that specifies the position of the coincidence image with respect to the projection image, the first coordinate, and the second coordinate, A movement information calculation unit that calculates movement information for moving the imaging region, an imaging control unit that controls imaging by the imaging unit so as to move the imaging region according to the movement information, and the captured image An operation detection unit that detects a user operation on the matching image, and a projection control unit that generates a projection image according to the detected user operation and controls projection of the generated projection image by the projection unit. Have.

  In addition, the control program according to the present invention is capable of detecting an coincidence image representing an image in a projection image that coincides with an operation target image representing an image to be operated by a user, and capable of capturing the projection image. A first coordinate calculating step for calculating a first coordinate for specifying a position of an imaging region representing the region; a second coordinate calculating step for calculating a second coordinate for specifying the position of the coincidence image with respect to the projection image; A movement information calculation step for calculating movement information for moving the imaging area from the first coordinates and the second coordinates, and imaging by the imaging apparatus so as to move the imaging area according to the movement information. An imaging control step for controlling, and an operation detection step for detecting a user operation on the coincidence image using a captured image captured by the imaging device , It generates a projection image in accordance with said detected user operation to execute a projection control step controls the projection device of the generated projected image to the computer.

  In addition, the control device according to the present invention is capable of capturing an image detection unit that detects a coincidence image that represents an image in a projection image that coincides with an operation target image that represents an image to be operated by the user, A first coordinate calculation unit that calculates a first coordinate that specifies a position of an imaging region that represents the region; a second coordinate calculation unit that calculates a second coordinate that specifies the position of the coincidence image with respect to the projection image; A movement information calculation unit that calculates movement information for moving the imaging region from the first coordinate and the second coordinate, and imaging by the imaging device so as to move the imaging region according to the movement information. An imaging control unit that controls, an operation detection unit that detects a user operation on the coincidence image using a captured image captured by the imaging device, and a projection image according to the detected user operation It generates, and a projection control unit that controls the projection device of the generated projection image.

  Further, the control method according to the present invention includes an image detection step for detecting a coincidence image representing an image in a projection image that coincides with an operation target image representing an image to be operated by a user, and capable of capturing the projection image. A first coordinate calculating step for calculating a first coordinate for specifying a position of an imaging region representing the region; a second coordinate calculating step for calculating a second coordinate for specifying the position of the coincidence image with respect to the projection image; A movement information calculation step for calculating movement information for moving the imaging area from the first coordinates and the second coordinates, and imaging by the imaging apparatus so as to move the imaging area according to the movement information. An imaging control step for controlling, an operation detecting step for detecting a user operation on the matched image using a captured image captured by the imaging device, It has been said to produce a projection image in accordance with a user operation, and a projection control step controls the projection device of the generated projection image.

  According to one aspect of the present invention, it is possible to improve the detection accuracy of a user operation on the projection apparatus.

FIG. 1 is a diagram illustrating a configuration example of a projection system according to the present embodiment. FIG. 2 is a functional block diagram illustrating a device configuration example according to the first embodiment. FIG. 3 is a diagram for explaining the matching image detection process according to the first embodiment. FIG. 4 is a diagram for explaining the movement amount calculation processing of the imaging region according to the first embodiment. FIG. 5 is a flowchart showing an example of the flow of movement amount calculation processing according to the first embodiment. FIG. 6 is a functional block diagram illustrating a device configuration example according to the second embodiment. FIG. 7 is a diagram for explaining movement of the imaging region according to the second embodiment. FIG. 8 is a flowchart showing an example of the flow of movement amount calculation processing according to the second embodiment. FIG. 9 is a functional block diagram showing a device configuration example according to the third embodiment. FIG. 10 is a diagram for explaining the movement of the imaging region according to the third embodiment. FIG. 11 is a flowchart illustrating an example of a flow of a movement amount calculation process according to the third embodiment. FIG. 12 is a functional block diagram illustrating a device configuration example according to the fourth embodiment. FIG. 13 is a diagram illustrating movement of the imaging region according to the fourth embodiment. FIG. 14 is a flowchart illustrating an example of the flow of a movement amount calculation process according to the fourth embodiment. FIG. 15 is a functional block diagram showing a device configuration example according to the fifth embodiment. FIG. 16 is a diagram illustrating movement of the imaging region according to the fifth embodiment. FIG. 17 is a flowchart illustrating an example of the flow of a movement amount calculation process according to the fifth embodiment. FIG. 18 is a functional block diagram showing a device configuration example according to the sixth embodiment. FIG. 19 is a diagram illustrating movement of the imaging region according to the sixth embodiment. FIG. 20 is a flowchart illustrating an example of the flow of a movement amount calculation process according to the sixth embodiment. FIG. 21 is a functional block diagram illustrating an example of a device configuration according to the seventh embodiment. FIG. 22 is a diagram illustrating movement of the imaging region according to the seventh embodiment. FIG. 23 is a flowchart illustrating an example of the flow of a movement amount calculation process according to the seventh embodiment. FIG. 24 is a diagram illustrating movement of the imaging region according to a modification of the seventh embodiment. FIG. 25 is a flowchart illustrating an example of the flow of a movement amount calculation process according to a modification of the seventh embodiment. FIG. 26 is a functional block diagram showing a device configuration example according to the eighth embodiment. FIG. 27 is a flowchart illustrating an example of the flow of movement amount calculation processing according to the eighth embodiment. FIG. 28 is a diagram illustrating an example of feedback of the imaging region. FIG. 29 is a diagram illustrating an example of object feedback.

  Embodiments of a projection system, a control program, a control apparatus, and a control method according to the present invention will be described below with reference to the accompanying drawings. In addition, this invention is not limited by the following embodiment. Moreover, each embodiment can be combined suitably as long as the content is not contradicted.

(Embodiment 1)
[System configuration]
The configuration of the projection system according to the present embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating a configuration example of a projection system according to the present embodiment.

  As shown in FIG. 1, the projection system 10 includes an imaging device 20, a projection device 30, and a control device 100. Among these, the imaging device 20 is a camera or the like that captures a projection image projected on a projection plane and generates a captured image. The imaging device 20 is connected to the control device 100. Hereinafter, an area that can be imaged by the imaging apparatus 20 may be referred to as an “imaging area”. The size of the imaging region changes according to the distance between the imaging device 20 and the projection plane. The projection device 30 is a projector or the like that projects a projection image on a projection surface. The projection device 30 is connected to the control device 100.

  The control device 100 detects a user operation on the projection image from the captured image generated by the imaging device 20, generates a new projection image according to the detected user operation, and projects the projection image by the projection device 30. Control. In addition, the control device 100 calculates movement information for moving the imaging region based on the captured image generated by the imaging device 20 and controls imaging by the imaging device 20. As described above, the size of the imaging region changes according to the distance between the imaging device 20 and the projection plane. For this reason, the control device 100 calculates movement information for moving the imaging region in order to realize a user operation on the projection image. Note that the connection of each device may be either wired or wireless.

  For example, the projection system 10 is used for a presentation such as a meeting or for displaying contents to a user in a store. That is, the user performs a user operation of bringing a hand close to or touching an object such as a button image arranged in the projection image. By performing such a user operation, it is possible to switch the projection image according to the operation as an operation on the object. Further, in order to realize a user operation, an object must be included in the imaging area. Depending on the application of the projection system 10, there is a possibility that a sufficient distance between the projection surface and the imaging device 20 cannot be secured, and a case where the imaging region cannot include the projection image occurs. For this reason, the control apparatus 100 calculates movement information for moving the imaging region based on the captured image.

[Apparatus Configuration According to Embodiment 1]
The apparatus configuration according to the first embodiment will be described with reference to FIG. FIG. 2 is a functional block diagram illustrating a device configuration example according to the first embodiment.

  As illustrated in FIG. 2, the imaging device 20 includes an imaging unit 21. Under the control of the control device 100, the imaging unit 21 captures an imaging region that represents an imageable region, and generates a captured image. Then, the imaging device 20 transmits the generated captured image to the control device 100. Since the captured image is an image obtained by capturing the imaging region, depending on the distance between the projection plane and the imaging device 20, the captured image may not be an image including all the projected images. The projection device 30 includes a projection unit 31. The projection unit 31 projects a projection image on the projection surface under the control of the control device 100.

  The control device 100 includes a storage unit 110 and a control unit 120. Among these, the memory | storage part 110 memorize | stores the data utilized by the various processes by the control part 120, and the various process results by the control part 120, and has the captured image memory | storage part 111. FIG. The captured image storage unit 111 stores a captured image captured by the imaging device 20. The storage unit 110 is, for example, a semiconductor memory device such as a random access memory (RAM), a read only memory (ROM), and a flash memory, or a storage device such as a hard disk or an optical disk.

  The control unit 120 has an internal memory for storing a control program, a program defining various processing procedures, and required data. The control unit 120 includes an image detection unit 121a, a first coordinate calculation unit 121b, a second coordinate calculation unit 121c, an inclusion determination unit 121d, a movement information calculation unit 121e, and an imaging control unit 121f. The processing by these functional blocks is mainly executed to control the imaging device 20. The control unit 120 includes an operation detection unit 122a and a projection control unit 122b. The processing by these functional blocks is mainly executed for controlling the projection device 30. The control unit 120 is, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), or an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). .

  The image detection unit 121a detects an object for a user operation arranged in the projection image. More specifically, the image detection unit 121a acquires an operation target image that represents an image to be operated by the user. The operation target image may be stored in the storage unit 110 or may be acquired from an external storage device, a server device, or the like. In addition, the image detection unit 121a acquires a captured image from the captured image storage unit 111. Then, the image detection unit 121a detects a matching image representing an image in the projection image that matches the operation target image from the acquired operation target image and the captured image. The coincidence image corresponds to an object for a user operation arranged in the projection image. Here, for the matching image detection process by the image detection unit 121a, the imaging control unit 121f moves the imaging region by controlling the direction of the camera lens of the imaging device 20. Thereby, the imaging part 21 produces | generates a some captured image by imaging, moving an imaging region.

  FIG. 3 is a diagram for explaining the matching image detection process according to the first embodiment. The left side of the upper part of FIG. 3 represents a default state. As shown on the left in the upper part of FIG. 3, the imaging region 3 is smaller than the projected image 1 on which the button image 2 is arranged. Further, the projection image 1, the button image 2, and the imaging region 3 are square. In such a state, the control device 100 starts the following processing when the power of the projection device 30 is turned on or when the projected image 1 to be projected is switched.

  The imaging control unit 121 f instructs the imaging device 20 to move the imaging region 3 in the upper left direction of the projection image 1 in order to search for coordinate information of each vertex embedded in the projection image 1. Upon receiving the instruction, the imaging unit 21 detects the coordinate information of the upper left vertex of the projection image 1. Then, the imaging control unit 121f instructs the imaging device 20 so that the upper left vertex of the imaging region 3 overlaps the upper left vertex coordinate of the detected projection image 1. The imaging unit 21 that has received the instruction moves the imaging region 3 so that the upper left vertex of the imaging region 3 overlaps with the upper left vertex coordinate of the projection image 1, as shown on the right side of FIG. The captured area 3 is captured to generate a captured image. The generated captured image is stored in the captured image storage unit 111 via the imaging control unit 121f.

  Thereafter, the imaging control unit 121f displays the vertex coordinates of the projection image 1 and the corresponding vertices of the imaging region 3 as shown on the right side in the middle of FIG. 3, the left side in the middle of FIG. The imaging device 20 is instructed to sequentially move the imaging area so that the two overlap each other. Thus, the captured image storage unit 111 stores a plurality of captured images generated by the imaging unit 21. In the example illustrated in FIG. 3, the button image 2 is included in the captured image captured when the coordinate information of the upper left vertex of the projection image 1 is detected. 3, when the detection of the entire projection image 1 is completed, the imaging control unit 121f controls the imaging unit 21 to return the position of the imaging region 3 to the default state. In addition, the imaging control unit 121f instructs the imaging device 20 to capture an image of the imaging region 3 in the default state. Thereby, the captured image storage unit 111 includes captured images captured in a default state.

  After the detection processing of the entire projection image 1 is completed, the image detection unit 121a acquires an operation target image and acquires a plurality of captured images from the captured image storage unit 111. Then, the image detection unit 121a detects the button image 2 arranged in the projection image by detecting a matching image that matches the operation target image in the plurality of captured images.

  The first coordinate calculation unit 121b calculates first coordinates representing coordinates for specifying the position of the imaging region with respect to the projection image. More specifically, the first coordinate calculation unit 121b calculates all the vertex coordinates of the current imaging region with respect to the projection image when the lower left vertex coordinate of the projection image is (0, 0). The vertex coordinates of the current imaging area are obtained from a plurality of captured images stored in the captured image storage unit 111. The calculated vertex coordinates of the current imaging region are the vertex coordinates of the imaging region in the default state (see the left side in the upper part of FIG. 3).

  The second coordinate calculation unit 121c calculates second coordinates representing coordinates for specifying the position of the coincidence image with respect to the projection image. More specifically, the second coordinate calculation unit 121c calculates all the vertex coordinates of the button image with respect to the projection image when the lower left vertex coordinate of the projection image is (0, 0). The vertex coordinates of the button image are obtained from a plurality of captured images stored in the captured image storage unit 111.

  The inclusion determination unit 121d determines whether or not the imaging region includes a matching image. More specifically, the inclusion determination unit 121d determines from the vertex coordinates of the current imaging area calculated by the first coordinate calculation unit 121b and the vertex coordinates of the button image calculated by the second coordinate calculation unit 121c. Then, it is determined whether or not the current imaging area includes a button image. Then, the inclusion determination unit 121d notifies the movement information calculation unit 121e of the determination result as to whether or not the current imaging region includes a button image.

  Here, an example of the determination process by the inclusion determination unit 121d will be described. What is necessary is just to compare about each vertex coordinate as follows.

Comparison of the lower left vertex coordinates.
If “the x coordinate of the lower left vertex of the button image <the x coordinate of the lower left vertex of the imaging region”, the determination result is that the current imaging region does not include the button image.
If “x coordinate of the lower left vertex of the imaging region <= x coordinate of the lower left vertex of the button image” cannot be determined, the following comparison is performed.
If “y coordinate of the lower left vertex of the button image <y coordinate of the lower left vertex of the imaging region”, the determination result indicates that the current imaging region does not include the button image.
If “y coordinate of the lower left vertex of the imaging region <= y coordinate of the lower left vertex of the button image” cannot be determined, the following comparison is performed.

Top left vertex coordinate comparison.
If “the x coordinate of the upper left vertex of the button image <the x coordinate of the upper left vertex of the imaging region”, the determination result indicates that the current imaging region does not include the button image.
If “x coordinate of the upper left vertex of the imaging region <= x coordinate of the upper left vertex of the button image” cannot be determined, the following comparison is performed.
If “y coordinate of the upper left vertex of the imaging region <y coordinate of the upper left vertex of the button image”, the determination result indicates that the current imaging region does not include the button image.
If “y coordinate of the upper left vertex of the button image <= y coordinate of the upper left vertex of the imaging region”, since it cannot be determined, the following comparison is performed.

Comparison of the lower right vertex coordinates.
If “the x coordinate of the lower right vertex of the imaging region <the x coordinate of the lower right vertex of the button image”, the determination result indicates that the current imaging region does not include the button image.
If “x coordinate of the lower right vertex of the button image <= x coordinate of the lower right vertex of the imaging region”, it cannot be determined, so the following comparison is performed.
If “y coordinate of the lower right vertex of the button image <y coordinate of the lower right vertex of the imaging region”, the determination result is that the current imaging region does not include the button image.
If “y coordinate of the lower right vertex of the imaging region <= y coordinate of the lower right vertex of the button image” cannot be determined, the following comparison is performed.

Top right vertex coordinate comparison.
If “the x coordinate of the upper right vertex of the imaging region <the x coordinate of the upper right vertex of the button image”, the determination result indicates that the current imaging region does not include the button image.
If “x coordinate of upper right vertex of button image <= x coordinate of upper right vertex of imaging region” cannot be determined, the following comparison is performed.
If “y coordinate of the upper right vertex of the imaging region <y coordinate of the upper right vertex of the button image”, the determination result is that the current imaging region does not include the button image.
If “y coordinate of the upper right vertex of the button image <= y coordinate of the upper right vertex of the imaging region”, the determination result indicates that the current imaging region includes the button image.

  The movement information calculation unit 121e calculates movement information for moving the imaging region. More specifically, the movement information calculation unit 121e receives a determination result from the inclusion determination unit 121d. At this time, when it is determined that the current imaging region does not include the button image, the movement information calculation unit 121e calculates the current information from each vertex coordinate of the current imaging region calculated by the first coordinate calculation unit 121b. The coordinates serving as the center of gravity of the imaging region are calculated. In addition, the movement information calculation unit 121e determines the center of gravity of the button image from the vertex coordinates of the button image calculated by the second coordinate calculation unit 121c when it is determined that the current imaging region does not include the button image. Calculate the coordinates.

  Then, the movement information calculation unit 121e calculates, as movement information, the amount of movement from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the button image. The movement information calculation unit 121e can accept a user operation without moving the imaging area when it is determined that the current imaging area includes a button image. The movement information set to “0” is calculated. Thereafter, the movement information calculation unit 121e notifies the imaging control unit 121f of the calculated movement amount.

  Here, an example of movement amount calculation processing by the movement information calculation unit 121e will be described. As described below, the coordinates of the center of gravity are obtained by obtaining the midpoint coordinates of the x direction and the y direction from the respective vertex coordinates, and the value for moving between the obtained midpoint coordinates is set as the movement amount of the imaging region. Just do it.

Coordinates that are the center of gravity of the current imaging area.
"X coordinate =
(X coordinate of the lower left vertex of the imaging area + x coordinate of the upper right vertex of the imaging area) / 2
"Y coordinate =
(Y coordinate of the lower left vertex of the imaging area + y coordinate of the upper right vertex of the imaging area) / 2 ”

The coordinates that are the center of gravity of the button image.
"X coordinate =
(X coordinate of lower left vertex of button image + x coordinate of upper right vertex of button image) / 2
"Y coordinate =
(Y coordinate of the lower left vertex of the button image + y coordinate of the upper right vertex of the button image) / 2 ”

Calculated amount of movement.
"X coordinate =
X-coordinate of the coordinate serving as the center of gravity of the button image-x-coordinate of the coordinate serving as the center of gravity of the current imaging region "
"Y coordinate =
The y-coordinate of the coordinates that will be the center of gravity of the button image-the y-coordinate of the coordinates that will be the center of gravity of the current imaging area "

  The imaging control unit 121f controls imaging by the imaging unit 21. More specifically, the imaging control unit 121f receives the movement amount of the imaging region from the movement information calculation unit 121e, and controls imaging by the imaging unit 21 so as to move the imaging region according to the received movement amount. Note that the imaging control unit 121f does not control the imaging unit 21 when the movement amount notified by the movement information calculation unit 121e is “0”.

  FIG. 4 is a diagram for explaining the movement of the imaging region according to the first embodiment. In the example shown in the upper part of FIG. 4, the vertex coordinates of the button image 2 are (1, 9), (1, 11), (5, 9), and (5, 11). Similarly, the vertex coordinates of the imaging region 3 are (6, 4), (6, 8), (12, 4), and (12, 8). The projection image 1 includes a button image 2 having the above vertex coordinates and an imaging region 3.

  In the state described above, the movement information calculation unit 121e calculates coordinates (9, 6) that are the center of gravity of the current imaging region 3 from the vertex coordinates of the imaging region 3. In FIG. 4, the coordinates serving as the center of gravity of the imaging region 3 are represented by black squares. Further, the movement information calculation unit 121e calculates coordinates (3, 10) that are the center of gravity of the button image 2 from the vertex coordinates of the button image 2. In FIG. 4, the coordinates serving as the center of gravity of the button image 2 are represented by black circles.

  Then, the movement information calculation unit 121e calculates the movement amount from the coordinates (9, 6) serving as the center of gravity of the current imaging region 3 to the coordinates (3, 10) serving as the center of gravity of the button image 2. In the calculation of the movement amount, since “x direction = 3-9 = −6” and “y direction = 10−6 = + 4”, the current imaging region 3 is moved “−6” in the x direction, A movement amount “(−6, +4)” to be moved “+4” in the y direction is obtained. Thereafter, the imaging control unit 121f moves the current imaging region 3 by “−6” in the x direction and “(−6, +4)” for moving “+4” in the y direction to the imaging device 20. To send. As a result, as shown in the lower part of FIG. 4, a movement result is obtained in which the coordinates serving as the center of gravity of the imaging region 3 overlap the coordinates serving as the center of gravity of the button image 2.

  The operation detection unit 122a detects a user operation on an object in the projection image. More specifically, the operation detection unit 122a acquires a captured image from the captured image storage unit 111, and detects a user operation on the button image from the acquired captured image. Detection of a user operation by the operation detection unit 122a is performed after the imaging region moving process by the imaging unit 21.

  The projection control unit 122b controls the projection of the projection image by the projection device 30. More specifically, the projection control unit 122b generates a projection image according to the user operation detected by the operation detection unit 122a, and controls the projection of the generated projection image by the projection unit 31.

[Movement amount calculation processing flow according to Embodiment 1]
Next, the flow of the movement amount calculation process according to the first embodiment will be described with reference to FIG. FIG. 5 is a flowchart showing an example of the flow of movement amount calculation processing according to the first embodiment.

  As shown in FIG. 5, the first coordinate calculation unit 121b calculates the vertex coordinates of the current imaging region (step S101). Further, the second coordinate calculation unit 121c calculates the vertex coordinates of the button image (step S102). Then, the inclusion determination unit 121d determines whether or not the button image is included in the current imaging region (step S103).

  At this time, when it is determined by the inclusion determination unit 121d that it is not included (step S103: No), the movement information calculation unit 121e determines the current imaging region from the vertex coordinates calculated by the first coordinate calculation unit 121b. The coordinates serving as the center of gravity are calculated (step S104). In addition, the movement information calculation unit 121e calculates a coordinate serving as the center of gravity of the button image from the vertex coordinates calculated by the second coordinate calculation unit 121c (step S105).

  Then, the movement information calculation unit 121e calculates the amount of movement from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the button image (step S106). If it is determined by the inclusion determination unit 121d that the movement is calculated (step S103: Yes), the movement information calculation unit 121e does not have to move the current imaging region, and thus the movement amount is “0”. (Step S107).

[Effects of Embodiment 1]
In the present embodiment, the imaging apparatus calculates a movement amount from the coordinates serving as the center of gravity of the imaging region to the coordinates serving as the center of gravity of the button image, and moves the imaging region so as to overlap the button image based on the calculated movement amount. To. As a result, according to the present embodiment, it is possible to improve the detection accuracy of the user operation on the projection device even when the distance between the projection surface and the imaging device cannot be sufficiently secured.

(Embodiment 2)
In the first embodiment, the case has been described in which the movement amount of the imaging region for superimposing the centroid of the imaging region on the centroid of the object for user operation is calculated. In the second embodiment, a case where a smaller movement amount of the imaging area is calculated will be described.

[Apparatus configuration according to Embodiment 2]
The apparatus configuration according to the second embodiment will be described with reference to FIG. FIG. 6 is a functional block diagram illustrating a device configuration example according to the second embodiment. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description of the same components may be omitted. Specifically, functions, configurations, and processes other than the movement information calculation unit 221e described below are the same as those in the first embodiment.

  As illustrated in FIG. 6, the imaging device 20 includes an imaging unit 21. In addition, the projection device 30 includes a projection unit 31. The control device 200 includes a storage unit 110 and a control unit 220. Among these, the storage unit 110 includes a captured image storage unit 111. The control unit 220 includes an image detection unit 121a, a first coordinate calculation unit 121b, a second coordinate calculation unit 121c, an inclusion determination unit 121d, a movement information calculation unit 221e, and an imaging control unit 121f. The control unit 220 includes an operation detection unit 122a and a projection control unit 122b.

  The movement information calculation unit 221e calculates movement information that minimizes the movement amount of the imaging region. More specifically, the movement information calculation unit 221e receives the determination result from the inclusion determination unit 121d. At this time, when it is determined that the current imaging region does not include the button image, the movement information calculation unit 221e, between the corresponding coordinates of each vertex coordinate of the imaging region and each vertex coordinate of the button image, The movement amount that minimizes the distance is calculated.

  Here, an example of movement amount calculation processing by the movement information calculation unit 221e will be described. As described below, the movement amount between the corresponding vertex coordinates is obtained, and the value that minimizes the obtained movement amount may be set as the movement amount of the imaging region.

The amount of movement between the coordinates of the lower left vertex.
“Movement amount in x direction = x coordinate of lower left vertex of button image−x coordinate of lower left vertex of imaging region”
“Y-direction movement amount = y coordinate of the lower left vertex of the button image−y coordinate of the lower left vertex of the imaging area”

The amount of movement between the coordinates of the top left vertex.
“Movement amount in x direction = x coordinate of upper left vertex of button image−x coordinate of upper left vertex of imaging region”
“Y-direction movement amount = y coordinate of the upper left vertex of the button image−y coordinate of the upper left vertex of the imaging region”

The amount of movement between the coordinates of the lower right vertex.
“Movement amount in x direction = x coordinate of lower right vertex of button image−x coordinate of lower right vertex of imaging region”
“Y-direction movement amount = y coordinate of the lower right vertex of the button image−y coordinate of the lower right vertex of the imaging region”

The amount of movement between the coordinates of the upper right vertex.
“Movement amount in x direction = x coordinate of upper right vertex of button image−x coordinate of upper right vertex of imaging region”
“Y-direction movement amount = y coordinate of the upper right vertex of the button image−y coordinate of the upper right vertex of the imaging area”

  FIG. 7 is a diagram for explaining movement of the imaging region according to the second embodiment. In the example shown in the upper part of FIG. 7, the vertex coordinates of the button image 2 are (1, 9), (1, 11), (5, 9), and (5, 11). Similarly, the vertex coordinates of the imaging region 3 are (6, 4), (6, 8), (12, 4), and (12, 8). The projection image 1 includes a button image 2 having the above vertex coordinates and an imaging region 3.

  In the state described above, the movement information calculation unit 221e calculates the movement amounts “x direction = 1-6 = −5” and “y direction = 9-4 = + 5” between the coordinates of the lower left vertex. As a result, the movement amount “(−5, +5)” for moving the current imaging region 3 by “−5” in the x direction and “+5” in the y direction is obtained. The distance of the movement amount “(−5, +5)” is “10”.

  Further, the movement information calculation unit 221e calculates the movement amounts “x direction = 1-6 = −5” and “y direction = 11−8 = + 3” between the coordinates of the upper left vertex. As a result, a movement amount “(−5, +3)” for moving the current imaging region 3 by “−5” in the x direction and “+3” in the y direction is obtained. The distance of the movement amount “(−5, +3)” is “8”.

  Further, the movement information calculation unit 221e calculates the movement amounts “x direction = 5-12 = −7” and “y direction = 9-4 = + 5” between the coordinates of the lower right vertex. As a result, a movement amount “(−7, +5)” for moving the current imaging region 3 “−7” in the x direction and “+5” in the y direction is obtained. The distance of the movement amount “(−7, +5)” is “12”.

  Further, the movement information calculation unit 221e calculates the movement amounts “x direction = 5-12 = −7” and “y direction = 11−8 = + 3” between the coordinates of the upper right vertex. As a result, a movement amount “(−7, +3)” for moving the current imaging region 3 by “−7” in the x direction and “+3” in the y direction is obtained. The distance of the movement amount “(−7, +3)” is “10”.

  Then, the movement information calculation unit 221e calculates the movement amount “(−5, +3)” that minimizes the distance between the coordinates as the movement information. The calculated movement amount “(−5, +3)” is transmitted to the imaging device 20 via the imaging control unit 121f, and the imaging region 3 is moved by the imaging device 20. As a result, as shown in the lower part of FIG. 7, a movement result is obtained in which the upper left vertex coordinates of the imaging region 3 overlap the upper left vertex coordinates of the button image 2.

[Moving amount calculation processing flow according to Embodiment 2]
Next, the flow of the movement amount calculation process according to the second embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing an example of the flow of movement amount calculation processing according to the second embodiment. FIG. 8 shows the processing after “Step S103: No” shown in FIG. That is, FIG. 8 shows the flow of the movement amount calculation process when the button image is not included in the imaging region.

  As illustrated in FIG. 8, the movement information calculation unit 221e includes distances between corresponding coordinates between the vertex coordinates calculated by the first coordinate calculation unit 121b and the vertex coordinates calculated by the second coordinate calculation unit 121c. Is calculated (step S201). Then, the movement information calculation unit 221e determines whether or not the distance between the coordinates of the lower left vertex is the minimum in the imaging region and the button image (step S202).

  At this time, if the movement information calculation unit 221e determines that the distance between the coordinates of the lower left vertex is the minimum (step S202: Yes), the movement amount between the coordinates of the lower left vertex is used as the movement information (step S202). S203). On the other hand, when the movement information calculation unit 221e determines that the distance between the coordinates of the lower left vertex is not the minimum (step S202: No), the distance between the coordinates of the upper left vertex is the minimum in the imaging region and the button image. It is determined whether or not (step S204).

  At this time, if the movement information calculation unit 221e determines that the distance between the coordinates of the upper left vertex is the minimum (step S204: Yes), the movement amount between the coordinates of the upper left vertex is used as the movement information (step S204). S205). On the other hand, when the movement information calculation unit 221e determines that the distance between the coordinates of the upper left vertex is not the minimum (step S204: No), the distance between the coordinates of the lower right vertex is the imaging region and the button image. It is determined whether or not it is minimum (step S206).

  At this time, when the movement information calculation unit 221e determines that the distance between the coordinates of the lower right vertex is the minimum (step S206: Yes), the movement amount between the coordinates of the lower right vertex is used as the movement information. (Step S207). On the other hand, if the movement information calculation unit 221e determines that the distance between the coordinates of the lower right vertex is not the minimum (step S206: No), the movement amount between the coordinates of the upper right vertex is used as the movement information (step S208). ).

[Effects of Embodiment 2]
In the present embodiment, the distance between the coordinates of corresponding vertices between the vertex coordinates of the imaging region and the vertex coordinates of the button image is obtained, and the movement amount between the coordinates at which the obtained distance is the minimum is determined as the movement amount of the imaging region. Calculate as As a result, according to the present embodiment, it is possible to improve the detection accuracy of the user operation on the projection apparatus while suppressing the processing load and processing time required for moving the imaging region.

(Embodiment 3)
In the first embodiment, the case has been described in which the movement amount of the imaging region for superimposing the centroid of the imaging region on the centroid of the object for user operation is calculated. In the third embodiment, a case will be described in which the movement amount of the imaging region is calculated according to the area of the object and the area of the imaging region.

[Apparatus configuration according to Embodiment 3]
The apparatus configuration according to the third embodiment will be described with reference to FIG. FIG. 9 is a functional block diagram showing a device configuration example according to the third embodiment. In the third embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description of the same components may be omitted. Specifically, functions, configurations, and processes other than the inclusion determination unit 321d, the movement information calculation unit 321e, and the area determination unit 321g described below are the same as those in the first embodiment.

  As illustrated in FIG. 9, the imaging device 20 includes an imaging unit 21. In addition, the projection device 30 includes a projection unit 31. The control device 300 includes a storage unit 110 and a control unit 320. Among these, the storage unit 110 includes a captured image storage unit 111. The control unit 320 includes an image detection unit 121a, a first coordinate calculation unit 121b, a second coordinate calculation unit 121c, an inclusion determination unit 321d, a movement information calculation unit 321e, an imaging control unit 121f, and an area determination unit 321g. And have. The control unit 320 includes an operation detection unit 122a and a projection control unit 122b.

  The inclusion determination unit 321d determines whether or not the imaging region includes a matching image. More specifically, the inclusion determination unit 321d determines from the vertex coordinates of the current imaging area calculated by the first coordinate calculation unit 121b and the vertex coordinates of the button image calculated by the second coordinate calculation unit 121c. Then, it is determined whether or not the current imaging area includes a button image. Then, the inclusion determination unit 321d notifies the area determination unit 321g of the determination result of whether or not the current imaging region includes a button image.

  The area determination unit 321g determines whether or not the area of the imaging region is smaller than the area of the coincidence image. More specifically, the area determination unit 321g receives the determination result from the inclusion determination unit 321d. At this time, when it is determined that the current imaging region does not include the button image, the area determination unit 321g calculates the current coordinate from the vertex coordinates of the current imaging region calculated by the first coordinate calculation unit 121b. The area of the imaging region is calculated. The area determination unit 321g determines the area of the button image from the vertex coordinates of the button image calculated by the second coordinate calculation unit 121b when it is determined that the current imaging region does not include the button image. calculate. Then, the area determination unit 321g compares the size of the current imaging area with the area of the button image, and notifies the movement information calculation unit 321e of the comparison result.

  The movement information calculation unit 321e calculates movement information for moving the imaging region. More specifically, the movement information calculation unit 321e receives a determination result from the area determination unit 321g. At this time, when it is determined that the area of the current imaging region is smaller than the area of the button image, the movement information calculation unit 321e uses the vertex coordinates of the current imaging region calculated by the first coordinate calculation unit 121b. Then, the coordinates serving as the center of gravity of the current imaging region are calculated. Further, the movement information calculation unit 321e determines the button image from the vertex coordinates of the button image calculated by the second coordinate calculation unit 121c when it is determined that the area of the current imaging region is smaller than the area of the button image. Calculate the coordinates of the center of gravity.

  Then, the movement information calculation unit 321e calculates, as movement information, the amount of movement from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the button image. Note that the movement information calculation unit 321e can accept a user operation without moving the imaging area if it is determined that the current imaging area includes a button image. The movement information set to “0” is calculated.

  In addition, when it is determined that the area of the current imaging region is larger than the area of the button image, the movement information calculation unit 321e determines whether the coordinates of the vertex coordinates of the imaging region and the vertex coordinates of the button image are corresponding. The amount of movement that minimizes the distance is calculated. That is, the movement information calculation unit 321e calculates movement information that minimizes the movement amount of the imaging region, as described in the second embodiment. Thereafter, the movement information calculation unit 321e notifies the imaging control unit 121f of the calculated movement amount. Even when it is determined that the area of the current imaging region is larger than the area of the button image, the amount of movement for superimposing the coordinates that become the center of gravity of the button image on the coordinates that become the center of gravity of the button image May be calculated.

  FIG. 10 is a diagram for explaining the movement of the imaging region according to the third embodiment. In FIG. 10, a case where the area of the imaging region is smaller than the area of the button image will be described as an example. In the example shown in the upper part of FIG. 10, the vertex coordinates of the button image 2 are (1, 9), (1, 11), (5, 9), and (5, 11). Similarly, the vertex coordinates of the imaging region 3 are (8, 5), (8, 7), (10, 5), and (10, 7). The projection image 1 includes a button image 2 having the above vertex coordinates and an imaging region 3.

  In the state described above, the movement information calculation unit 321e calculates the coordinates (9, 6) that are the center of gravity of the current imaging region 3 from the vertex coordinates of the imaging region 3. In FIG. 10, the coordinates serving as the center of gravity of the imaging region 3 are represented by black squares. In addition, the movement information calculation unit 321e calculates coordinates (3, 10) serving as the center of gravity of the button image 2 from each vertex coordinate of the button image 2. In FIG. 10, the coordinates serving as the center of gravity of the button image 2 are represented by black circles.

  Then, the movement information calculation unit 321e calculates the movement amount from the coordinates (9, 6) serving as the center of gravity of the current imaging region 3 to the coordinates (3, 10) serving as the center of gravity of the button image 2. In the calculation of the movement amount, since “x direction = 3-9 = −6” and “y direction = 10−6 = + 4”, the current imaging region 3 is moved “−6” in the x direction, A movement amount “(−6, +4)” to be moved “+4” in the y direction is obtained. Thereafter, the imaging control unit 121f moves the current imaging region 3 by “−6” in the x direction and “(−6, +4)” for moving “+4” in the y direction to the imaging device 20. To send. As a result, as shown in the lower part of FIG. 10, a movement result is obtained in which the coordinates serving as the center of gravity of the image area 3 overlap the coordinates serving as the center of gravity of the button image 2.

[Moving amount calculation processing flow according to Embodiment 3]
Next, the flow of the movement amount calculation process according to the third embodiment will be described with reference to FIG. FIG. 11 is a flowchart illustrating an example of a flow of a movement amount calculation process according to the third embodiment. FIG. 11 shows the processing after “Step S103: No” shown in FIG. That is, FIG. 11 shows the flow of the movement amount calculation process when the button image is not included in the imaging region.

  As illustrated in FIG. 11, the area determination unit 321g calculates the area of the current imaging region from each vertex coordinate of the current imaging region calculated by the first coordinate calculation unit 121b, and also uses the second coordinate calculation unit 121c. The area of the button image is calculated from the vertex coordinates of the button image calculated by (Step S301). Then, the area determination unit 321g determines whether or not the current area of the imaging region is larger than the area of the button image (step S302).

  At this time, when the area determination unit 321g determines that the area of the imaging region is smaller than the area of the button image (step S302: No), the movement information calculation unit 321e is calculated by the first coordinate calculation unit 121b. From the vertex coordinates, the coordinates serving as the center of gravity of the current imaging region are calculated (step S303). In addition, the movement information calculation unit 321e calculates the coordinates serving as the center of gravity of the button image from the vertex coordinates calculated by the second coordinate calculation unit 121c (step S304).

  Then, the movement information calculation unit 321e calculates the movement amount from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the button image (step S305). When the area determination unit 321g determines that the area of the imaging region is larger than the area of the button image (step S302: Yes), the movement information calculation unit 321e The movement amount that minimizes the distance between the coordinates of the corresponding vertices with the button image is calculated as movement information (step S306).

[Effects of Embodiment 3]
In the present embodiment, when the area of the imaging region is smaller than the area of the button image, the movement amount from the coordinates serving as the center of gravity of the imaging region to the coordinates serving as the center of gravity of the button image is calculated as the movement amount of the imaging region. To do. As a result, according to the present embodiment, even when the area of the imaging region is smaller than the area of the button image, the operability of the user operation is improved by overlaying the center of gravity of the imaging region on the center of gravity of the button image. Can be made.

(Embodiment 4)
In the first embodiment, the calculation of the movement amount of the imaging region when there is a single object for user operation in the projection image has been described. In the fourth embodiment, calculation of the moving amount of the imaging region when there are a plurality of objects for user operation in the projection image will be described.

[Apparatus Configuration According to Embodiment 4]
The apparatus configuration according to the fourth embodiment will be described with reference to FIG. FIG. 12 is a functional block diagram illustrating a device configuration example according to the fourth embodiment. In the fourth embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description of the same components may be omitted. Specifically, functions, configurations, and processes other than the movement information calculation unit 421e described below are the same as those in the first embodiment.

  In the processing by the image detection unit 121a according to the present embodiment, a plurality of button images arranged in the projection image are detected by detecting a plurality of matching images that match the operation target image in the plurality of captured images. And Further, in the process by the inclusion determination unit 121d according to the present embodiment, it is determined whether or not the current imaging region includes all the button images. Further, in the processing by the second coordinate calculation unit 121c according to the present embodiment, all vertex coordinates of all button images are calculated.

  As illustrated in FIG. 12, the imaging device 20 includes an imaging unit 21. In addition, the projection device 30 includes a projection unit 31. The control device 400 includes a storage unit 110 and a control unit 420. Among these, the storage unit 110 includes a captured image storage unit 111. The control unit 420 includes an image detection unit 121a, a first coordinate calculation unit 121b, a second coordinate calculation unit 121c, an inclusion determination unit 121d, a movement information calculation unit 421e, and an imaging control unit 121f. The control unit 420 includes an operation detection unit 122a and a projection control unit 122b.

  The movement information calculation unit 421e calculates movement information for moving the imaging region. More specifically, the movement information calculation unit 421e accepts that a plurality of button images have been detected by the image detection unit 121a, and accepts a determination result from the inclusion determination unit 121d. At this time, when it is determined that the current imaging region does not include all the button images, the movement information calculation unit 421e uses the vertex coordinates of the current imaging region calculated by the first coordinate calculation unit 121b, A coordinate serving as the center of gravity of the current imaging region is calculated.

  In addition, the movement information calculation unit 421e selects an arbitrary button image from a plurality of button images when it is determined that the current imaging region does not include all the button images. Then, the movement information calculation unit 421e calculates coordinates serving as the center of gravity of the selected arbitrary button image from the vertex coordinates of all the button images calculated by the second coordinate calculation unit 121c.

  Subsequently, the movement information calculation unit 421e calculates, as movement information, the amount of movement from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the selected arbitrary button image. Note that the movement information calculation unit 421e can accept a user operation without moving the imaging area when it is determined that the current imaging area includes all the button images. Is calculated as “0”. Thereafter, the movement information calculation unit 421e notifies the imaging control unit 121f of the calculated movement amount.

  FIG. 13 is a diagram illustrating movement of the imaging region according to the fourth embodiment. In the example shown in the upper part of FIG. 13, the projection image 1 includes a button image 2 a, a button image 2 b, a button image 2 c, and an imaging region 3. Further, the imaging area 3 does not include any button image.

  In the state described above, the movement information calculation unit 421e calculates a coordinate that is the center of gravity of the current imaging region 3 from each vertex coordinate of the current imaging region 3 calculated by the first coordinate calculation unit 121b. Then, the movement information calculation unit 421e selects the button image 2c from among the plurality of button images. Then, the movement information calculation unit 421e calculates a coordinate serving as the center of gravity of the button image 2c from the vertex coordinates of the button image 2c calculated by the second coordinate calculation unit 121c. Subsequently, the movement information calculation unit 421e calculates, as movement information, the amount of movement from the coordinates serving as the center of gravity of the current imaging region 3 to the coordinates serving as the center of gravity of the button image 2c. As a result, as shown in the lower part of FIG. 13, a movement result is obtained in which the coordinates serving as the center of gravity of the imaging region 3 overlap the coordinates serving as the center of gravity of the button image 2 c.

[Moving amount calculation processing flow according to Embodiment 4]
Next, the flow of the movement amount calculation process according to the fourth embodiment will be described with reference to FIG. FIG. 14 is a flowchart illustrating an example of the flow of a movement amount calculation process according to the fourth embodiment.

  As shown in FIG. 14, the first coordinate calculation unit 121b calculates the vertex coordinates of the current imaging region (step S401). Further, the second coordinate calculation unit 121c calculates the vertex coordinates of all the button images (step S402). Then, the inclusion determination unit 121d determines whether or not all button images are included in the current imaging region (step S403).

  At this time, when it is determined by the inclusion determination unit 121d that all are not included (step S403: No), the movement information calculation unit 421e determines the current coordinates from the vertex coordinates calculated by the first coordinate calculation unit 121b. The coordinates serving as the center of gravity of the imaging region are calculated (step S404). In addition, the movement information calculation unit 421e selects an arbitrary button image from the plurality of button images, and calculates coordinates serving as the center of gravity of the selected button image from the vertex coordinates calculated by the second coordinate calculation unit 121c (step) S405).

  Then, the movement information calculation unit 421e calculates a movement amount from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the selected button image (step S406). Note that if the inclusion determination unit 121d determines that all are included (step S403: Yes), the movement information calculation unit 421e does not have to move the current imaging region, so the movement amount is set to “ 0 ”(step S407).

[Effects of Embodiment 4]
In the present embodiment, the movement amount from the coordinates serving as the center of gravity of the imaging region to the coordinates serving as the center of gravity of any selected button image among a plurality of button images is calculated, and the imaging region is selected based on the calculated movement amount. The image pickup apparatus is controlled to move so as to overlap the button image. As a result, according to the present embodiment, it is possible to improve the detection accuracy of user operations on the projection apparatus.

(Embodiment 5)
In the fourth embodiment, when there are a plurality of objects for user operation in the projected image, an arbitrary object is selected, and the coordinates that are the center of gravity of the selected arbitrary object are set to the coordinates that are the center of gravity of the imaging area. The case where the movement amount for overlapping is calculated has been described. In the fifth embodiment, a case will be described in which a smaller movement amount of the imaging region is calculated when there are a plurality of objects for user operation in the projection image.

[Apparatus Configuration According to Embodiment 5]
The apparatus configuration according to the fifth embodiment will be described with reference to FIG. FIG. 15 is a functional block diagram showing a device configuration example according to the fifth embodiment. In the fifth embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description of the same components may be omitted. Specifically, functions, configurations, and processes other than the movement information calculation unit 521e described below are the same as those in the first embodiment.

  In the processing by the image detection unit 121a according to the present embodiment, a plurality of button images arranged in the projection image are detected by detecting a plurality of matching images that match the operation target image in the plurality of captured images. And Further, in the process by the inclusion determination unit 121d according to the present embodiment, it is determined whether or not the current imaging region includes all the button images. Further, in the processing by the second coordinate calculation unit 121c according to the present embodiment, all vertex coordinates of all button images are calculated.

  As illustrated in FIG. 15, the imaging device 20 includes an imaging unit 21. In addition, the projection device 30 includes a projection unit 31. The control device 500 includes a storage unit 110 and a control unit 520. Among these, the storage unit 110 includes a captured image storage unit 111. The control unit 520 includes an image detection unit 121a, a first coordinate calculation unit 121b, a second coordinate calculation unit 121c, an inclusion determination unit 121d, a movement information calculation unit 521e, and an imaging control unit 121f. In addition, the control unit 520 includes an operation detection unit 122a and a projection control unit 122b.

  The movement information calculation unit 521e calculates movement information for moving the imaging area. More specifically, the movement information calculation unit 521e accepts that a plurality of button images are detected by the image detection unit 121a and accepts a determination result from the inclusion determination unit 121d. At this time, when it is determined that the current imaging region does not include all the button images, the movement information calculation unit 521e uses the vertex coordinates of the current imaging region calculated by the first coordinate calculation unit 121b. A coordinate serving as the center of gravity of the current imaging region is calculated. Further, the movement information calculation unit 521e determines the button image from the vertex coordinates of all the button images calculated by the second coordinate calculation unit 121c when it is determined that the current imaging region does not include all the button images. Coordinates that are the center of gravity are calculated for all.

  Then, the movement information calculation unit 521e calculates the distance between the coordinates serving as the center of gravity of the current imaging region and the coordinates serving as the center of gravity of each button image. Subsequently, the movement information calculation unit 521e calculates a movement amount corresponding to the minimum distance among the calculated distances as movement information. Note that the movement information calculation unit 521e can accept a user operation without moving the imaging area when it is determined that the current imaging area includes all the button images. Is calculated as “0”. Thereafter, the movement information calculation unit 521e notifies the imaging control unit 121f of the calculated movement amount.

  FIG. 16 is a diagram illustrating movement of the imaging region according to the fifth embodiment. In the example shown in the upper part of FIG. 16, the projected image 1 includes a button image 2a, a button image 2b, a button image 2c, and an imaging region 3. Further, the imaging area 3 does not include any button image.

  In the state described above, the movement information calculation unit 521e calculates a coordinate that is the center of gravity of the current imaging region 3 from each vertex coordinate of the current imaging region 3 calculated by the first coordinate calculation unit 121b. In addition, the movement information calculation unit 521e calculates each of the button image 2a, the button image 2b, and the button image 2c from the respective vertex coordinates of the button image 2a, the button image 2b, and the button image 2c calculated by the second coordinate calculation unit 121c. Calculate the coordinates of the center of gravity.

  Then, the movement information calculation unit 521e calculates the distance between the coordinates that are the center of gravity of the current imaging region 3 and the coordinates that are the centers of gravity of the button image 2a, the button image 2b, and the button image 2c. Subsequently, the movement information calculation unit 521e compares the calculated distances and calculates a movement amount corresponding to the minimum distance as movement information. As a result, as shown in the lower part of FIG. 16, a movement result is obtained in which the coordinates serving as the center of gravity of the imaging region 3 overlap the coordinates serving as the center of gravity of the button image 2b.

[Moving Amount Calculation Processing Flow According to Embodiment 5]
Next, the flow of the movement amount calculation process according to the fifth embodiment will be described with reference to FIG. FIG. 17 is a flowchart illustrating an example of the flow of a movement amount calculation process according to the fifth embodiment. Note that FIG. 17 illustrates processing subsequent to “Step S403: No” illustrated in FIG. That is, FIG. 17 shows the flow of the movement amount calculation process when all the button images are not included in the imaging region.

  As illustrated in FIG. 17, the movement information calculation unit 521e calculates coordinates serving as the center of gravity of the current imaging region from the vertex coordinates calculated by the first coordinate calculation unit 121b (step S501). Further, the movement information calculation unit 521e calculates the coordinates that are the center of gravity of each button image from the vertex coordinates calculated by the second coordinate calculation unit 121c (step S502).

  Then, the movement information calculation unit 521e calculates the distance between the coordinates serving as the center of gravity of the current imaging region and the coordinates serving as the center of gravity of each button image (step S503). Subsequently, the movement information calculation unit 521e compares the calculated distances, and calculates a movement amount corresponding to the minimum distance as movement information (step S504).

[Effects of Embodiment 5]
In the present embodiment, the distance between the coordinates serving as the center of gravity of the imaging region and the coordinates serving as the center of gravity of each of the plurality of button images is obtained, and the movement amount corresponding to the minimum distance is calculated as movement information. As a result, according to the present embodiment, it is possible to improve the detection accuracy of the user operation on the projection apparatus while suppressing the processing load and processing time required for moving the imaging region.

(Embodiment 6)
In the fourth embodiment, when there are a plurality of objects for user operation in the projected image, an arbitrary object is selected, and the coordinates that are the center of gravity of the selected arbitrary object are set to the coordinates that are the center of gravity of the imaging area. The case where the movement amount for overlapping is calculated has been described. In the sixth embodiment, a case where the movement amount of the imaging region is calculated according to the area of the object when there are a plurality of objects for user operation in the projection image will be described.

[Apparatus configuration according to Embodiment 6]
The apparatus configuration according to the sixth embodiment will be described with reference to FIG. FIG. 18 is a functional block diagram showing a device configuration example according to the sixth embodiment. In the sixth embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description of the same components may be omitted. Specifically, functions, configurations, and processes other than the movement information calculation unit 621e described below are the same as those in the first embodiment.

  In the processing by the image detection unit 121a according to the present embodiment, a plurality of button images arranged in the projection image are detected by detecting a plurality of matching images that match the operation target image in the plurality of captured images. And Further, in the process by the inclusion determination unit 121d according to the present embodiment, it is determined whether or not the current imaging region includes all the button images. Further, in the processing by the second coordinate calculation unit 121c according to the present embodiment, all vertex coordinates of all button images are calculated.

  As illustrated in FIG. 18, the imaging device 20 includes an imaging unit 21. In addition, the projection device 30 includes a projection unit 31. The control device 600 includes a storage unit 110 and a control unit 620. Among these, the storage unit 110 includes a captured image storage unit 111. The control unit 620 includes an image detection unit 121a, a first coordinate calculation unit 121b, a second coordinate calculation unit 121c, an inclusion determination unit 121d, a movement information calculation unit 621e, and an imaging control unit 121f. The control unit 620 includes an operation detection unit 122a and a projection control unit 122b.

  The movement information calculation unit 621e calculates movement information for moving the imaging region. More specifically, the movement information calculation unit 621e accepts that a plurality of button images are detected by the image detection unit 121a and accepts a determination result from the inclusion determination unit 121d. At this time, when it is determined that the current imaging area does not include all the button images, the movement information calculation unit 621e uses the respective vertex coordinates of the current imaging area calculated by the first coordinate calculation unit 121b. A coordinate serving as the center of gravity of the current imaging region is calculated. In addition, the movement information calculation unit 621e determines the button image from the vertex coordinates of all the button images calculated by the second coordinate calculation unit 121c when it is determined that the current imaging region does not include all the button images. Calculate all areas.

  Then, the movement information calculation unit 621e compares the areas of all the calculated button images, and selects the button image having the maximum area. Subsequently, the movement information calculation unit 621e calculates coordinates that are the center of gravity of the selected button image. Thereafter, the movement information calculation unit 621e calculates, as movement information, the amount of movement from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the selected button image. Note that the movement information calculation unit 621e can accept a user operation without moving the imaging area when it is determined that the current imaging area includes all the button images. Is calculated as “0”. Then, the movement information calculation unit 621e notifies the imaging control unit 121f of the calculated movement amount.

  FIG. 19 is a diagram illustrating movement of the imaging region according to the sixth embodiment. In the example shown in the upper part of FIG. 19, the projection image 1 includes a button image 2a, a button image 2b, a button image 2c, and an imaging region 3. Further, the imaging area 3 does not include any button image.

  In the state described above, the movement information calculation unit 621e calculates a coordinate that is the center of gravity of the current imaging region 3 from each vertex coordinate of the current imaging region 3 calculated by the first coordinate calculation unit 121b. In addition, the movement information calculation unit 621e calculates each of the button image 2a, the button image 2b, and the button image 2c from each vertex coordinate of the button image 2a, the button image 2b, and the button image 2c calculated by the second coordinate calculation unit 121c. Calculate the area.

  Then, the movement information calculation unit 621e selects the button image 2c having the maximum calculated area. Subsequently, the movement information calculation unit 621e calculates coordinates that are the center of gravity of the selected button image 2c. Thereafter, the movement information calculation unit 621e calculates the movement amount from the coordinates that are the center of gravity of the current imaging region 3 to the coordinates that are the center of gravity of the button image 2c as movement information. As a result, as shown in the lower part of FIG. 19, a moving result is obtained in which the coordinates serving as the center of gravity of the imaging region 3 overlap the coordinates serving as the center of gravity of the button image 2 c.

  Of the plurality of button images, the button image having the largest area may not be moved, and which button image is moved can be arbitrarily changed.

[Moving amount calculation processing flow according to Embodiment 6]
Next, the flow of the movement amount calculation process according to the sixth embodiment will be described with reference to FIG. FIG. 20 is a flowchart illustrating an example of the flow of a movement amount calculation process according to the sixth embodiment. Note that FIG. 20 illustrates processing subsequent to “Step S403: No” illustrated in FIG. That is, FIG. 20 shows the flow of the movement amount calculation process when all the button images are not included in the imaging region.

  As illustrated in FIG. 20, the movement information calculation unit 621e calculates the area of all button images from the vertex coordinates calculated by the second coordinate calculation unit 121c, and selects the button image having the maximum calculated area. (Step S601). Then, the movement information calculation unit 621e calculates a coordinate serving as the center of gravity of the current imaging region from each vertex coordinate calculated by the first coordinate calculation unit 121b (step S602). Subsequently, the movement information calculation unit 621e calculates coordinates serving as the center of gravity of the selected button image (step S603). Thereafter, the movement information calculation unit 621e calculates the movement amount from the coordinates that are the center of gravity of the current imaging region to the coordinates that are the center of gravity of the selected button image (step S604).

[Effects of Embodiment 6]
In the present embodiment, the amount of movement from the coordinates serving as the center of gravity of the imaging region to the coordinates serving as the center of gravity of the button image selected according to the size of the area among the plurality of button images is calculated. As a result, according to the present embodiment, it is possible to improve the operability and detection accuracy of user operations on the projection apparatus. In other words, since the button image having the maximum area can be frequently operated in the projection image, the operability and detection accuracy of the user operation can be improved by overlapping the imaging region on the button image having the maximum area. Can be improved.

(Embodiment 7)
In the fourth embodiment, when there are a plurality of objects for user operation in the projected image, an arbitrary object is selected, and the coordinates that are the center of gravity of the selected arbitrary object are set to the coordinates that are the center of gravity of the imaging area. The case where the movement amount for overlapping is calculated has been described. In the seventh embodiment, a case will be described in which the amount of movement for sequentially moving the imaging region for each object is calculated when there are a plurality of objects for user operation in the projected image.

[Apparatus Configuration According to Embodiment 7]
The apparatus configuration according to the seventh embodiment will be described with reference to FIG. FIG. 21 is a functional block diagram illustrating an example of a device configuration according to the seventh embodiment. In the seventh embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description of the same components may be omitted. Specifically, functions, configurations, and processes other than the movement information calculation unit 721e described below are the same as those in the first embodiment.

  In the processing by the image detection unit 121a according to the present embodiment, a plurality of button images arranged in the projection image are detected by detecting a plurality of matching images that match the operation target image in the plurality of captured images. And Further, in the process by the inclusion determination unit 121d according to the present embodiment, it is determined whether or not the current imaging region includes all the button images. Further, in the processing by the second coordinate calculation unit 121c according to the present embodiment, all vertex coordinates of all button images are calculated.

  As illustrated in FIG. 21, the imaging device 20 includes an imaging unit 21. In addition, the projection device 30 includes a projection unit 31. The control device 700 includes a storage unit 110 and a control unit 720. Among these, the storage unit 110 includes a captured image storage unit 111. The control unit 720 includes an image detection unit 121a, a first coordinate calculation unit 121b, a second coordinate calculation unit 121c, an inclusion determination unit 121d, a movement information calculation unit 721e, and an imaging control unit 121f. The control unit 720 includes an operation detection unit 122a and a projection control unit 122b.

  The movement information calculation unit 721e calculates movement information for moving the imaging area. More specifically, the movement information calculation unit 721e accepts that a plurality of button images have been detected by the image detection unit 121a, and accepts a determination result from the inclusion determination unit 121d. At this time, when it is determined that the current imaging area does not include all the button images, the movement information calculation unit 721e uses the vertex coordinates of the current imaging area calculated by the first coordinate calculation unit 121b. A coordinate serving as the center of gravity of the current imaging region is calculated.

  The movement information calculation unit 721e selects an arbitrary button image from the plurality of button images when it is determined that the current imaging region does not include all the button images. Then, the movement information calculation unit 721e calculates coordinates serving as the center of gravity of the selected arbitrary button image from the vertex coordinates of all the button images calculated by the second coordinate calculation unit 121c.

  Subsequently, the movement information calculation unit 721e calculates, as movement information, the movement amount from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the selected arbitrary button image. The movement information calculation unit 721e can accept a user operation without moving the imaging area when it is determined that the current imaging area includes all the button images. Is calculated as “0”. Thereafter, the movement information calculation unit 721e notifies the imaging control unit 121f of the calculated movement amount.

  In addition, the movement information calculation unit 721e selects another button image when a predetermined time elapses without any user operation after notification of the movement amount. Whether or not a user operation has been performed may be determined by determining whether or not a user operation has been detected by the operation detection unit 122a. That is, when a user operation is performed, since the next projection image has been switched, processing for the next projection image is started.

  Then, the movement information calculation unit 721e calculates coordinates that are the center of gravity of the other selected button images. Subsequently, the movement information calculation unit 721e calculates, as movement information, the amount of movement from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the other selected button image. Here, since the coordinates that are the center of gravity of the current imaging region are in the same state as the coordinates that are the center of gravity of any button image, the coordinates that are the center of gravity of any button image that has already been calculated are What is necessary is just to utilize as a coordinate used as the gravity center of an area | region. Thereafter, the movement information calculation unit 721e notifies the imaging control unit 121f of the calculated movement amount. As described above, the movement information calculation unit 721e sequentially selects button images at predetermined time intervals until a user operation is performed, and calculates a movement amount for overlaying the imaging region on the sequentially selected button images. I will do it. In addition, when the button image to be selected has made a round, an arbitrary button image may be newly selected. However, if the imaging region is moved in the order selected in the first round, the movement amount is calculated again. Since it is not necessary, the processing load can be suppressed.

  FIG. 22 is a diagram illustrating movement of the imaging region according to the seventh embodiment. In the example shown in the upper part of FIG. 22, the projection image 1 includes a button image 2a, a button image 2b, a button image 2c, and an imaging region 3. Further, the imaging area 3 does not include any button image.

  In the state described above, the movement information calculation unit 721e calculates coordinates that are the center of gravity of the current imaging region 3 from the respective vertex coordinates of the current imaging region 3 calculated by the first coordinate calculation unit 121b. In addition, the movement information calculation unit 721e selects the button image 2c from among the plurality of button images. Then, the movement information calculation unit 721e calculates coordinates serving as the center of gravity of the button image 2c from the vertex coordinates of the button image 2c calculated by the second coordinate calculation unit 121c. Subsequently, the movement information calculation unit 721e calculates, as movement information, the amount of movement from the coordinates serving as the center of gravity of the current imaging region 3 to the coordinates serving as the center of gravity of the button image 2c. As a result, as shown in the upper part of the middle part of FIG. 22, a movement result is obtained in which the coordinates serving as the center of gravity of the imaging region 3 overlap the coordinates serving as the center of gravity of the button image 2c.

  After the movement of the imaging region 3, the movement information calculation unit 721e selects the button image 2a from among the plurality of button images when a predetermined time has elapsed without any user operation. Then, the movement information calculation unit 721e calculates coordinates serving as the center of gravity of the button image 2a from the vertex coordinates of the button image 2a calculated by the second coordinate calculation unit 121c. Subsequently, the movement information calculation unit 721e calculates the amount of movement from the coordinates serving as the center of gravity of the current imaging region 3 that are the same as the coordinates serving as the center of gravity of the button image 2c to the coordinates serving as the center of gravity of the button image 2a. Calculate as movement information. As a result, as shown in the lower part of the middle part of FIG. 22, a movement result is obtained in which the coordinates that become the center of gravity of the button image 2 a overlap the coordinates that become the center of gravity of the imaging region 3.

  After movement of the imaging region 3, the movement information calculation unit 721e selects the button image 2b from among the plurality of button images when a predetermined time has elapsed without any user operation. Then, the movement information calculation unit 721e calculates coordinates serving as the center of gravity of the button image 2b from the vertex coordinates of the button image 2b calculated by the second coordinate calculation unit 121c. Subsequently, the movement information calculation unit 721e calculates the amount of movement from the coordinate that is the center of gravity of the current imaging region 3 that is the same as the coordinate that is the center of gravity of the button image 2a to the coordinate that is the center of gravity of the button image 2b. Calculate as movement information. As a result, as shown in the lower part of FIG. 22, a movement result is obtained in which the coordinates serving as the center of gravity of the imaging region 3 overlap the coordinates serving as the center of gravity of the button image 2b.

  As described above, if no user operation is performed, a button image is sequentially selected every time a predetermined time elapses, and processing for moving the imaging region is executed. Further, as described above, it is preferable not to select the selected button image until the selection of the button image is completed.

[Moving amount calculation processing flow according to Embodiment 7]
Next, the flow of the movement amount calculation process according to the seventh embodiment will be described with reference to FIG. FIG. 23 is a flowchart illustrating an example of the flow of a movement amount calculation process according to the seventh embodiment. FIG. 23 illustrates processing subsequent to “Step S403: No” illustrated in FIG. That is, FIG. 23 shows the flow of the movement amount calculation process when all the button images are not included in the imaging region.

  As illustrated in FIG. 23, the movement information calculation unit 721e calculates a coordinate serving as the center of gravity of the current imaging region from the vertex coordinates calculated by the first coordinate calculation unit 121b (step S701). In addition, the movement information calculation unit 721e selects an arbitrary button image from the plurality of button images, and calculates a coordinate serving as the center of gravity of the selected button image from the vertex coordinates calculated by the second coordinate calculation unit 121c (step) S702).

  Then, the movement information calculation unit 721e calculates the amount of movement from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the selected button image (step S703). Thereafter, the movement information calculation unit 721e determines whether or not a predetermined time has elapsed (step S704). At this time, if the movement information calculation unit 721e determines that the predetermined time has elapsed (step S704: Yes), it determines whether or not a user operation has been performed based on the detection of the user operation by the operation detection unit 122a. Determination is made (step S705). On the other hand, when it is determined that the predetermined time has not elapsed (step S704: No), the movement information calculation unit 721e waits until the predetermined time has elapsed.

  If the movement information calculation unit 721e determines that no user operation has been performed (step S705: No), the movement information calculation unit 721e executes the process of step S701 again. However, in the process in step S701, when the movement amount of the imaging region is calculated from the second time onward, the coordinates serving as the center of gravity of the button image overlapping the current imaging region are used. Further, in the process in step S702, the center of gravity does not have to be calculated again when the button image to be selected has made a round. On the other hand, when the movement information calculation unit 721e determines that a user operation has been performed (step S705: Yes), the movement amount calculation process ends.

[Effects of Embodiment 7]
In the present embodiment, the amount of movement from the coordinates serving as the center of gravity of the imaging region to the coordinates serving as the center of gravity of the sequentially selected button images is calculated. As a result, according to the present embodiment, it is possible to improve the detection accuracy of user operations on the projection apparatus.

(Modification of Embodiment 7)
In the seventh embodiment, a case has been described in which when there are a plurality of objects for user operation in the projected image, the amount of movement for sequentially moving the imaging region for each object is calculated. In the modification of the seventh embodiment, a case will be described in which when a plurality of objects for user operation are present in the projection image, a movement amount for sequentially moving the imaging region is calculated according to the priority order of the objects. In addition, in the modification of Embodiment 7, it demonstrates using the functional block of the apparatus structure which concerns on Embodiment 7 shown in FIG.

  FIG. 24 is a diagram illustrating movement of the imaging region according to a modification of the seventh embodiment. In the example shown in the upper part of FIG. 24, the projection image 1 includes a button image 2a, a button image 2b, a button image 2c, and an imaging region 3. Further, the imaging area 3 does not include any button image. Also, the numbers enclosed in a circle shown in the upper part of FIG. 24 represent the priority order of each button image. In the example of FIG. 24, the priority order is higher in the order of the button image 2c, the button image 2b, and the button image 2a.

  In the state described above, the movement information calculation unit 721e calculates coordinates that are the center of gravity of the current imaging region 3 from the respective vertex coordinates of the current imaging region 3 calculated by the first coordinate calculation unit 121b. Further, the movement information calculation unit 721e acquires information on the priority order of the objects that move the imaging region 3. Such priority order information may be stored in the storage unit 110 or may be acquired from an external storage device, a server device, or the like. In the priority order information, the operation target image (corresponding to the object) and the priority order are associated with each other. Thereby, since the priority order of the coincidence image (corresponding to the object) detected by the image detection unit 121a can be recognized, the priority order of the objects arranged in the projection image 1 can also be recognized.

  Then, the movement information calculation unit 721e selects the button image 2c from among the plurality of button images according to the acquired priority order. Subsequently, the movement information calculation unit 721e calculates coordinates serving as the center of gravity of the button image 2c from the vertex coordinates of the button image 2c calculated by the second coordinate calculation unit 121c. Thereafter, the movement information calculation unit 721e calculates, as movement information, the amount of movement from the coordinates that are the center of gravity of the current imaging region 3 to the coordinates that are the center of gravity of the button image 2c. As a result, as shown in the upper part of the middle part of FIG. 24, a movement result is obtained in which the coordinates serving as the center of gravity of the imaging region 3 overlap the coordinates serving as the center of gravity of the button image 2c.

  After the movement of the imaging region 3, the movement information calculation unit 721e selects the button image 2b having the second highest priority after the button image 2c among the plurality of button images when a predetermined time has elapsed without any user operation. To do. Then, the movement information calculation unit 721e calculates coordinates serving as the center of gravity of the button image 2b from the vertex coordinates of the button image 2b calculated by the second coordinate calculation unit 121c. Subsequently, the movement information calculation unit 721e calculates the amount of movement from the coordinates that are the center of gravity of the current imaging region 3 that are the same as the coordinates that are the center of gravity of the button image 2c, to the coordinates that are the center of gravity of the button image 2b. Calculate as movement information. As a result, as shown in the lower part of the middle part of FIG. 24, a movement result is obtained in which the coordinates serving as the center of gravity of the imaging region 3 overlap the coordinates serving as the center of gravity of the button image 2b.

  After the movement of the imaging region 3, the movement information calculation unit 721e selects the button image 2a having the next highest priority after the button image 2b among the plurality of button images when a predetermined time has elapsed without any user operation. To do. Then, the movement information calculation unit 721e calculates coordinates serving as the center of gravity of the button image 2a from the vertex coordinates of the button image 2a calculated by the second coordinate calculation unit 121c. Subsequently, the movement information calculation unit 721e calculates the amount of movement from the coordinates serving as the center of gravity of the current imaging region 3 that is the same as the coordinates serving as the center of gravity of the button image 2b to the coordinates serving as the center of gravity of the button image 2a. Calculate as movement information. As a result, as shown in the lower part of FIG. 24, a movement result is obtained in which the coordinates serving as the center of gravity of the imaging region 3 overlap the coordinates serving as the center of gravity of the button image 2a.

  As described above, if no user operation is performed, a button image is selected according to the priority order every time a predetermined time elapses, and processing for moving the imaging region is executed. What is necessary is just to set a high priority with respect to the object with high frequency with which user operation is performed. As a result, an object that is frequently operated by a user is an object to be moved first in the imaging region, so that the projection image can be switched quickly.

[Moving Amount Calculation Processing Flow According to Modification of Embodiment 7]
Next, the flow of movement amount calculation processing according to a modification of the seventh embodiment will be described with reference to FIG. FIG. 25 is a flowchart illustrating an example of the flow of a movement amount calculation process according to a modification of the seventh embodiment. Note that FIG. 25 illustrates processing subsequent to “Step S403: No” illustrated in FIG. In other words, FIG. 25 shows the flow of the movement amount calculation process when all the button images are not included in the imaging region.

  As illustrated in FIG. 25, the movement information calculation unit 721e calculates coordinates that are the center of gravity of the current imaging region from the vertex coordinates calculated by the first coordinate calculation unit 121b (step S801). Further, the movement information calculation unit 721e acquires the priority order of the objects, selects a button image according to the acquired priority order, and calculates the center of gravity of the selected button image from the vertex coordinates calculated by the second coordinate calculation unit 121c. Is calculated (step S802).

  Then, the movement information calculation unit 721e calculates the amount of movement from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the selected button image (step S803). Thereafter, the movement information calculation unit 721e determines whether a predetermined time has elapsed (step S804). At this time, when the movement information calculation unit 721e determines that the predetermined time has elapsed (step S804: Yes), it determines whether or not a user operation has been performed based on the detection of the user operation by the operation detection unit 122a. Determination is made (step S805). On the other hand, when it is determined that the predetermined time has not elapsed (step S804: No), the movement information calculation unit 721e waits until the predetermined time has elapsed.

  Further, when the movement information calculation unit 721e determines that no user operation is performed (step S805: No), the movement information calculation unit 721e executes the process of step S801 again. However, in the process in step S801, when the movement amount of the imaging region is calculated for the second time and thereafter, the coordinates serving as the center of gravity of the button image overlapping the current imaging region are used. In the processing in step S802, the center of gravity does not have to be calculated again when the button image to be selected has made a round. On the other hand, when the movement information calculation unit 721e determines that a user operation has been performed (step S805: Yes), the movement amount calculation process ends.

[Effects of Modification of Embodiment 7]
In the present embodiment, the amount of movement from the coordinates serving as the center of gravity of the imaging region to the coordinates serving as the center of gravity of the button images sequentially selected according to the priority order is calculated. As a result, according to the present embodiment, it is possible to preferentially operate objects with higher user needs, so that the operability of user operations with respect to the projection apparatus can be improved.

(Embodiment 8)
In the above fourth to seventh embodiments, various calculation of the movement amount of the imaging region when there are a plurality of objects for user operation in the projection image has been described. In the eighth embodiment, a case will be described in which the calculation of the movement amount of the imaging region is set from various types.

[Apparatus Configuration according to Embodiment 8]
The apparatus configuration according to the eighth embodiment will be described with reference to FIG. FIG. 26 is a functional block diagram showing a device configuration example according to the eighth embodiment. In the eighth embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description of the same components may be omitted. Specifically, functions, configurations, and processes other than the movement setting information storage unit 812, movement information calculation unit 821e, and movement setting unit 821h described below are the same as those in the first embodiment.

  In the processing by the image detection unit 121a according to the present embodiment, a plurality of button images arranged in the projection image are detected by detecting a plurality of matching images that match the operation target image in the plurality of captured images. And Further, in the process by the inclusion determination unit 121d according to the present embodiment, it is determined whether or not the current imaging region includes all the button images. Further, in the processing by the second coordinate calculation unit 121c according to the present embodiment, all vertex coordinates of all button images are calculated.

  As illustrated in FIG. 26, the imaging device 20 includes an imaging unit 21. In addition, the projection device 30 includes a projection unit 31. The control device 800 includes a storage unit 810 and a control unit 820. Among these, the storage unit 810 includes a captured image storage unit 111 and a movement setting information storage unit 812.

  The movement setting information storage unit 812 stores movement setting information representing setting items related to movement of the imaging region. More specifically, the movement setting information storage unit 812 arbitrarily sets movement setting information for moving the imaging area to the button image closest to the imaging area, movement setting information for moving the imaging area to the largest button image, and an imaging area. Movement setting information for sequentially moving to the button image or movement setting information for moving the imaging region to the button image selected according to the priority order is stored. That is, in this embodiment, any one of the above movement setting information is stored in the movement setting information storage unit 812, and a movement amount calculation process corresponding to the movement setting information is executed. If the movement setting information is not stored in the movement setting information storage unit 812, a movement amount calculation process for moving the imaging region to an arbitrary button image is executed. The movement amount calculation process corresponding to each movement setting information has been described in the above fifth to seventh embodiments. Further, the movement amount calculation process for moving the imaging region to an arbitrary button image has been described in the fourth embodiment.

  The control unit 820 includes an image detection unit 121a, a first coordinate calculation unit 121b, a second coordinate calculation unit 121c, an inclusion determination unit 121d, a movement information calculation unit 821e, an imaging control unit 121f, and a movement setting unit 821h. And have. The control unit 820 includes an operation detection unit 122a and a projection control unit 122b.

  The movement setting unit 821h sets setting items related to movement of the imaging region. More specifically, the movement setting unit 821h moves the imaging area to the button image closest to the imaging area, moves the imaging area to the largest button image, and moves the imaging area to any button image. Move setting information such as sequentially moving the image area or moving the imaging area to the button image selected in accordance with the priority order. The user makes settings in advance from a terminal or the like connected to the control device 800 via a network. Then, the movement setting unit 821h stores the received movement setting information in the movement setting information storage unit 812.

  The movement information calculation unit 821e calculates movement information for moving the imaging region according to the movement setting information. More specifically, the movement information calculation unit 821e acquires movement setting information from the movement setting information storage unit 812. Then, the movement information calculation unit 821e moves according to the acquired movement setting information, a movement amount for moving the imaging region to the button image closest to the imaging region, and a movement amount for moving the imaging region to the largest button image. Then, a movement amount for sequentially moving the imaging region to an arbitrary button image or a movement amount for moving the imaging region to the button image selected according to the priority order is appropriately calculated. The calculation of each movement amount is the same as in the fifth to seventh embodiments. When the movement setting information does not exist in the movement setting information storage unit 812, a movement amount for moving the imaging region to an arbitrary button image is calculated as in the fourth embodiment.

[Movement amount calculation processing flow according to Embodiment 8]
Next, the flow of the movement amount calculation process according to the eighth embodiment will be described with reference to FIG. FIG. 27 is a flowchart illustrating an example of the flow of movement amount calculation processing according to the eighth embodiment. FIG. 27 illustrates a process after “Step S403: No” illustrated in FIG. In other words, FIG. 27 shows the flow of the movement amount calculation process when all the button images are not included in the imaging region.

  As illustrated in FIG. 27, the movement information calculation unit 821e determines whether movement setting information exists in the movement setting information storage unit 812 (step S901). At this time, when it is determined that the movement setting information does not exist (step S901: No), the movement information calculation unit 821e selects an arbitrary button image, and selects the selected arbitrary image from the coordinates serving as the center of gravity of the current imaging region. The amount of movement to the coordinates serving as the center of gravity of the button image is calculated (step S902).

  On the other hand, when the movement information calculation unit 821e determines that movement setting information exists (step S901: Yes), the movement information calculation unit 821e determines whether the movement setting information is to move the imaging area to the button image closest to the imaging area. (Step S903). At this time, when the movement information calculation unit 821e determines that the movement setting information is to move the imaging region to the button image closest to the imaging region (step S903: Yes), the movement amount calculation according to the movement setting information is performed. The process is executed to calculate the movement amount (step S904).

  On the other hand, when the movement information calculation unit 821e determines that the movement information is not movement setting information for moving the imaging area to the button image closest to the imaging area (step S903: No), the movement setting for moving the imaging area to the largest button image. It is determined whether the information is information (step S905). At this time, when it is determined that the movement information calculation unit 821e is movement setting information for moving the imaging region to the largest button image (step S905: Yes), a movement amount calculation process corresponding to the movement setting information is executed. Then, the movement amount is calculated (step S906).

  On the other hand, when it is determined that the movement information calculation unit 821e is not movement setting information for moving the imaging region to the largest button image (step S905: No), the movement information calculation unit 821e uses movement setting information for sequentially moving the imaging region to an arbitrary button image. It is determined whether or not there is (step S907). At this time, if it is determined that the movement information calculation unit 821e is movement setting information for sequentially moving the imaging region to an arbitrary button image (step S907: Yes), a movement amount calculation process corresponding to the movement setting information is performed. This is executed and the movement amount is calculated (step S908).

  On the other hand, if the movement information calculation unit 821e determines that it is not movement setting information for sequentially moving the imaging area to an arbitrary button image (step S907: No), the imaging area is moved to the button image selected according to the priority order. Since the movement setting information is to be moved, a movement amount calculation process corresponding to the movement setting information is executed to calculate the movement amount (step S909).

[Effects of Embodiment 8]
In the present embodiment, the movement amount calculation process of the imaging region is executed according to movement setting information that indicates which process should be executed. As a result, according to the present embodiment, since the movement amount calculation process desired by the user is executed, the operability of the user operation on the projection apparatus can be improved.

(Embodiment 9)
Although the embodiment of the projection system 10 according to the present invention has been described so far, the present invention may be implemented in various different forms other than the above-described embodiment. Therefore, different embodiments of (1) imaging area feedback, (2) object feedback, (3) imaging area and object shape, (4) configuration, and (5) program will be described.

(1) Feedback of imaging region In the above embodiment, when the area of the imaging region is smaller than the area of the button image, the center of gravity of the imaging region is set to the coordinates that are the center of gravity of the button image in order to facilitate user operation. A case has been described in which the amount of movement for overlapping the coordinates to be calculated is calculated. In the present embodiment, the imaging area may be fed back in order to facilitate user operation.

  FIG. 28 is a diagram illustrating an example of feedback of the imaging region. In the example shown in FIG. 28, the projected image 1 includes a button image 2 and an imaging region 3. In addition, the example illustrated in FIG. 28 represents a state when the imaging region 3 is moved. As shown in FIG. 28, when the imaging region 3 is moved, the imaging region 3 is displayed with a thick frame or the like so that the user can recognize the range in which the user operation can be detected. Such a projection image 1 is generated by the projection control unit 122b.

  In the embodiment described above, the detection accuracy of the user operation is improved without making the user aware of the imaging region. However, when the area of the imaging area is smaller than the area of the button image, the user operation cannot be detected in the entire area of the button image. For this reason, in the present embodiment, by generating a projection image in which the imaging region is displayed, the user is made aware of the imaging region and the detection accuracy of the user operation is improved. Note that any method may be used for displaying the imaging region as long as the current imaging region can be recognized by the user, such as blinking the imaging region, as well as displaying a frame.

(2) Object Feedback In the above embodiment, calculation of the movement amount of the imaging region when there are a plurality of objects for user operation in the projection image has been described. In the present embodiment, the object may be fed back in order to facilitate the user operation.

  FIG. 29 is a diagram illustrating an example of object feedback. In the example shown in FIG. 29, the projection image 1 includes a button image 2a, a button image 2b, a button image 2c, and an imaging region 3. In addition, the example illustrated in FIG. 29 represents a state when the imaging region 3 is moved. As shown in FIG. 29, when the imaging area 3 is moved, the button image 2c included in the imaging area 3 is changed to a predetermined display such as a color different from that of the other button images. Such a projection image 1 is generated by the projection control unit 122b.

  In the embodiment described above, the operability of the user operation is improved without making the user aware of the current imaging area. However, when there are a plurality of objects in the projection image, it is difficult for the user to recognize which object is an operable object. For this reason, in the present embodiment, by generating a projection image in which the color of an object included in the current imaging region is changed, the user can recognize an operable object and improve the operability of the user operation. In addition to changing the color, any change in the display of the object can be done by any method as long as the user can recognize the current operable object, such as blinking or transmitting the object. May be adopted. In addition, the object included in the current imaging area may include a plurality of objects depending on the arrangement of the object and the size of the imaging area. It is.

(3) Shape of Imaging Area and Object In the above embodiment, the shape of the projected image, the imaging area, the object, and the like has been described as being a square. In the present embodiment, these do not have to be square. For example, if a projected image, an imaging region, an object, or the like is a circle, the smallest square including the circle is adopted. That is, the above-described movement amount calculation process may be executed using a minimum square including a projection image, an imaging region, an object, and the like. However, as for the imaging area, if the smallest square including an imaging area that is not a square is adopted, the imaging area itself may become too large. The maximum square included in the region may be adopted. The object may be not only a button image but also other objects such as a combo box and a group box.

(4) Configuration In addition, information including processing procedures, control procedures, specific names, various data, parameters, and the like shown in the document and drawings can be arbitrarily changed unless otherwise specified. Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution or integration of each device is not limited to the one shown in the figure, and all or a part thereof is functionally or physically distributed in arbitrary units according to various burdens or usage conditions. Or they can be integrated.

  In the above embodiment, the projection system 10 including the imaging device 20, the projection device 30, and the control device 100 has been described. However, these devices can be functionally or physically distributed or integrated in arbitrary units. . For example, the imaging device 20, the projection device 30, and the control device 100 may be realized as an integrated device, or only the control device 100 and the projection device 30 may be realized as an integrated device. It can be changed. The control device 100 may be arranged on a server device connected to a network.

  In the above embodiment, when there are a plurality of objects, it is determined whether or not all the button images are included in the current imaging region. If all the button images are not included, the imaging region is moved. It was described as calculating the amount. In the present embodiment, the movement amount of the imaging region may be set to “0” even if all the button images are not included. For example, if the number of button images included in the imaging region, the ratio of the number of button images included in the imaging region to the total number of button images, or the like is greater than or equal to a certain value, the movement amount of the imaging region may be set to “0”. . Further, as described above, the embodiments can be appropriately combined within a range in which the contents are not contradictory.

(5) Program The control program executed by the control device 100 may be a file in an installable format or an executable format as a form of CD-ROM, flexible disk (FD), CD-R, DVD ( And recorded on a computer-readable recording medium such as a digital versatile disk. Further, the control program executed by the control device 100 may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The control program executed by the control device 100 may be provided or distributed via a network such as the Internet. The control program may be provided by being incorporated in advance in a ROM or the like.

  The control program executed by the control device 100 includes the above-described units (the image detection unit 121a, the first coordinate calculation unit 121b, the second coordinate calculation unit 121c, the inclusion determination unit 121d, the movement information calculation unit 121e, the imaging control unit 121f, The module configuration includes an operation detection unit 122a and a projection control unit 122b). As actual hardware, a CPU (processor) reads out and executes a control program from a storage medium, so that the above-described units are on the main storage device. The image detection unit 121a, the first coordinate calculation unit 121b, the second coordinate calculation unit 121c, the inclusion determination unit 121d, the movement information calculation unit 121e, the imaging control unit 121f, the operation detection unit 122a, and the projection control unit 122b are mainly used. It is generated on a storage device.

DESCRIPTION OF SYMBOLS 10 Projection system 20 Imaging apparatus 21 Imaging part 30 Projection apparatus 31 Projection part 100 Control apparatus 110 Storage part 111 Captured image storage part 120 Control part 121a Image detection part 121b 1st coordinate calculation part 121c 2nd coordinate calculation part 121d Inclusion determination part 121e Movement information calculation unit 121f Imaging control unit 122a Operation detection unit 122b Projection control unit

Japanese Patent No. 4710734 JP 2005-339269 A

Claims (20)

A projection unit that projects a projection image on a projection surface;
An imaging unit that captures an imaging region representing an imageable region and generates a captured image;
An image detection unit for detecting a coincidence image representing an image in the projection image that coincides with an operation target image representing an image to be operated by a user;
A first coordinate calculation unit that calculates first coordinates for specifying the position of the imaging region with respect to the projection image;
A second coordinate calculation unit for calculating a second coordinate for specifying the position of the coincidence image with respect to the projection image;
A movement information calculation unit that calculates movement information for moving the imaging region from the first coordinates and the second coordinates;
An imaging control unit that controls imaging by the imaging unit so as to move the imaging region according to the movement information;
An operation detection unit that detects a user operation on the matching image using the captured image;
A projection system, comprising: a projection control unit configured to generate a projection image in accordance with the detected user operation and to control projection of the generated projection image by the projection unit.   The movement information calculation unit calculates a movement amount from a coordinate serving as a centroid of the current imaging region to a coordinate serving as a centroid of the coincidence image as the movement information. Projection system.   The movement information calculation unit is a movement amount from the first coordinate to the second coordinate, and a movement amount that minimizes a distance between the corresponding coordinates of the current first coordinate and the second coordinate. The projection system according to claim 1, wherein the projection information is calculated as the movement information. An inclusion determination unit that determines whether or not the imaging region includes the coincidence image from the first coordinates and the second coordinates;
The said movement information calculation part calculates the said movement information, when it determines with the said imaging region including the said coincidence image, The movement information calculation part as described in any one of Claims 1-3 characterized by the above-mentioned. Projection system. An inclusion determination unit that determines whether the imaging region includes the coincidence image from the first coordinate and the second coordinate;
When it is determined that the imaging region does not include the coincidence image, it is determined from the first coordinate and the second coordinate whether the area of the imaging region is smaller than the area of the coincidence image. And an area determination unit to
The projection system according to claim 1, wherein the movement information calculation unit calculates the movement information when it is determined that an area of the imaging region is smaller than an area of the coincidence image.   The projection system according to claim 5, wherein the projection control unit generates a projection image in which the imaging region is displayed from the current first coordinate.   The movement information calculation unit selects an arbitrary matching image from the plurality of matching images when a plurality of the matching images are detected, and selects the selected arbitrary image from the coordinates serving as the center of gravity of the current imaging region. The projection system according to claim 1, wherein a movement amount to a coordinate serving as a center of gravity of the coincidence image is calculated as the movement information.   The movement information calculation unit calculates, when a plurality of the coincidence images are detected, a movement amount from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the centers of gravity of the plurality of coincidence images, The projection system according to claim 1, wherein a minimum movement amount is calculated as the movement information.   The movement information calculation unit calculates the areas of the plurality of matching images when a plurality of the matching images are detected, selects the matching images according to the calculated areas, and calculates the center of gravity of the current imaging region. 2. The projection system according to claim 1, wherein a movement amount from the coordinates to the coordinates serving as the center of gravity of the selected matching image is calculated as the movement information.   The movement information calculation unit sequentially selects arbitrary matching images from the plurality of matching images when a plurality of the matching images are detected, and sequentially selects the arbitrary matching images from the coordinates that are the center of gravity of the current imaging region. The projection system according to claim 1, wherein a movement amount to a coordinate serving as a center of gravity of the coincidence image is calculated as the movement information.   The movement information calculation unit sequentially selects matching images according to the priority order of the plurality of matching images determined in advance when a plurality of matching images are detected, and coordinates serving as the center of gravity of the current imaging region 2. The projection system according to claim 1, wherein a movement amount to a coordinate that is a center of gravity of the sequentially selected matching images is calculated as the movement information. It further has a movement setting unit for performing movement settings representing setting items related to movement of the imaging region,
The projection system according to claim 7, wherein the movement information calculation unit calculates the movement information according to the movement setting.   The projection control unit generates a projection image in which the display of the coincidence image included in the current imaging region is changed to a predetermined display from the current first coordinate and the second coordinate. The projection system according to any one of claims 7 to 11. An image detection step for detecting a coincidence image representing an image in a projection image that coincides with an operation target image representing an image to be operated by a user;
A first coordinate calculating step for calculating a first coordinate for specifying a position of an imaging region representing an imageable region with respect to the projection image;
A second coordinate calculating step for calculating a second coordinate for specifying the position of the coincidence image with respect to the projection image;
A movement information calculation step of calculating movement information for moving the imaging region from the first coordinates and the second coordinates;
An imaging control step of controlling imaging by an imaging device so as to move the imaging area according to the movement information;
An operation detection step of detecting a user operation on the matching image using a captured image captured by the imaging device;
A control program for causing a computer to execute a projection control step of generating a projection image in accordance with the detected user operation and controlling projection of the generated projection image by a projection device.   The movement information calculation step calculates a movement amount from the coordinates serving as the center of gravity of the current imaging region to the coordinates serving as the center of gravity of the coincidence image as the movement information. Control program.   The movement information calculation step is a movement amount from the first coordinate to the second coordinate, and a movement amount that minimizes the distance between the corresponding coordinates of the current first coordinate and the second coordinate. The control program according to claim 14, wherein the control information is calculated as the movement information. An inclusion determination step for determining whether or not the imaging region includes the coincidence image from the first coordinates and the second coordinates;
The movement information calculation step calculates the movement information when it is determined that the imaging area includes the coincidence image. Control program. An inclusion determination step of determining whether or not the imaging region includes the coincidence image from the first coordinates and the second coordinates;
When it is determined that the imaging region does not include the coincidence image, it is determined from the first coordinate and the second coordinate whether the area of the imaging region is smaller than the area of the coincidence image. And further performing an area determination step to perform,
The control program according to claim 14 or 15, wherein the movement information calculation step calculates the movement information when it is determined that an area of the imaging region is smaller than an area of the coincidence image. An image detection unit for detecting a coincidence image representing an image in a projection image that coincides with an operation target image representing an image to be operated by a user;
A first coordinate calculation unit that calculates a first coordinate for specifying a position of an imaging region representing an imageable region with respect to the projection image;
A second coordinate calculation unit for calculating a second coordinate for specifying the position of the coincidence image with respect to the projection image;
A movement information calculation unit that calculates movement information for moving the imaging region from the first coordinates and the second coordinates;
An imaging control unit that controls imaging by the imaging device so as to move the imaging region according to the movement information;
An operation detection unit that detects a user operation on the matching image using a captured image captured by the imaging device;
And a projection control unit configured to generate a projection image according to the detected user operation and control projection of the generated projection image by the projection apparatus. An image detection step for detecting a coincidence image representing an image in a projection image that coincides with an operation target image representing an image to be operated by a user;
A first coordinate calculating step for calculating a first coordinate for specifying a position of an imaging region representing an imageable region with respect to the projection image;
A second coordinate calculating step for calculating a second coordinate for specifying the position of the coincidence image with respect to the projection image;
A movement information calculation step of calculating movement information for moving the imaging region from the first coordinates and the second coordinates;
An imaging control step of controlling imaging by an imaging device so as to move the imaging area according to the movement information;
An operation detection step of detecting a user operation on the matching image using a captured image captured by the imaging device;
A projection control step of generating a projection image in response to the detected user operation and controlling projection of the generated projection image by a projection apparatus.

Images