JP2008271156A - Remote instruction system, information processor, and control program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a remote instruction system, information processor, and control program which can reduce input of annotation image projected in the state hard to be seen for a user on a projection domain side. <P>SOLUTION: A remote instruction system includes personal computers (PC) 1 and 2. These are mutually connected through a network 3. A projector 4, a video camera 5, and a three-dimensional measuring apparatus 7 are connected to the PC1. A control section 201 of the PC2 determines whether an annotation image is included to an area occupied by a background displayed on a display section 205 of the PC2 based on the data of a distance measured by the three-dimensional measuring apparatus 7 (step S16), and displays a warning dialog on the display section 205 when the annotation image is included to the area occupied by the background (step S19). <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a remote instruction system, an information processing apparatus, and a control program.

  Conventionally, a remote instruction system including a server (for example, a computer) to which a video camera and a projector are connected and a remote client (for example, a computer) connected to the server via a network is known (for example, Patent Documents 1 and 2).

  In this remote instruction system, a video camera connected to a server captures a three-dimensional object as an object, and transmits a captured image to a client via a network. The client displays a captured image including a three-dimensional object on a display screen of a display device connected to the client, and when an annotation image (line, figure, character) or the like is written on the display screen, the annotation image is displayed on the network. To the server via The server sends the received annotation image to the projector, and the projector sends the received annotation image to an appropriate position (annotation image projection region) of the projection region including the three-dimensional object (this is the client's annotation image within the captured image range). (Corresponding to the position where is written).

In the remote instruction system, the client user can write an instruction or the like as an annotation image on the display screen of the display device. This instruction is projected at an appropriate position in the projection area including the three-dimensional object, and the user on the projection area side can grasp the intention of the user of the client by viewing the projected instruction.
US Patent Publication No. 2004/0070674 JP 2005-33756 A

  However, if an instruction is written in a place other than the portion where the three-dimensional object is displayed on the display screen (that is, the background), the instruction is projected to the back of the three-dimensional object. It may become. As a result, there is often a problem that the user on the projection area side cannot grasp the intention of the user of the client even when viewing the projected instruction.

  In addition, the above-described problem also occurs for instructions written across the background and the three-dimensional object for the same reason.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a remote instruction system and an information processing apparatus capable of reducing the number of input annotation images projected in a state that is difficult for the user on the projection area side to see. And providing a control program. Specifically, distance information from the video camera (or projection device) is acquired in an area captured by the video camera, a head in the projection area of the annotation image is calculated from the distance information, and an annotation image is calculated according to the head. Is to provide a remote instruction system, an information processing apparatus, and a control program for controlling the display area and display form (including display enablement or disablement).

  In order to achieve the above object, a remote instruction system according to claim 1 is an imaging device that images an object, a projection device that projects an annotation image on a projection area including the object and a background, and the object from the imaging device. A first information processing apparatus connected to an object and a measuring apparatus for measuring a distance to the background; an input means for inputting the annotation image; and the input annotation image, the object and the background. A display unit; a determination unit that determines whether or not the annotation image is included in an area occupied by a background displayed on the display unit based on distance data measured by the measurement device; and the background occupies And a second information processing apparatus including a warning unit that issues a warning when the annotation image is included in the region.

  The remote instruction system according to claim 2, an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a distance from the imaging device to the object and the background A first information processing apparatus connected to a measurement apparatus for measuring the image, an input means for inputting the annotation image, distance data measured by the measurement apparatus, and an annotation image input by the input means Determining means for determining whether or not the annotation image is projected with a step on the object and the background by the projection device; and a step on the annotation image on the object and the background by the projection device. And a second information processing device including a warning unit that issues a warning when projected in a certain state.

  The remote instruction system according to a third aspect is the remote instruction system according to the second aspect, wherein the second information processing device has a distance measured by the measurement device for each point constituting the input annotation image. An acquisition unit that acquires data; and a dividing unit that performs cluster analysis on the acquired distance data and divides the annotation image into a plurality of clusters. When the difference between the distance measured by the measurement device corresponding to one cluster and the distance measured by the measurement device corresponding to the other divided cluster is larger than a predetermined distance, The projection device determines that the annotation image is projected with a step on the object and the background.

  The remote instruction system according to claim 4 is the remote instruction system according to any one of claims 1 to 3, wherein the warning means transmits the annotation image input by the input means to the projection apparatus. Or a response as to whether or not to correct the annotation image.

  The remote instruction system according to claim 5, an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a distance from the imaging device to the object and the background And an extraction means for extracting an area occupied by the background in a photographed image photographed by the photographing device based on distance data measured by the measuring device. A second information processing apparatus comprising: an information processing apparatus; an input unit that inputs the annotation image; and a setting unit that sets an area occupied by the extracted background as an input prohibited area of the annotation image. Features.

  The remote instruction system according to claim 6, an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a distance from the imaging device to the object and the background A region occupied by the background is extracted from a captured image captured by the imaging device based on data of a distance measured by the first information processing device connected to the measurement device and the measurement device. A second information processing apparatus comprising: extraction means; input means for inputting the annotation image; and setting means for setting an area occupied by the extracted background as an input prohibited area of the annotation image. To do.

  The remote instruction system according to claim 7 is the remote instruction system according to claim 5 or 6, wherein the second information processing apparatus includes display means for displaying the object and the background, and the background is displayed by the input means. When an annotation image is input to an area occupied by the background, the display means displays information indicating input prohibition in the area occupied by the background.

  The remote instruction system according to claim 8, an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a distance from the imaging device to the object and the background A region occupied by the background is extracted from a captured image captured by the imaging device based on data of a distance measured by the first information processing device connected to the measurement device and the measurement device. Extraction means, input means for inputting the annotation image, and annotation image projection area onto which the annotation image is projected by the projection device, the annotation image is moved and / or reduced and the annotation image is projected. And a second information processing apparatus comprising adjustment means for adjusting the projection so as to be projected onto a region having no step.

  The information processing apparatus according to claim 9, an imaging apparatus that captures an object, a projection apparatus that projects an annotation image on a projection area including the object and the background, and a distance from the imaging apparatus to the object and the background An information processing device connected via a communication network to an external information processing device connected to a measuring device for measuring the input image, the input means for inputting the annotation image, the input annotation image, and the object And a display means for displaying the background, and a determination means for determining whether or not the annotation image is included in an area occupied by the background displayed on the display means based on the distance data measured by the measuring device And warning means for giving a warning when the annotation image is included in the area occupied by the background.

  The information processing apparatus according to claim 10, an imaging apparatus that captures an object, a projection apparatus that projects an annotation image on a projection area including the object and the background, and a distance from the imaging apparatus to the object and the background An information processing device connected via a communication network to an external information processing device connected to a measurement device that measures the distance, input means for inputting the annotation image, and distance data measured by the measurement device And a determination unit that determines whether the annotation image is projected with a step on the object and the background based on the annotation image input by the input unit; and the projection device When the annotation image is projected in a state where there is a step on the object and the background, warning means for performing a warning is provided.

  The information processing apparatus according to claim 11, an imaging apparatus that captures an object, a projection apparatus that projects an annotation image on a projection area including the object and the background, and a distance from the imaging apparatus to the object and the background And an extraction means for extracting an area occupied by the background in a photographed image photographed by the photographing device based on distance data measured by the measuring device. An information processing apparatus connected to the information processing apparatus via a communication network, the input means for inputting the annotation image, and the setting means for setting an area occupied by the extracted background as an input prohibited area of the annotation image It is characterized by providing.

  The information processing apparatus according to claim 12, an imaging apparatus that captures an object, a projection apparatus that projects an annotation image onto a projection area including the object and the background, and a distance from the imaging apparatus to the object and the background An information processing apparatus connected to an external information processing apparatus connected to a measurement apparatus that measures the image, and photographed by the imaging apparatus based on distance data measured by the measurement apparatus Extraction means for extracting the area occupied by the background in the captured image, input means for inputting the annotation image, and setting means for setting the area occupied by the extracted background as an input prohibited area of the annotation image It is characterized by providing.

  The information processing apparatus according to claim 13, an imaging apparatus that captures an object, a projection apparatus that projects an annotation image onto a projection area including the object and the background, and a distance from the imaging apparatus to the object and the background An information processing apparatus connected to an external information processing apparatus connected to a measurement apparatus for measuring the image, and based on distance data measured by the measurement apparatus, in a captured image captured by the imaging apparatus An extraction means for extracting an area occupied by the background, an input means for inputting the annotation image, and a movement of the annotation image when there is a step in the annotation image projection area onto which the annotation image is projected by the projection device. And / or adjustment means for adjusting the annotation image so that the annotation image is projected onto an area having no step.

  A control program according to a fourteenth aspect includes a photographing device for photographing an object, a projection device for projecting an annotation image onto a projection area including the subject and the background, and a distance from the photographing device to the object and the background. A computer connected via a communication network to an external information processing device connected to a measuring device for measurement displays the input means for inputting the annotation image, the input annotation image, the object and the background. Display means, discrimination means for determining whether or not the annotation image is included in the area occupied by the background displayed on the display means, based on the distance data measured by the measuring device, and the area occupied by the background When the annotation image is included in the image, it is made to function as a warning means for giving a warning.

  According to a fifteenth aspect of the present invention, there is provided a control program for photographing an object, a projection apparatus for projecting an annotation image onto a projection area including the object and the background, and a distance from the photographing apparatus to the object and the background. A computer connected via a communication network to an external information processing device connected to a measuring device to be measured is input using the input means for inputting the annotation image, the distance data measured by the measuring device, and the input means. And determining means for determining whether or not the annotation image is projected with a step on the object and the background based on the annotation image, and the annotation image is determined by the projection device. When projected with a step on the object and the background, it functions as a warning means for giving a warning.

  A control program according to a sixteenth aspect includes a photographing device for photographing an object, a projection device for projecting an annotation image onto a projection region including the subject and the background, and a distance from the photographing device to the object and the background. External information that is connected to a measuring device to be measured and includes an extracting unit that extracts a region occupied by the background in a photographed image photographed by the photographing device based on distance data measured by the measuring device. A computer connected to a processing apparatus via a communication network is caused to function as an input unit for inputting the annotation image and a setting unit for setting an area occupied by the extracted background as an input prohibited area of the annotation image. Features.

  The control program according to claim 17 includes: an imaging device that captures an object; a projection device that projects an annotation image onto a projection area including the object and the background; and a distance from the imaging device to the object and the background. A computer connected to an external information processing apparatus connected to a measuring apparatus to be measured via a communication network is connected to a computer within a captured image captured by the imaging apparatus based on distance data measured by the measuring apparatus. An extraction means for extracting an area occupied by the background, an input means for inputting the annotation image, and a setting means for setting the area occupied by the extracted background as an input prohibited area of the annotation image, .

  The control program according to claim 18 includes: an imaging device that captures an object; a projection device that projects an annotation image onto a projection area including the object and the background; and a distance from the imaging device to the object and the background. An area occupied by the background in a photographed image photographed by the photographing device based on distance data measured by the measuring device on a computer connected to an external information processing device connected to the measuring device to be measured Extraction means for extracting the annotation image, input means for inputting the annotation image, and when there is a step in the annotation image projection area onto which the annotation image is projected by the projection device, the annotation image is moved and / or reduced to reduce the annotation It is characterized by functioning as an adjusting means for adjusting so that an image is projected onto an area having no step.

  According to the first, second, eighth, ninth, tenth, thirteenth, fourteenth, fifteenth, and eighteenth inventions, it is possible to reduce the number of input annotation images projected in a state that is difficult for the user on the projection area side to see.

  According to the invention of claim 3, it is possible to accurately determine that the annotation image is projected by the projection device with a step on the object and the background.

  According to the invention of claim 4, the user of the second information processing apparatus can select whether to transmit the annotation image or to correct the annotation image.

  According to the fifth, sixth, eleventh, twelfth, sixteenth and seventeenth aspects, the area occupied by the background in the photographed image is set as the input prohibited area of the annotation image, so that it is difficult for the user on the projection area side to see. The input of the projected annotation image can be reduced.

  According to the invention of claim 7, it is possible to notify the user of the second information processing apparatus that the input of the annotation image is prohibited.

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a configuration of a remote instruction system according to an embodiment of the present invention.

  The remote instruction system in FIG. 1 includes a personal computer (PC) 1 (first information processing apparatus, external information processing apparatus) that functions as a server, and a PC 2 (second information processing apparatus, information processing apparatus) that functions as a client. These are connected to each other via the network 3. Connected to the PC 1 are a projector 4 (projection device), a video camera 5 (imaging device), and a three-dimensional measurement device 7 (measurement device). The projector 4 irradiates the object 8 and the screen 10 placed on the table 9 with a simple light beam via the half mirror 6 based on a control command from the PC 1 or an annotation image (annotation image). Project. Note that the annotation image includes images of all aspects such as lines, characters, symbols, figures, colors, fonts (typefaces), and the like.

  The video camera 5 captures a reflected image of the screen 10 including the object 8 via the half mirror 6 and outputs the captured image to the PC 1.

  The three-dimensional measuring apparatus 7 measures the distances from the lens surface of the video camera 5 to the object 8, the table 9, and the screen 10, and outputs the respective distance data to the PC 1. More specifically, in the case where the photographed image photographed by the video camera 5 is as shown in FIG. 2, the three-dimensional measuring device 7 can detect points (points on each coordinate of the photographed image) within the photographing region from the video camera 5. All distances d (x, y) to (x, y)) are measured. Since the captured image is displayed in the display area 12 of the display unit 205, the coordinates (x, y) in the captured image coincide with the coordinates (x, y) in the display area 12.

  The PC 1 outputs the captured image captured by the video camera 5 to the PC 2 via the network 3. The PC 2 is connected to a display unit 205, and the display unit 205 displays a display area 12 for a captured image and a user interface (UI) 14. Note that the PC 2 may be configured as an integrated personal computer including the display unit 205.

  The UI 14 includes a group of buttons such as pen, text, and erase, and icons of line types and color types. In the display area 12, a captured image captured by the video camera 5 is displayed. In FIG. 1, a photographed image obtained by photographing the screen 10 including the object 8 with the video camera 5 is displayed in the display area 12.

  For example, when a pen button on the UI 14 is pressed and a figure or the like is written on the object 8 in the display area 12, information on the figure is output from the PC 2 to the projector 4 via the PC 1. The projector 4 draws a graphic on the object 8 based on the graphic information.

  Further, the PC 2 outputs a control command to the PC 1, and the operations of the projector 4 and the video camera 5 (for example, the shooting angle of the video camera 5, the brightness of the captured image, and the brightness of the projection image of the projector 4). Can be controlled.

  In the figure, one client PC 2 is shown, but the remote instruction system may include a plurality of clients (PCs).

  FIG. 3 is a block diagram illustrating a functional configuration of the PCs 1 and 2.

  The PC 1 controls the entire apparatus and controls the operation of the projector 4, the video camera 5, and the three-dimensional measuring apparatus 7, and transmission / reception that transmits / receives information and data to / from the PC 2 via the network 3. Unit 102, a storage unit 103 that stores control programs, data, information, and the like, and an interface (IF) unit 104 for connecting to projector 4, video camera 5, and three-dimensional measurement 7. The control unit 101 is connected to the transmission / reception unit 102, the storage unit 103, and the IF unit 104, and is further connected to the projector 4, the video camera 5, and the three-dimensional measurement apparatus 7 via the IF unit 104.

  The PC 2 includes a control unit 201 (determination unit, warning unit, acquisition unit, division unit, extraction unit, setting unit) that controls the entire apparatus, and a transmission / reception unit 202 that transmits / receives information and data to / from the PC 1 via the network 3. , An operation unit 203 (input means) composed of a mouse and a keyboard, a storage unit 204 for storing control programs, data, information, and the like, and a display unit 205 (display means, warning means). The control unit 201 is connected to the transmission / reception unit 202, the operation unit 203, the storage unit 204, and the display unit 205.

  FIG. 4 is a block diagram illustrating a hardware configuration of the PC 2.

  The PC 2 is a CPU 21 for controlling the entire apparatus, a ROM 22 having a control program, a RAM 23 functioning as a working area, a hard disk drive (HDD) 24 having various information and programs, a mouse and keyboard 25, and for connecting to other computers. A network interface 26, a display 27 composed of a liquid crystal monitor or CRT, and a USB (universal serial bus) interface 28 for connecting to a USB device (not shown) are provided. The CPU 21 is connected to a ROM 22, a RAM 23, a hard disk drive (HDD) 24, a mouse and keyboard 25, a network interface 26, a display 27, and a USB interface 28 via a system bus 29.

  The control unit 201 corresponds to the CPU 21 that executes various processes according to the control program. The transmission / reception unit 202 corresponds to the network interface 26, and the storage unit 203 corresponds to the hard disk drive (HDD) 24. The operation unit 204 corresponds to the mouse and keyboard 25, and the display unit 305 corresponds to the display 27.

  The PC 1 is configured to exclude the mouse and keyboard 25 and the display 27 from the configuration of the PC 2, but may include the mouse and keyboard 25 and the display 27. Note that the projector 4, the video camera 5, and the three-dimensional measuring device 7 are connected to the PC 1 via a USB interface.

  FIG. 5 is a flowchart showing processing executed by the PC1 and the PC2.

  First, processing executed on the PC 1 will be described.

  The control unit 101 of the PC 1 receives a captured image from the video camera 5 (step S1). The captured image is sent from the video camera 5 as a frame image every 0.5 seconds, for example, and stored in the storage unit 103.

  The control unit 101 calculates the difference between the captured images (frame images) and determines whether there is a change in the captured image due to the movement of the object 8 or the video camera 5 (step S2).

  If NO in step S2, the process proceeds to step S5 described later. On the other hand, in the case of YES in step S2, the three-dimensional measuring device 7 performs three-dimensional measurement by a known method such as a stereo method or an optical laser method, that is, a point (photographed image) from the video camera 5 in the photographing region. All the distances d (x, y) to the point (x, y)) on each coordinate are measured (step S3). The control unit 101 receives distance data (that is, data of all the measured distances d (x, y)) from the three-dimensional measuring device 7 and transmits it to the PC 2 (step S4).

  Thereafter, the control unit 101 outputs an initial projection image to the projector 4 (step S5). The initial projection image is a black image.

  Next, the control unit 101 outputs the captured image captured by the video camera 5 to the PC 1 via the transmission / reception unit 102 and the network 3 (step S6), and waits until a command is received from the PC 2 (step S7). The command from the PC 2 is specifically annotation image data. For example, when the annotation image is a circle, it is information in which the center of the circle and the length of the radius are converted into text.

  Next, when the transmission / reception unit 102 receives a drawing command from the PC 2 (step S8), the control unit 101 executes an image projection process according to the drawing command (step S9). Specifically, when the annotation image data is received from the PC 2, the control unit 101 outputs the annotation image data to the projector 4. The projector 4 projects an annotation image based on this data.

  Next, the control unit 101 determines whether or not all the processes to be executed in accordance with the command from the PC 2 have been completed (step S10). If NO in step S10, the control unit 101 returns to step S2. On the other hand, if YES in step S10, the process of PC1 is terminated.

  Next, processing executed on the PC 2 will be described.

  When receiving an input of a command from the operation unit 204 (step S11), the control unit 201 determines whether a screen processing command for drawing an annotation image is received from the operation unit 204 (step S12).

  If NO in step S12, the process proceeds to step S18 described later. If YES in step S12, the control unit 201 draws an annotation image in the display area 12 in accordance with the screen processing command (step S13).

  Next, the control unit 201 collates the distance data received from the PC 1 via the transmission / reception unit 202 (data of the distance d (x, y)) with the annotation image drawn in the display area 12 to display the display area 12. Distance data (distance d (x, y) data) is acquired for each point constituting the annotation image drawn in (Step S14).

  Then, the control unit 201 performs a known cluster analysis on the acquired distance data, and divides the annotation image into a plurality of clusters (step S15).

  Next, the control unit 201 determines the distance from the object 8 or the screen 10 corresponding to the divided one cluster to the video camera 5 and the object 8 or the screen corresponding to the other divided cluster. It is determined whether or not the difference from the distance from 10 to the video camera 5 is smaller than a predetermined distance (step S16). Here, it is determined whether or not the annotation image is projected in a state where the annotation image extends over the object 8 and the screen 10 and there is a step in the depth direction from the object 8 to the screen 10. When the difference in distance is smaller than the predetermined distance, the annotation image is projected with almost no step in the depth direction. However, when the difference in distance is larger than the predetermined distance, the annotation image is projected in the depth direction. An annotation image is projected with a step.

  For example, the distance of the portion projected on the object 8 in the annotation image (that is, the distance from the object 8 to the video camera 5) is “0.90 to 1.00”, and the cluster analysis is executed. As a result, the annotation image of the portion projected on the object 8 is divided into cluster A (distance “0.90 to 0.95”) and cluster B (distance “0.96 to 1.00”). Suppose that In this case, since the difference between the distances of the clusters A and B is smaller than a predetermined distance (for example, 0.1), the annotation image is projected with almost no step in the depth direction.

  On the other hand, for example, the distance of the portion projected on the object 8 in the annotation image (that is, the distance from the object 8 to the video camera 5) is “1”, and the distance of the portion projected on the screen 10 ( That is, when the processing from the screen 10 to the video camera 5 is “2”, as a result of executing the cluster analysis, the annotation image is a cluster C (distance “1”) of the portion projected on the object 8. And the cluster D (distance “2”) of the portion projected on the screen 10. In this case, since the difference between the distances of the clusters C and D is larger than a predetermined distance (for example, 0.1), the annotation image is projected with a step in the depth direction.

  Further, through the processing of steps S14 to S16, the control unit 201 has an annotation image in an area occupied by the screen 10 (background) displayed on the display unit 205 based on the distance data measured by the three-dimensional measuring device 7. It is determined whether or not it is included.

  If YES in step S16, the control unit 201 transmits the annotation image as a drawing command to the PC 1 via the transmission / reception unit 202 (step S17). The drawing command transmitted to the PC 1 is received by the transmission / reception unit 102 in the process of step S8.

  The control unit 201 determines whether or not all the processes to be executed by the PC 2 have been completed (step S18). If NO in step S18, the control unit 201 returns to step S11. On the other hand, if YES in step S18, the process of PC2 is terminated.

  If NO in step S16, the control unit 201 displays a warning dialog shown in FIG. 6A on the display area 12 of the display unit 205 (step S19). If the “Send” button in the warning dialog of FIG. 6A is pressed, the process proceeds to step S17. If the “Do not send” button is pressed, the process proceeds to step S18. As shown in FIG. 6B, the warning dialog displayed in step S19 may be a dialog that recommends correction of the annotation image and requests a response as to whether or not to correct the annotation image.

  As described above in detail, according to the present embodiment, the control unit 201 annotates the area occupied by the background displayed on the display unit 205 based on the distance data measured by the three-dimensional measurement device 7. Whether or not an image is included is determined, and when an annotation image is included in the area occupied by the background, a warning dialog is displayed on the display unit 205. Image input can be reduced.

  Further, the control unit 201 uses the distance data measured by the three-dimensional measurement device 7 and the annotation image input via the operation unit 204 to cause the annotation image to be stepped on the object 8 and the screen 10 by the projector 4. It is determined whether or not the image is projected in a certain state, and when the annotation image is projected on the object 8 and the screen 10 with a step, a warning dialog is displayed on the display unit 205, so It is possible to reduce the input of annotation images projected in a state that is difficult for the user to see.

  Further, the control unit 201 acquires distance data measured by the three-dimensional measuring device 7 for each point constituting the input annotation image, and performs cluster analysis on the acquired distance data. Then, the annotation image is divided into a plurality of clusters. The control unit 201 measures the distance measured by the three-dimensional measuring device 7 corresponding to the one divided cluster and the distance measured by the three-dimensional measuring device 7 corresponding to the other divided cluster. When the difference is larger than a predetermined distance, the projector 4 determines that the annotation image is projected on the object 8 and the screen 10 with a step.

  Therefore, it is possible to accurately determine that the annotation image is projected on the object 8 and the screen 10 with a step by the projector 4.

  Furthermore, the control unit 201 obtains a response whether to transmit the annotation image to the projector 4 via the display unit 205 or a response whether to correct the annotation image. Therefore, the user of the PC 2 can select whether to transmit the annotation image to the projector 4 or whether to correct the annotation image.

  In the above embodiment, when an annotation image is included in the area occupied by the background, a warning dialog is displayed on the display unit 205. For example, when the annotation image straddles the background and the object, or the annotation image 10, the control unit 201 moves the annotation image so that the annotation image is projected onto a region with few steps as shown in FIG. Or you may make it reduce. Thereby, it is possible to reduce the input of the annotation image projected in a state that is difficult for the user on the projection area side to see.

  In this case, the control unit 201 temporarily moves or reduces the annotation image, performs a known cluster analysis on the distance data for each point constituting the moved or reduced annotation image, and generates a plurality of annotation images. Divide into clusters. After that, the control unit 201 executes the same process as step S16. If NO in step S16, the control unit 201 temporarily moves or reduces the annotation image and executes a known cluster analysis. Repeat until YES in step S16. If YES in step S16, the control unit 201 transmits the annotation image after movement or reduction to the PC 1 via the transmission / reception unit 202 as a drawing command. As a result, the annotation image is projected onto an area having few steps. As described above, the control unit 201 can adjust the annotation image so that the annotation image is projected onto an area having no step by moving and / or reducing the annotation image.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings.

  In the first embodiment, when the control unit 201 determines that the annotation image is projected in a state where there is a step in the depth direction from the object 8 to the screen 10, a warning is given and the projection area side The input of annotation images projected in a state that is difficult for the user to see was reduced. In the present embodiment, the input of the annotation image to the area on the screen 10, that is, the background area is prohibited in advance, thereby reducing the input of the annotation image projected in a state that is difficult to see for the user on the projection area side.

  Note that the configuration of the remote instruction system according to the present embodiment is the same as the configuration of the remote instruction system according to the first embodiment described above, and a description thereof will be omitted.

  FIG. 7 is a diagram illustrating a display example of the display area 12 of the display unit 205.

  As shown in the figure, when the user of the PC 2 tries to input an annotation image to the area of the screen 10, that is, the background area via the operation unit 204, a cross icon 40 is displayed, and the annotation image is displayed. Input is prohibited. On the other hand, when the user of the PC 2 tries to input an annotation image to the area of the target object 8 via the operation unit 204, the cross icon 41 is displayed, and the input of the annotation image is executed. Note that the input of the annotation image to the table 9 is not prohibited, but can be prohibited by the setting of the control unit 201 described later.

  FIG. 8 is a flowchart showing processing executed by the PC1 and the PC2. The same processes as those in the flowchart of FIG. 5 are given the same step numbers, and the description thereof is omitted.

  First, after the process of step S3, the control unit 101 receives distance data (that is, data of all measured distances d (x, y)) from the three-dimensional measurement apparatus 7, and applies K to the distance data. -Data analysis such as an average method is executed, and an area of the screen 10, that is, a background area is extracted from the photographed image (step S31). The extracted background area data is a collection of data of coordinates (x, y) in the captured image.

  Next, the control unit 101 transmits the extracted background area data to the PC 2 via the transmission / reception unit 102 (step S32), and proceeds to step S5.

  The control unit 201 of the PC 2 receives a command input from the operation unit 204 (step S11). Next, the control unit 201 receives the background area data from the PC 1, and sets the background area as an annotation image input prohibited area based on the background area data (step S33). Thereafter, the process proceeds to step S12. Other processing is the same as the processing of FIG.

  In FIG. 8, the control unit 201 does not execute the processes of steps S14 to S16 and S19 of FIG.

  FIG. 9 is a flowchart showing a modification of the processing executed by PC1 and PC2 of FIG. The same processes as those in the flowchart of FIG. 5 are given the same step numbers, and the description thereof is omitted.

  In FIG. 9, unlike FIG. 8, the control unit 201 of the PC 2 executes the background region extraction processing in step S <b> 31.

  Specifically, the control unit 101 of the PC 1 receives distance data (that is, data of all measured distances d (x, y)) from the three-dimensional measuring device 7 and transmits it to the PC 2 (step S4). .

  The control unit 201 of the PC 2 receives distance data (that is, data of all measured distances d (x, y)) from the three-dimensional measuring device 7 via the PC 1 and performs K-average on the distance data. Data analysis such as a method is executed, and an area of the screen 10, that is, a background area is extracted from the captured image (step S31). The extracted background area data is a collection of data of coordinates (x, y) in the captured image.

  Next, the control unit 201 sets the background area as an input prohibition area of the annotation image based on the data of the background area (step S33), and proceeds to the process of step S12. Other processing is the same as the processing of FIG.

  As described above in detail, according to the present embodiment, the control unit 101 of the PC 1 or the control unit 201 of the PC 2 is based on the distance data measured by the three-dimensional measurement device 7 in the captured image. The area occupied by the background is extracted, and the area occupied by the extracted background is set as the annotation image input prohibited area. Therefore, it is possible to reduce the input of the annotation image projected in a state that is difficult for the user on the projection area side to see.

  When an attempt is made to input an annotation image to the background area, the cross icon 40 is displayed, so that the user of the PC 2 can be notified that the input of the annotation image is prohibited.

  The above-mentioned implementation can also be realized by supplying a recording medium in which a software program for realizing the functions of PC1 to PC2 is recorded to each PC, and the CPU of each PC reads and executes the program stored in the storage medium. The same effect as that of the embodiment is produced. Examples of the storage medium for supplying the program include a CD-ROM, a DVD, or an SD card.

  Further, the same effects as those of the above-described embodiment can also be obtained when the CPU of each PC executes a software program for realizing the function of each PC.

  Note that the present invention is not limited to the above-described embodiment, and can be implemented with various modifications without departing from the scope of the invention.

It is a block diagram which shows the structure of the remote instruction | indication system which concerns on the 1st Embodiment of this invention. It is a figure which shows the relationship between a picked-up image and the coordinate of the said picked-up image. It is a block diagram which shows the function structure of PC1-2. It is a block diagram which shows the hardware constitutions of PC2. It is a flowchart which shows the process performed by PC1 and PC2. (A) is a figure which shows an example of a warning dialog, (B) is a figure which shows the other example of a warning dialog. It is a figure which shows the example of a display of the display area 12 of the display part 205 which concerns on 2nd Embodiment. It is a flowchart which shows the process performed by PC1 and PC2. It is a flowchart which shows the modification of the process performed by PC1 and PC2 of FIG. It is a figure which shows the state which moved or reduced the annotation image.

Explanation of symbols

1 PC (first information processing device)
2 PC (second information processing device)
3 Network 4 Projector (Projector)
5 Video camera (imaging device)
6 Half mirror 7 Three-dimensional measuring device (measuring device)
101 Control unit (extraction means)
102 transmission / reception unit 103 storage unit 104 IF unit 105 display unit 201 control unit (determination unit, warning unit, acquisition unit, division unit, extraction unit, setting unit)
202 Transmission / reception unit 203 Operation unit (input means)
204 Storage unit 205 Display unit (warning means, display means)

Claims (18)

Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background A first information processing apparatus,
Input means for inputting the annotation image;
Display means for displaying the inputted annotation image, the object and the background;
A determination unit that determines whether or not the annotation image is included in an area occupied by a background displayed on the display unit, based on distance data measured by the measurement device;
A remote instruction system comprising: a second information processing apparatus including a warning unit that issues a warning when the annotation image is included in an area occupied by the background. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background A first information processing apparatus,
Input means for inputting the annotation image;
Whether or not the annotation image is projected with a step on the object and the background by the projection device based on the distance data measured by the measurement device and the annotation image input by the input unit Discriminating means for discriminating;
A remote instruction system comprising: a second information processing device including a warning unit that issues a warning when the annotation image is projected with a step between the object and the background by the projection device. . The second information processing apparatus
Obtaining means for obtaining distance data measured by the measuring device for each point constituting the input annotation image;
A clustering unit for performing cluster analysis on the acquired distance data and dividing the annotation image into a plurality of clusters;
The discriminating means has a predetermined difference between a distance measured by the measuring device corresponding to the one divided cluster and a distance measured by the measuring device corresponding to the other divided cluster. 3. The remote indication system according to claim 2, wherein when the distance is larger than the distance, the projection apparatus determines that the annotation image is projected with a step on the object and the background. 4. The warning means obtains a response as to whether or not to transmit the annotation image input by the input means to the projection device or a response as to whether or not to correct the annotation image. The remote instruction system according to any one of the above. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background And a first information processing apparatus comprising extraction means for extracting an area occupied by the background in a photographed image photographed by the photographing apparatus based on distance data measured by the measuring apparatus;
A remote instruction system comprising: an input unit that inputs the annotation image; and a second information processing apparatus that includes a setting unit that sets an area occupied by the extracted background as an input prohibited area of the annotation image. . Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background A first information processing apparatus,
Based on the distance data measured by the measurement device, the extraction means for extracting the area occupied by the background in the photographed image photographed by the photographing device, the input means for inputting the annotation image, and the extraction And a second information processing apparatus comprising: setting means for setting an area occupied by the background as an input prohibited area of the annotation image. The second information processing apparatus includes display means for displaying the object and the background, and when the annotation means is input to the area occupied by the background by the input means, the display means displays the area occupied by the background. 7. The remote instruction system according to claim 5, wherein information indicating prohibition of input is displayed on the screen. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background A first information processing apparatus,
Based on the distance data measured by the measurement device, an extraction unit that extracts an area occupied by the background in the captured image captured by the imaging device, an input unit that inputs the annotation image, and the projection device Adjusting means for moving and / or reducing the annotation image so that the annotation image is projected onto an area without a step when the annotation image projection area onto which the annotation image is projected has a step. And a second information processing apparatus. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background An information processing apparatus connected to an external information processing apparatus via a communication network,
Input means for inputting the annotation image;
Display means for displaying the inputted annotation image, the object and the background;
A determination unit that determines whether or not the annotation image is included in an area occupied by a background displayed on the display unit, based on distance data measured by the measurement device;
An information processing apparatus comprising: a warning unit that issues a warning when the annotation image is included in an area occupied by the background. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background An information processing apparatus connected to an external information processing apparatus via a communication network,
Input means for inputting the annotation image;
Whether or not the annotation image is projected with a step on the object and the background by the projection device based on the distance data measured by the measurement device and the annotation image input by the input unit Discriminating means for discriminating;
An information processing apparatus comprising: a warning unit that issues a warning when the annotation image is projected with a step between the object and the background. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background And an external information processing apparatus including an extraction unit that extracts an area occupied by the background in a captured image captured by the imaging apparatus based on distance data measured by the measurement apparatus via a communication network An information processing apparatus to be connected,
An information processing apparatus comprising: input means for inputting the annotation image; and setting means for setting an area occupied by the extracted background as an input prohibited area of the annotation image. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background An information processing apparatus connected to an external information processing apparatus via a communication network,
Based on the distance data measured by the measurement device, the extraction means for extracting the area occupied by the background in the photographed image photographed by the photographing device, the input means for inputting the annotation image, and the extraction An information processing apparatus comprising: setting means for setting an area occupied by the background as an input prohibited area of the annotation image. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background An information processing apparatus connected to an external information processing apparatus,
Based on the distance data measured by the measurement device, an extraction unit that extracts an area occupied by the background in the captured image captured by the imaging device, an input unit that inputs the annotation image, and the projection device Adjusting means for moving and / or reducing the annotation image so that the annotation image is projected onto an area without a step when the annotation image projection area onto which the annotation image is projected has a step. An information processing apparatus comprising: Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background A computer connected to the external information processing apparatus via a communication network,
Input means for inputting the annotation image;
Display means for displaying the input annotation image, the object and the background;
Based on the distance data measured by the measuring device, the determination means for determining whether or not the annotation image is included in the area occupied by the background displayed on the display means, and the annotation in the area occupied by the background A control program which functions as a warning means for giving a warning when an image is included. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background A computer connected to the external information processing apparatus via a communication network,
Input means for inputting the annotation image;
Whether or not the annotation image is projected with a step on the object and the background by the projection device based on the distance data measured by the measurement device and the annotation image input by the input unit And a control program that functions as warning means for giving a warning when the annotation device projects the annotation image with a step on the object and the background. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background And an external information processing apparatus including an extraction unit that extracts an area occupied by the background in a captured image captured by the imaging apparatus based on distance data measured by the measurement apparatus via a communication network Connect the computer
A control program that functions as an input unit that inputs the annotation image and a setting unit that sets an area occupied by the extracted background as an input prohibited region of the annotation image. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background A computer connected to the external information processing apparatus via a communication network,
Based on the distance data measured by the measurement device, the extraction means for extracting the area occupied by the background in the photographed image photographed by the photographing device, the input means for inputting the annotation image, and the extracted A control program that functions as setting means for setting an area occupied by a background as an input prohibited area of the annotation image. Connected to an imaging device for imaging an object, a projection device for projecting an annotation image onto a projection area including the object and the background, and a measurement device for measuring the distance from the imaging device to the object and the background A computer connected to the external information processing device
Based on the distance data measured by the measuring device, the extraction means for extracting the area occupied by the background in the photographed image photographed by the photographing device, the input means for inputting the annotation image, and the projection device When there is a step in the annotation image projection area onto which the annotation image is projected, the annotation image is moved and / or reduced to function as an adjustment unit that adjusts the annotation image to be projected onto a region without a step. A control program characterized by

Images