JP2009290488A - Information processing system and information processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information processing system and apparatus which projects an annotation image on a region including an object by a simple configuration either on the photographic device side or on the projector side. <P>SOLUTION: An input processing section 101 inputs the coordinates of an annotation image from a tablet PC 8, and a projection image generating section 105 generates the annotation image, based on the coordinates of the annotation image. A detection processing section 102 detects the position and posture of the tablet PC 8 from a plurality of images photographed by a camera 5, and a projection processing section 106 executes projection transformation on the annotation image generated by the projection image generating section 105 and outputs the projection-transformed annotation image to a projector 4. Then, the projector 4 projects the projection-transformed annotation image on the tablet PC 8. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an information processing system and an information processing apparatus.

2. Description of the Related Art Conventionally, a system that communicates between users using a camera and a projector is known (see, for example, Patent Document 1). This system adds a first point where a camera and a projector whose optical point is optically the same as the camera and an annotation (annotation) to an image taken by the camera at the first point. And a second point having a capable terminal. When an instruction to add an annotation is given at the terminal at the second point, the annotation is projected by the projector at the first point.
JP 2005-33756 A

  However, in the above-mentioned patent document 1, an annotation cannot be added unless the user is at the second point. Although the system of Patent Document 1 can communicate between users even when the first point and the second point are separated, the annotation can be added only at the second point.

  An object of the present invention is to allow an annotation image to be projected onto a region including an object with a simple configuration on the imaging device side or the projection device side, and to make an annotation image easy to see for the user on the imaging device side or the projection device side. An object is to provide an information processing system and an information processing apparatus that can be provided.

  In order to achieve the above object, the information processing system according to claim 1 includes first display means for displaying an annotation image, an input device for inputting the annotation image or an instruction, and imaging for capturing an area including the input device. An apparatus, a projection device that projects an annotation image onto the area, a detection unit that detects a position and orientation of the input device based on a captured image captured by the imaging device, and the annotation image or the instruction for receiving the instruction Generation means for generating a first image, output means for converting the first image into an image for the projection apparatus based on the position and orientation of the input apparatus detected by the detection means, and outputting the image to the projection apparatus A first input means for inputting coordinate information input to the input device; an annotation image or instruction content is determined based on the coordinate information and the first image projected on the input device; If an image is included, the output means converts the annotation image into an image for the projection device projected onto the area and outputs the image to the projection device. An information processing device having an execution unit that executes processing according to the content, and the projection device projects the annotation image or the image reflecting the instruction content on the detected input device, When the first display unit displays the annotation image, the projection device does not project the annotation image on the detected input device.

  2. The information processing system according to claim 2, further comprising a remote terminal connected to the information processing apparatus via a network, wherein the remote terminal displays a captured image from the imaging apparatus. A second display unit that inputs the annotation image or the instruction, and the annotation image input by the second input unit overlaps the input device displayed on the second display unit. Sometimes, the annotation image of the overlapped portion is transmitted to the input device, and the first display means displays the annotation image of the overlapped portion.

  The information processing system according to claim 3 is the information processing system according to claim 2, wherein a remaining annotation image obtained by removing the annotation image of the overlapping portion from the annotation image input by the second input unit is the projection device. It is projected from.

  The information processing system according to claim 4 is the information processing system according to claim 2, wherein the second display means displays a screen including a hot spot area for transferring data to the input device by drag and drop. And

  The information processing system according to claim 5 is the information processing system according to claim 2 or 3, wherein when an instruction of a sharing mode is further input from the input device and coordinates of the annotation image are input, The two display means displays the annotation image, and the projection device projects the annotation image to the position of the input device in the captured image when the annotation image is input.

  The information processing system according to claim 6 is the information processing system according to claim 2 or 3, further comprising a determination unit that determines whether or not the input device has not moved for a predetermined time, When it is determined that the input device has not moved for a predetermined time, the second display means displays the annotation image, and the projection device projects the annotation image at a position of the input device that has not moved. It is characterized by that.

  The information processing system according to claim 7 is the information processing system according to any one of claims 1 to 6, wherein the information processing apparatus includes a user detection unit that detects a position of the user, and the detection unit includes: A line-shaped mark attached to at least one place of the input device is detected from a photographed image photographed by the photographing device, and the execution means is based on detection information of a user position detected by the user detection means. Then, one side is selected from the four sides of the imaging region of the imaging device or the projection region of the projection device, and the output means applies the detected line-shaped mark and the selected one side. Based on this, a rotation process is performed on the annotation image, and the annotation image is output to the projection device.

  The information processing apparatus according to claim 8 includes first display means for displaying an annotation image, an input device for inputting the annotation image or an instruction, an imaging device for photographing an area including the input device, and an annotation for the area. An information processing apparatus connected to a projection apparatus for projecting an image, the detection means for detecting the position and orientation of the input apparatus based on a captured image captured by the imaging apparatus, the annotation image or the instruction is received The first image is converted into an image for the projection device based on the position and orientation of the input device detected by the detection unit and the generation unit for generating the first image for output to the projection device Based on the output means, the first input means for inputting the coordinate information input to the input device, the coordinate information and the first image projected on the input device, the annotation image or the instruction content is determined; When an image is included, the output means converts the annotation image into an image for the projection device projected onto the area and outputs the image to the projection device. Execution means for executing processing according to contents is provided, wherein the projection apparatus projects the annotation image or the image reflecting the instruction contents onto the detected input device, and the first display means When displaying the annotation image, the projection device does not project the annotation image onto the detected input device.

  According to the first and eighth aspects of the present invention, the annotation image can be projected onto the region including the object with a simple configuration on the photographing apparatus side or the projection apparatus side. Furthermore, since the annotation image is displayed according to the resolution of the first display means regardless of the resolution of the projection device, it is possible to provide an easy-to-view annotation image for the user on the photographing device side or the projection device side.

  According to the second aspect of the present invention, since the portion of the annotation image input by the remote terminal that overlaps the input device displayed on the second display means is displayed by the first display means, the projection device overlaps the image. Compared with the case where a partial annotation image is projected, it is possible to provide an easy-to-view annotation image for the user on the photographing apparatus side or the projection apparatus side. Even when the second display means is a liquid crystal display, it is possible to provide an easy-to-see annotation image for the user on the photographing apparatus side or the projection apparatus side.

  According to the invention of claim 3, the first display means and the projection device can provide the entire annotation image input at the remote terminal to the user on the photographing device side or the projection device side.

  According to the invention of claim 4, the user of the remote terminal can transfer desired data to the input device by a simple operation, and communicates with the user of the input device (that is, the user on the photographing device side or the projection device side). can do.

  According to the invention of claim 5, the annotation image can be simultaneously provided to a plurality of users. Further, like so-called post-it, the annotation image can be pasted on the position of the tablet PC 8 when the annotation image is input.

  According to the invention of claim 6, an annotation image can be provided to a plurality of users at the same time. Further, like so-called post-it, the annotation image can be pasted on the position of the tablet PC 8 when the annotation image is input.

  According to the seventh aspect of the present invention, the annotation image can be automatically rotated in a direction that is easy for the user to see.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention.

  The information processing system shown in FIG. 1 includes a local system and a remote system. The local system includes a personal computer (PC) 1 (information processing apparatus), a projector 4 (projection apparatus), a camera 5 (imaging apparatus), and a tablet PC 8 (input apparatus) functioning as a server. A projector 4, a camera 5, and a tablet PC 8 are connected to the PC 1. Further, the PC 1 may include a human sensor 10 (user detection means) that detects the position of the user on the projector 4 side or the camera 5 side. The remote system includes a PC 2 (remote terminal) that functions as a client, and a display device 21 is connected to the PC 2. The PC 1 and the PC 2 are connected to each other via the network 3. The local system may include a tablet with a display instead of the tablet PC 8.

  The projector 4 projects an annotation image (annotation image) or the like on the area of the table 7 on which the object 6 is placed based on a control command from the PC 1. An annotation image includes images of all forms such as lines, characters, symbols, figures, colors, fonts (typefaces), and the like. The annotation image is input by the PC 2 or the tablet PC 8.

  The camera 5 is composed of a video camera or a PTZ (Pan Tilt Zoom) camera. The camera 5 captures an area of the table 7 and outputs a captured image to the PC 1. This captured image is displayed on the display device 21. A part of the captured image may be projected onto the tablet PC 8 via the projector 4 when the tablet PC 8 is on the table 7.

  The tablet PC 8 has a quadrangular shape and a pressure-sensitive panel. The user inputs an instruction or annotation image by pressing or tracing the pressure-sensitive panel using a pen. The pressure-sensitive panel is fixed by being overlapped on the display panel. The tablet PC 8 outputs the input instruction or annotation image to the PC 1. When the user on the local side inputs an annotation image to the tablet PC 8, the annotation image is projected onto the area of the table 7 via the projector 4 or displayed on the display panel of the tablet PC 8. The tablet PC 8 is connected to the PC 1 by wireless communication such as infrared or bluetooth (registered trademark), but may be connected to the PC 1 by wired communication using a communication cable.

  The PC 1 outputs the captured image captured by the camera 5 to the PC 2 via the network 3. The PC 2 is connected to a display unit 21. The display unit 21 displays a captured image display area 23 and a user interface (UI) 22. The PC 2 may be configured as an integrated personal computer including the display device 21.

  The UI 22 is a hot spot area in which buttons such as pen, text, and erase, line type and color type icons, and data (files) such as text and images can be transferred to the tablet PC 8 by drag and drop. 25. In the display area 23, a photographed image photographed by the camera 5 is displayed. In the figure, a photographed image obtained by photographing the area of the table 7 including the tablet PC 8 with the camera 5 is displayed in the display area 23. For example, when a pen button of the UI 22 is pressed and a graphic or the like is written in the display area 23, information on the graphic is output from the PC 2 to the projector 4 via the PC 1. The projector 4 draws a graphic on the table 7 or the tablet PC 8 based on the graphic information.

  In the figure, one client PC 2 is shown, but a plurality of clients (PCs) may be provided. Further, the number of cameras that capture the area of the table 7 is not limited to one, and a plurality of cameras may be provided. Further, the number of tablets connected to the PC 1 may be one or plural.

  FIG. 2A is a block diagram illustrating a hardware configuration of the PC 1. FIG. 2B is a block diagram illustrating a hardware configuration of the PC 2.

  The PC 1 includes a CPU 11 that controls the entire apparatus, a ROM 12 that includes a control program, a RAM 13 that functions as a working area, a hard disk drive (HDD) 14 that includes various information and programs, a network interface 15 for connecting to other computers, and a projector. 4 and a USB (universal serial bus) interface 16 for connecting to USB devices such as the camera 5, and a communication unit 17 that performs wired or wireless communication with the tablet PC 8. The CPU 11 is connected to the ROM 12, RAM 13, hard disk drive (HDD) 14, network interface 15, USB interface 16, and communication unit 17 via the system bus 18.

  The PC 2 includes a CPU 31 that controls the entire apparatus, a ROM 32 that includes a control program, a RAM 33 that functions as a working area, a hard disk drive (HDD) 34 that includes various information and programs, a network interface 35 for connecting to other computers, and a keyboard. And a USB (universal serial bus) interface 36 for connection with an input device such as a mouse and a display device 21. The CPU 31 is connected to the ROM 32, RAM 33, hard disk drive (HDD) 34, network interface 35, USB interface 36, and display device 21 via the system bus 37.

  FIG. 3 is an external view of the tablet PC 8.

  As shown in FIG. 3, the tablet PC 8 includes a touch panel 81 (first display means). The touch panel 81 receives data and information input using the pen 82, and further displays various data and information.

  FIG. 4 is a block diagram showing the functional configuration of the PC1 and PC2.

  The PC 1 detects the movement of the tablet PC 8 by detecting coordinate information or instruction input from the tablet PC 8 (first input means), and detects the movement of the tablet PC 8 (detection means). A determination unit), a storage unit 103 that stores various data and information, a control unit 104 (execution unit) that controls the entire PC 1, a projection image generation unit 105 (generation unit) that generates a projection image, and a projection image A projection processing unit 106 (output means) that executes conversion processing of an image to be projected when the image is projected onto the tablet PC 8.

  The storage unit 103 includes a GUI database 103a that stores various GUIs (Graphical User Interfaces) projected on the tablet PC 8, a frame buffer area 103b that stores captured images captured by the camera 5, and a camera when executing the copy mode. 5 includes a snapshot area 103c for storing the still image (snapshot) photographed in step 5, and a position storage area 103d for storing position information of the tablet PC 8 (that is, coordinate information of the tablet PC 8) when the copy mode is executed. .

  The control unit 104 and the projection image generation unit 105 correspond to the CPU 21 that executes various processes according to the control program. The input processing unit 101 corresponds to the CPU 21 and the communication unit 17, the detection processing unit 102 and the projection processing unit 106 correspond to the CPU 21 and the USB interface 16, and the storage unit 103 corresponds to the hard disk drive (HDD) 14.

  The PC 2 includes a control unit 201 that controls the entire PC 1, a storage unit 202 that stores various data and information, an input unit 203 (second input unit) that inputs data and instructions using a keyboard and a mouse, and a captured image. Display unit 204 (second display means). The control unit 201 corresponds to the CPU 31 that executes various processes according to the control program, and the storage unit 202 corresponds to the hard disk drive (HDD) 34. The input unit 203 corresponds to a keyboard and mouse (not shown) and the USB interface 36 to which these are connected, and the display unit 204 corresponds to the display device 21.

  Hereinafter, processing executed by each functional block will be specifically described.

  First, the camera 5 continuously shoots the area of the table 7 including the tablet PC 8 (for example, every 0.5 seconds). The captured image is transmitted from the camera 5 to the detection processing unit 102.

  The detection processing unit 102 stores the captured image from the camera 5 in the storage unit 103 (frame buffer area 103b), and recognizes the tablet PC 8 from the sequence of the captured image by detecting the shape of the tablet PC 8 and template matching.

  Specifically, the detection processing unit 102 performs noise removal and smoothing on an image sequence photographed as preprocessing, and detects an edge using a known Canny method. Next, the detection processing unit 102 detects a plurality of straight lines from the detected edges using a known Hough transform. At this time, the detection processing unit 102 removes line segments that are not related to the tablet PC 8 in consideration of the size and parallelism of the tablet PC 8. Thereafter, the detection processing unit 102 detects a parallelogram from the detected intersections of a plurality of straight lines, and sets the detected parallelogram as a candidate for the tablet PC 8. Finally, the detection processing unit 102 compares the pre-captured image of the tablet PC 8 with the candidates for the tablet PC 8, and compares the characteristics of the tablet PC 8 (for example, the shape of the frame or the buttons attached to the tablet PC 8). Let the candidate to have be tablet PC8.

  The detection processing unit 102 calculates coordinates that specify the position and orientation of the recognized tablet PC 8 (here, the coordinates of the four corners of the tablet PC 8), and outputs the calculated coordinates to the control unit 104, the projection image generation unit 105, and the projection processing unit 106. To do. In addition, the detection processing unit 102 outputs a captured image from the camera 5 to the control unit 104.

  The projection image generation unit 105 generates an image to be projected on the tablet PC 8. That is, the projection image generation unit 105 cuts out a partial image projected on the tablet PC 8 from the captured image based on the coordinates specifying the position and orientation of the tablet PC 8 and outputs the partial image to the projection processing unit 106. The partial image cut out by the projection image generation unit 105 is a captured image of a portion where the tablet PC 8 overlaps the region of the table 7 including the object 6 when the region of the table 7 is viewed from the projector 4 side. Therefore, when the tablet PC 8 overlaps an arbitrary area of the table 7, the projection image generation unit 105 cuts out the partial image from the captured image that is captured immediately before the tablet PC 8 overlaps the arbitrary area.

  The projection processing unit 106 estimates projection conversion parameters based on the coordinates of the tablet PC 8 calculated by the detection processing unit 102 and the coordinates of the four corners of the projection partial image generated by the projection image generation unit 105. Then, partial image conversion (hereinafter referred to as projection conversion) is performed using the estimated parameters.

  The projection processing unit 106 outputs the projection-converted image to the projector 4, and the projector 4 projects the projection-converted image onto the tablet PC 8 as a part of all projection contents. At this time, the projector 4 may perform normal projection on an area other than the tablet PC 8. That is, the projector 4 can project images with different contents simultaneously on the tablet PC 8 and an area other than the tablet PC 8. FIG. 5 shows a state in which the projection-converted image is projected on the tablet PC 8.

  The tablet PC 8 displays the annotation image input with the pen 82 on the touch panel 81 and outputs the coordinates of the annotation image input with the pen 82 to the input processing unit 101. The input processing unit 101 performs the inverse transformation of the projection transformation on the coordinates of the annotation image, calculates the relative coordinates on the projection image, and outputs the relative coordinates to the control unit 104.

  The control unit 104 outputs the relative coordinates to the projection image generation unit 105, and the projection image generation unit 105 generates a projection annotation image on the image to be projected on the tablet PC 8 and outputs it to the projection processing unit 106. FIG. 6 shows a state in which the backprojection-transformed annotation image is formed on the projection image. As described above, the projection processing unit 106 performs projection conversion using the estimated parameters, outputs the projection-converted image to the projector 4, and the projector 4 outputs the projection-converted image to the tablet PC 8. Project.

  As described above, the annotation image input with the pen 82 of the tablet PC 8 is displayed on the touch panel 81 and can be further projected from the projector 4 to the tablet PC 8. When an annotation image is input with the pen 82 of the tablet PC 8, only the touch panel 81 displays the annotation image, and the PC 1 can be set not to project the annotation image from the projector 4 to the tablet PC 8.

  In addition, the control unit 104 inputs a captured image from the camera 5 and coordinates specifying the position and orientation of the tablet PC 8 from the detection processing unit 102, and outputs them to the PC 2 via the network 3. Thereby, as shown in FIG. 1, the state of the table 7 on which the local object 6 and the tablet PC 8 are placed is reproduced in the display area 23.

  FIG. 7A is a diagram showing an example in which an annotation image drawn in the display area 23 is displayed on the tablet PC 8, and FIG. 7B shows an annotation image file in the hot spot 25 or the display area 23. It is a figure which shows the example by which the said annotation image is displayed on tablet PC8 when it is dragged and dropped on tablet PC8 currently displayed.

  FIG. 8 shows that when an annotation image drawn in the display area 21 protrudes from the tablet PC 8 displayed in the display area 23, a part of the annotation image is displayed on the tablet PC 8 on the local side, and the annotation image is displayed. It is a figure which shows the example by which the remainder of is projected.

  As shown in FIG. 7A, when the remote user draws the annotation image 40 on the tablet PC 8 displayed in the display area 23, the annotation image 40 is displayed on the local tablet PC 8. The Further, as shown in FIG. 7B, when the user on the remote side drags and drops the file of the annotation image 40 onto the tablet PC 8 displayed in the hot spot 25 or the display area 23, the user on the local side The annotation image 40 is displayed on the tablet PC 8. In this case, the drag-and-drop file is not limited to an annotation image file, and may be a text, still image, moving image, or audio file.

  In addition, as shown in FIG. 8, when the text “ABC...” And the arrow image are drawn as the annotation image in the display area 23 and the arrow image protrudes from the tablet PC 8 displayed in the display area 23. On the local side, the text “ABC...” And an image of a part of the arrow are displayed on the tablet PC 8 as the annotation image 51, and the remaining image of the arrow is projected from the projector 4 to the table 7 as the annotation image 52. Although the annotation image 52 in FIG. 8 is displayed with a broken line, the broken line indicates that it is projected.

  FIG. 9 is a flowchart showing basic processing executed by PC1 and PC2.

  First, the detection processing unit 102 stores the captured image from the camera 5 in the storage unit 103, and coordinates (in this case, the tablet PC8) specifying the position and orientation of the tablet PC8 by shape detection and template matching of the tablet PC8 from the sequence of the captured image. The coordinates of the four corners of the PC 8 are calculated and output to the control unit 104, the projection image generation unit 105, and the projection processing unit 106. Further, the detection processing unit 102 outputs a captured image from the camera 5 to the control unit 104 (step S1). The control unit 104 outputs the coordinates specifying the position and orientation of the tablet PC 8 and the captured image from the camera 5 to the PC 2 via the network 3 (step S2).

  The control unit 201 of the PC 2 receives the coordinates specifying the position and orientation of the tablet PC 8 from the PC 1 and the captured image from the camera 5, stores the coordinates specifying the position and orientation of the tablet PC 8 in the storage unit 202, and The captured image is displayed on the display unit 204 (step S3).

  Next, the control unit 201 determines whether there is an editing operation from the input unit 203 (step S4). Here, the editing operation is an operation for writing an annotation image or an operation for dragging and dropping a file onto the hot spot 25 or the tablet PC 8 displayed in the display area 23.

  If NO in step S4, the process ends. If YES in step S4, and if a drag and drop operation has been performed, the process proceeds to step S6. In the case of YES in step S4, when the annotation image writing operation is performed, the control unit 201 displays the written annotation image in the display area 23 based on the coordinates specifying the position and orientation of the tablet PC 8. It is determined whether or not the tablet PC 8 displayed on the screen does not protrude (step S5).

  If YES in step S5, the control unit 201 transmits the coordinates of the written annotation image or the file on which the drag and drop operation has been performed to the control unit 104 of the PC 1 (step S6).

  The control unit 104 outputs the annotation image coordinates from the control unit 201 or the file on which the drag and drop operation has been performed to the tablet PC 8 via the input processing unit 101, and the annotation image is displayed on the touch panel 81 of the tablet PC 8. The file that is displayed or dragged and dropped on the tablet PC 8 is played back (step S7). Here, the coordinates of the annotation image from the control unit 201 are converted into coordinates for the tablet PC 8 by the input processing unit 101 and output to the tablet PC 8.

  In the case of NO in step S5, the control unit 201 protrudes from the tablet PC 8 where the written annotation image is displayed in the display area 23 based on the coordinates specifying the position and orientation of the tablet PC 8. Divide into parts that do not protrude (step S8). Next, the control unit 201 transmits the coordinates of the annotation image that does not protrude from the tablet PC 8 displayed in the display area 23 to the control unit 104 of the PC 1, that is, executes the same processing as step S6. Thereafter, the process of step S7 is executed for the coordinates of the annotation image that does not protrude from the tablet PC 8 displayed in the display area 23.

  The control unit 201 adds a flag to the coordinates of the annotation image protruding from the tablet PC 8 displayed in the display area 23 and transmits the flag to the control unit 104 of the PC 1 (step S9). The control unit 104 outputs the coordinates of the annotation image with the flag from the control unit 201 to the projection image generation unit 105 (step S10). As described above, the flag is used by the control unit 104 to determine the output destination of the coordinates of the annotation image (that is, the input processing unit 101 or the shadow image generation unit 105).

  The projection image generation unit 105 generates an annotation image based on the coordinates of the annotation image and outputs the annotation image to the projection processing unit 106 (step S11).

  The projection processing unit 106 performs projection conversion of the annotation image from the projection image generation unit 105, outputs the projection-converted annotation image to the projector 4 (step S12), and ends this processing. The projector 4 projects the projection-transformed annotation image onto the area of the table 7 including the object 6.

  As described above, in the process of FIG. 9, the annotation image written on the tablet PC 8 displayed in the display area 23 is displayed on the touch panel 81 of the tablet PC 8 and displayed on the hot spot 25 or the display area 23. The file dragged and dropped onto the tablet PC 8 being played is played back on the tablet PC 8. Further, the annotation image protruding from the tablet PC 8 displayed in the display area 23 is projected onto the area of the table 7 via the projector 4.

  FIG. 10 is a flowchart illustrating a process of projecting a GUI on the tablet PC 8 and a process of executing a mode designated on the tablet PC 8. This process is executed by the PC 1.

  First, the detection processing unit 102 recognizes the tablet PC 8 from the image sequence photographed by the camera 5 by shape detection and template matching of the tablet PC 8, calculates coordinates for specifying the position and orientation of the recognized tablet PC 8, and the tablet The coordinates of the PC 8 are output to the projection image generation unit 105 and the projection processing unit 106 (step S21).

  Next, the projection image generation unit 105 generates a GUI image to be projected on the tablet PC 8 based on various GUI elements of the GUI database 103a (step S22).

  The projection processing unit 106 estimates projection conversion parameters based on the coordinates of the tablet PC 8 calculated by the detection processing unit 102 and the coordinates of the four corners of the projection GUI image generated by the projection image generation unit 105. Then, using this estimated parameter, the GUI image is projected and converted, and the projected and converted GUI image is output to the projector 4 (step S23). The projector 4 projects the projection-converted GUI image onto the tablet PC 8 as a part of the entire projection content (step S24). An example of a GUI image projected onto the tablet PC 8 is shown in FIG.

  The tablet PC 8 outputs the coordinates input with the pen to the input processing unit 101 (step S25). The input processing unit 101 performs inverse transformation of the above-described projection transformation on the coordinates input with the pen, calculates the relative coordinates on the GUI image acquired from the projection image generation unit 105, and determines the mode input with the pen (Step S26).

  The control unit 104 inputs the mode information determined from the input processing unit 101, executes a process according to the mode information (step S27), and ends the present process.

  For example, when the input mode is the print mode, the control unit 104 outputs a print start instruction to a printer driver (not shown). When the input mode is the copy mode, the control unit 104 outputs an instruction to start processing in FIGS. 12 and 13 described later to the detection processing unit 102, the projection image generation unit 105, and the projection processing unit 106. . When the input mode is the sharing mode, the control unit 104 outputs an instruction to start the processing of FIG. 15 described later to the detection processing unit 102, the projection image generation unit 105, and the projection processing unit 106.

  According to the processing in FIG. 10, the user selects any one of a plurality of modes defined by the GUI image projected on the tablet PC 8 with the tablet PC 8, and performs processing corresponding to the mode on the control unit 104. Can be executed.

  In the process of FIG. 10, the GUI image is projected on the tablet PC 8 from the projector 4, but the GUI image may be displayed on the touch panel 81 of the tablet PC 8. In this case, steps S25 to S27 are executed in the above process.

  12 and 13 are flowcharts showing the copy mode and non-copy mode processing executed by the PC 1.

  In this process, it is assumed that the camera 5 is continuously shooting the area of the table 7 including the tablet PC 8 (for example, shooting every 0.5 seconds).

  First, the control unit 104 determines whether a copy mode instruction is input from the tablet PC 8 via the input processing unit 101 (step S31). Here, the copy mode refers to a region of the table 7 viewed from the projector 4 side, and when the tablet PC 8 overlaps with an arbitrary region of the table 7, a pre-photographed image corresponding to the arbitrary region is displayed on the tablet. In this mode, the screen is fixedly projected on the tablet PC 8 regardless of the movement of the PC 8. When this mode is executed, a partial image of the real object (that is, the table 7 including the object 6) in front of the user's eyes is always projected on the tablet PC 8 in a fixed manner. Therefore, the user can project the image of the place to be photographed on the tablet PC 8 by superimposing the tablet PC 8 on the place to be photographed in the area of the table 7 and inputting a copy mode instruction.

  When the user inputs a copy mode instruction on the tablet PC 8 in step S11, a button (not shown) provided on the tablet PC 8 is pressed or the position of the copy mode in the GUI projected from the projector 4 onto the tablet PC 8 is set. It is necessary to tap.

  If the copy mode instruction is not input in step S11 (NO), the detection processing unit 102 stores the captured image from the camera 5 in the frame buffer area 103b and specifies the position and orientation of the tablet PC 8. Coordinates (here, the coordinates of the four corners of the tablet PC 8) are calculated and output to the projection image generation unit 105 and the projection processing unit 106 (step S32).

  The projection image generation unit 105 reads the captured image from the frame buffer area 103b, cuts out the partial image projected on the tablet PC 8 from the captured image based on the coordinates specifying the position and orientation of the tablet PC 8, and extracts the partial image. Is output to the projection processing unit 106 (step S33). The partial image cut out by the projection image generation unit 105 is a captured image of a portion where the tablet PC 8 overlaps the region of the table 7 including the object 6 when the region of the table 7 is viewed from the projector 4 side. Therefore, when the tablet PC 8 overlaps an arbitrary area of the table 7, the projection image generation unit 105 cuts out the partial image from the captured image that is captured immediately before the tablet PC 8 overlaps the arbitrary area. FIG. 14A is a diagram showing a positional relationship between the tablet PC 8 and the table 7 including the object 6 when the table 7 is viewed from the projector 4 side, and FIG. It is a figure which shows the sequence of the partial image projected on tablet PC8 to pass.

  The projection processing unit 106 estimates projection conversion parameters based on the coordinates of the tablet PC 8 calculated by the detection processing unit 102 and the coordinates of the four corners of the projection partial image generated by the projection image generation unit 105. Then, projection conversion of the projection partial image is performed using the estimated parameters, and the projection-converted partial image is output to the projector 4 (step S34). The projector 4 projects the projection-converted partial image onto the tablet PC 8 as a part of the entire projection content (step S35), and returns to step S32.

  Through the loop processing of steps S32 to S35, the position and orientation of the tablet PC 8 are detected, and a partial image corresponding to the position and orientation of the tablet PC 8 is projected onto the tablet PC 8.

  When the copy mode instruction is input in step S11 (YES), the control unit 104 stores the snapshot of the captured image (that is, the image when the shutter of the camera 5 is released) and the snapshot. An instruction to store coordinates specifying the position and orientation of the tablet PC 8 at the moment of shooting is output to the detection processing unit 102 (step S36).

  The detection processing unit 102 stores a snapshot of the captured image in the snapshot area 103c and stores coordinates that specify the position and orientation of the tablet PC 8 at the moment of capturing the snapshot in the position storage area 103d (step S37). .

  Next, the control unit 104 outputs a copy mode instruction and a projection image generation instruction to the projection image generation unit 105 (step S38).

  The projection image generation unit 105 cuts out and extracts a partial image for projection from the snapshot saved in the snapshot area 103c based on the coordinates of the tablet PC 8 at the moment of taking the snapshot saved in the position saving area 103d. The partial image is stored in the snapshot area 103c and is output to the projection processing unit 106 (step S39).

  The projection processing unit 106 estimates projection conversion parameters based on the coordinates of the tablet PC 8 at the moment of taking the snapshot and the coordinates of the four corners of the projection partial image generated by the projection image generation unit 105. Then, projection conversion of the partial image for projection is performed using the estimated parameters, and the projected partial image is output to the projector 4 (step S40). In the projection conversion, enlargement / reduction of a partial image or the like may be executed as appropriate. The projector 4 projects the projection-converted partial image onto the tablet PC 8 as a part of the entire projection content (step S41).

  Thereafter, the detection processing unit 102 calculates the coordinates of the tablet PC 8 from the captured image of the camera 5 that has captured the image projected by the projector 4, and outputs the coordinates to the projection image generation unit 105 and the projection processing unit 106 (step S42).

  The projection image generation unit 105 determines whether there is an input of the coordinates of the annotation image from the tablet PC 8 via the input processing unit 101 and the control unit 104 (step S43).

  If the coordinates of the annotation image are not input in step S43 (NO), the process proceeds to step 39, and the projection image generation unit 105 outputs the partial image for projection stored in the snapshot area 103c to the projection processing unit 106. To do. In the subsequent processing in step S20, the projection processing unit 106 uses the coordinates of the tablet PC 8 output from the detection processing unit 102 in step S42 instead of the coordinates of the tablet PC 8 at the moment of taking the snapshot.

  In the above-described step S31, YES → step S43, NO → step S39, the movement of the tablet PC 8 is detected, and the projection partial image cut out from the snapshot is obtained regardless of the movement position of the tablet PC 8. Projected onto the tablet PC 8.

  If the coordinates of the annotation image are input in step S43 (YES), the projection image generation unit 105 generates an annotation image based on the coordinates of the annotation image, outputs the annotation image to the projection processing unit 106, and generates the annotation image. The annotation image is added to the partial image for projection stored in the snapshot area 103c, and is output to the projection processing unit 106 (step S44).

  The projection processing unit 106 estimates projection conversion parameters based on the coordinates of the tablet PC 8 output from the detection processing unit 102 in step S22 and the coordinates of the four corners of the projection partial image with the annotation image. Using this estimated parameter, the projection partial image with the annotation image is subjected to projection conversion, and the projection-converted partial image and annotation image are output to the projector 4 (step S45), and the process proceeds to step S41. . In the subsequent process of step S <b> 21, the projector 4 projects the projection-converted partial image onto the tablet PC 8 and projects the annotation image onto the area of the table 7 including the target object 6.

  In the above step S31, YES → step S43 YES → step S41, the movement of the tablet PC 8 is detected, and the projection partial image to which the annotation image is added is detected regardless of the movement position of the tablet PC 8. The image is projected onto the tablet PC 8 and the annotation image is projected onto the real object (that is, the area of the table 7 including the object 6) alone.

  Next, shared mode processing executed by the PC 1 will be described.

  The sharing mode is a mode in which a plurality of users share one tablet PC 8 and project the same image as the image displayed on the touch panel of the tablet PC 8 onto an arbitrary area of the table 7. At this time, since the tablet PC 8 has a display function, it is not necessary for the projector 4 to project an image on the tablet PC 8, but an image may be projected. When an annotation image is input to the tablet PC 8, the annotation image is also projected on an arbitrary area of the table 7.

  FIG. 15 is a flowchart showing the processing in the sharing mode executed by the PC 1.

  First, the control unit 104 determines whether or not a share mode instruction is input from the tablet PC 8 via the input processing unit 101 (step S51).

  When the user inputs a sharing mode instruction on the tablet PC 8 in step S31, a button (not shown) provided on the tablet PC 8 is pressed, or the position of the sharing mode in the GUI projected on the tablet PC 8 from the projector 4 is set. It is necessary to tap.

  If the sharing mode instruction is not input in step S51 (NO), the process is terminated. On the other hand, when the sharing mode instruction is input in step S51 (YES), the control unit 104 specifies coordinates for specifying the position of the tablet PC 8 in the captured image (hereinafter, the coordinates of the tablet PC 8 are referred to as shared coordinates). Is output to the detection processing unit 102 (step S52).

  The detection processing unit 102 calculates shared coordinates from the captured image, and outputs them to the projection image generation unit 105 and the projection processing unit 106 (step S53). The detection processing unit 102 calculates the coordinates (shared coordinates) of the tablet PC 8 each time a new captured image is input from the camera 5, and outputs the calculated coordinates to the projection image generation unit 105 and the projection processing unit 106.

  The projection image generation unit 105 determines whether there is an input of the coordinates of the annotation image via the control unit 104 (step S54). The coordinates of the annotation image are not limited to being input from the tablet PC 8 and may be input from the PC 2.

  If there is no input of the coordinates of the annotation image in step S54 (NO), this determination is repeated.

  If there is an input of an annotation image coordinate in step S54 (YES), the projection image generation unit 105 generates an annotation image based on the annotation image coordinate and outputs it to the projection processing unit 106 (step S55). .

  On the other hand, if the coordinates of the annotation image are input from the PC 2, the control unit 104 outputs the coordinates of the annotation image to the tablet PC 8 via the input processing unit 101 and causes the tablet PC 8 to display the annotation image (step S56). When the annotation image is directly input to the tablet PC 8, the tablet PC 8 displays the annotation image, and thus the process of step S56 is omitted.

  The projection processing unit 106 performs projection conversion of the annotation image and outputs the projection-converted annotation image to the projector 4 (step S57).

  The projector 4 projects the annotation image that has been projected and converted to the position of the tablet PC 8 when the annotation image is input (step S58).

  Thereafter, the detection processing unit 102 calculates the coordinates (shared coordinates) of the tablet PC 8 from the captured image of the camera 5 that has captured the image projected by the projector 4 and outputs the calculated coordinate to the projection image generation unit 105 and the projection processing unit 106 ( Step S59) and return to Step S54.

  15, the annotation image is displayed on the tablet PC 8 detected by the detection processing unit 102, and the annotation image is projected on the position of the tablet PC 8 in the captured image when the sharing mode instruction is input. Therefore, an annotation image can be provided to a plurality of users at the same time. Further, like so-called post-it, the annotation image can be pasted on the position of the tablet PC 8 when the annotation image is input.

  Further, when the content of the annotation image is updated by the loop of steps S54 to S59, the annotation image displayed on the tablet PC 8 and the projected annotation image are also updated sequentially.

  In the sharing mode, the annotation image that has been subjected to projection conversion is projected on coordinates (shared coordinates) that specify the position of the tablet PC 8 in the captured image when the sharing mode instruction is input from the tablet PC 8. The unit 102 determines whether or not the tablet PC 8 has not moved for a predetermined time (for example, 5 minutes). If the tablet PC 8 has not moved for a predetermined time (for example, 5 minutes), the tablet PC 8 has not moved. You may make it project the annotation image by which projection conversion was carried out to the position of tablet PC8. In this case, when the tablet PC 8 moves, the moving tablet PC 8 displays the annotation image, and the projected annotation image is projected at the position of the tablet PC 8 that has not moved. Can be provided to any user.

  In the sharing mode, an example in which an annotation image is projected onto the position of the tablet PC 8 when a snapshot is taken has been described. However, as shown in FIG. 16A, the local user is not facing the front of the tablet PC 8. There is a case. In this case, if the annotation image includes characters, there is a problem that it is difficult for the user to read. Therefore, a process in which the PC 1 automatically rotates the annotation image in a direction that is easy for the user to see will be described.

  16A is a diagram showing the relationship between the area of the desk 7, the imaging area of the camera 5, the projection area of the projector 4, and the position of the tablet PC 8, and FIG. 16B is the imaging area of the camera 5. FIG. 16C is a diagram illustrating the relationship between the projection area of the projector 4 after the annotation image rotation process and the position of the tablet PC 8.

  It is assumed that the human sensor 10 for detecting the position of the user on the local side is connected to the control unit 104 of the PC 1. Further, it is assumed that the tablet PC 8 is provided with a line-shaped mark indicating the bottom side.

  FIG. 17 is a flowchart showing the annotation image rotation processing executed by the PC 1.

  First, the detection processing unit 102 recognizes the tablet PC 8 from the image sequence photographed by the camera 5 by shape detection and template matching of the tablet PC 8, calculates coordinates for specifying the position and orientation of the recognized tablet PC 8, and the tablet The coordinates of the PC 8 are output to the control unit 104, the projection image generation unit 105, and the projection processing unit 106 (step S71). In addition, the detection processing unit 102 detects a mark attached to the bottom side of the tablet PC 8 from the image sequence photographed by the camera 5, and outputs the mark to the projection processing unit 106 (step S72).

  Next, the control unit 104 selects one of the four sides of the imaging region or the projection region as a reference based on the detection information of the user position from a human sensor (not shown), and the reference side Is sent to the projection processing unit 106 (step S73). For example, in FIGS. 16A to 16C, the control unit 104 uses the right side of the imaging region or the projection region as a reference side.

  The control unit 104 outputs the coordinates of the annotation image displayed or projected on the tablet PC 8 or the coordinates of the annotation image displayed on the tablet PC 8 received from the PC 2 to the projection image generation unit 105 (step S74).

  The projection image generation unit 105 generates a projection annotation image based on the coordinates of the annotation image (step S75).

  The projection processing unit 106 sets parameters for projection conversion based on the coordinates of the tablet PC 8 calculated by the detection processing unit 102 and the coordinates of the four corners of the rectangular area that is inscribed in the projection annotation image. Then, projection conversion is performed on the annotation image for projection using the estimated parameters (step S76). Further, the projection processing unit 106 executes a rotation process on the annotation image that has been subjected to the projection conversion based on the mark attached to the bottom side of the PC 8 and the reference side information, and the annotation image after the rotation process is displayed on the projector. 4 (step S77). More specifically, the projection processing unit 106 has the shortest distance between the mark attached to the bottom side of the PC 8 and the reference side, and the mark attached to the bottom side of the PC 8 and the reference side are parallel to each other. The annotation image that has undergone the projection conversion is rotated so that

  The projector 4 projects the annotation image that has been projected and converted to the position of the tablet PC 8 when the annotation image is input (step S78), and ends the present process.

  According to the above rotation processing, one of the four sides of the imaging region or the projection region is selected based on the position of the user, and based on the line-shaped mark attached to this side and the bottom side of the PC 8, Since the rotation processing for the annotation image is executed, the annotation image can be automatically rotated in a direction that is easy for the user to see. In the above embodiment, a mark is attached to the bottom side of the tablet PC 8, but this mark may be attached to the upper part or the left and right sides of the tablet PC 8.

  Even when a recording medium in which a software program for realizing the functions of PC1 and PC2 is recorded is supplied to PC1 and PC2, and the CPU of PC1 and PC2 reads and executes the program stored in the storage medium, The same effects as in the above embodiment are achieved. Examples of the storage medium for supplying the program include a CD-ROM, a DVD, or an SD card.

  In addition, the same effect as that of the above-described embodiment can be obtained when the CPU of the PC 1 executes a software program for realizing the functions of the PC 1.

  Note that the present invention is not limited to the above-described embodiment, and can be implemented with various modifications without departing from the scope of the invention.

It is a block diagram which shows the structure of the information processing system which concerns on embodiment of this invention. (A) is a block diagram showing a hardware configuration of the PC 1, and (B) is a block diagram showing a hardware configuration of the PC 2. It is an external view of tablet PC8. It is a block diagram which shows the function structure of PC1 and PC2. It is a figure which shows the state by which the image by which projection conversion was carried out is projected on tablet PC8. It is a figure which shows the state in which the annotation image by which back projection conversion was carried out was formed on the image for projection. (A) is a figure which shows the example in which the annotation image drawn on the display area 23 is displayed on tablet PC8, (B) is the file of the annotation image displayed on the hot spot 25 or the display area 23. It is a figure which shows the example by which the said annotation image is displayed on tablet PC8 when it is dragged and dropped on tablet PC8. FIG. 8 shows that when an annotation image drawn in the display area 21 protrudes from the tablet PC 8 displayed in the display area 23, a part of the annotation image is displayed on the tablet PC 8 on the local side, and the annotation image is displayed. It is a figure which shows the example by which the remainder of is projected. It is a flowchart which shows the basic process performed by PC1 and PC2. It is a flowchart which shows the process which projects the GUI on the tablet PC8, and the mode designated on the tablet PC8. It is a figure which shows an example of the GUI image projected on tablet PC8. It is a flowchart which shows the process of the copy mode and non-copy mode which are performed with PC1. It is a flowchart which shows the process of the copy mode and non-copy mode which are performed with PC1. (A) is a figure which shows the positional relationship of tablet PC8 at the time of seeing the table 7 from the projector 4 side, and the table 7 containing the target object 6, (B) is tablet PC8 which passes on the target object 6 It is a figure which shows the sequence of the partial image projected on. It is a flowchart which shows the process of the sharing mode performed with PC1. (A) is a figure which shows the relationship between the area | region of the desk 7, the imaging | photography area | region of the camera 5, the projection area | region of the projector 4, and the position of tablet PC8, (B) is the imaging | photography area | region of the camera 5 and tablet PC8. It is a figure which shows the relationship with a position, (C) is a figure which shows the relationship between the projection area | region of the projector 4 after the rotation process of an annotation image, and the position of tablet PC8. It is a flowchart which shows the rotation process of the annotation image performed by PC1.

Explanation of symbols

1, 2 PC
3 Network 4 Projector 5 Camera 7 Desk 8 Tablet PC
DESCRIPTION OF SYMBOLS 10 Human sensor 101 Input processing part 102 Detection processing part 103 Storage part 104 Control part 105 Projection image generation part 106 Projection processing part 201 Control part 202 Storage part 203 Input part 204 Display part

Claims (8)

An input device that includes first display means for displaying an annotation image and that inputs the annotation image or instruction;
An imaging device for imaging an area including the input device;
A projection device for projecting an annotation image onto the region;
Detecting means for detecting the position and orientation of the input device based on a photographed image photographed by the photographing device;
Generating means for generating a first image for receiving the annotation image or the instruction;
Output means for converting the first image into an image for the projection device based on the position and orientation of the input device detected by the detection means, and outputting the image to the projection device;
First input means for inputting coordinate information input to the input device;
An annotation image or instruction content is determined based on the coordinate information and the first image projected on the input device, and when the annotation image is included, the annotation image is projected onto the area on the output means. An information processing apparatus having an execution unit that converts the image into an image for the projection apparatus to be output to the projection apparatus and executes processing according to the instruction content when the instruction content is included.
The projection device projects the annotation image or the image reflecting the instruction content onto the detected input device,
When the first display unit displays the annotation image, the projection device does not project the annotation image on the detected input device. A remote terminal connected to the information processing apparatus via a network;
The remote terminal includes second display means for displaying a photographed image from the photographing apparatus, and second input means for inputting the annotation image or the instruction,
When the annotation image input by the second input unit overlaps the input device displayed on the second display unit, the annotation image of the overlapped portion is transmitted to the input device, and the first display unit overlaps the input image. 2. The information processing system according to claim 1, wherein an annotation image of the selected part is displayed.   The information processing system according to claim 2, wherein the remaining annotation image obtained by removing the overlapping portion of the annotation image from the annotation image input by the second input unit is projected from the projection device.   The information processing system according to claim 2, wherein the second display unit displays a screen including a hot spot area for transferring data to the input device by drag and drop.   Further, when an instruction for sharing mode is input from the input device and coordinates of the annotation image are input, the second display means displays the annotation image, and the projection device receives the annotation image. The information processing system according to claim 2, wherein the annotation image is projected to a position of the input device in the captured image at that time. The information processing apparatus further includes a determination unit that determines whether or not the input device has not moved for a predetermined time,
When it is determined that the input device has not moved for a predetermined time, the second display means displays the annotation image, and the projection device projects the annotation image at a position of the input device that has not moved. The information processing system according to claim 2 or 3, The information processing apparatus includes user detection means for detecting a position of a user,
The detection means detects a line-shaped mark attached to at least one place of the input device from a photographed image photographed by the photographing device,
The execution unit selects one side from four sides of the imaging region of the imaging device or the projection region of the projection device based on the detection information of the user position detected by the user detection unit,
The output means executes a rotation process on the annotation image based on the detected line-shaped mark and the selected one side, and outputs the rotation to the projection device. The information processing system as described in any one of thru | or 6. First input means for displaying an annotation image is connected to an input device for inputting the annotation image or instruction, a photographing device for photographing an area including the input device, and a projection device for projecting the annotation image onto the area. An information processing apparatus comprising:
Detecting means for detecting the position and orientation of the input device based on a photographed image photographed by the photographing device;
Generating means for generating a first image for receiving the annotation image or the instruction;
Output means for converting the first image into an image for the projection device based on the position and orientation of the input device detected by the detection means, and outputting the image to the projection device;
First input means for inputting coordinate information input to the input device;
An annotation image or instruction content is determined based on the coordinate information and the first image projected on the input device, and when the annotation image is included, the annotation image is projected onto the region by the output means. An execution unit that converts the image into an image for the projection device to be output to the projection device and, when the instruction content is included, executes processing according to the instruction content;
The projection device projects the annotation image or the image reflecting the instruction content onto the detected input device,
When the first display unit displays the annotation image, the projection device does not project the annotation image on the detected input device.

Images