JP2009188685A - Controller and control program, instruction device and instruction program, and projection system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an instruction device and instruction program, capable of instructing the projection of a specific image that specifies the portion to be annotated, a controller and a control program capable of controlling to project the specific image, in accordance with an instruction, and to provide a projection system capable of projecting the specific image, in accordance with the instruction. <P>SOLUTION: The instruction device is provided with: a receiving means for receiving a picked-up image obtained by imaging a document; a control means for controlling a display device to display a portion to be annotated of the document, on the basis of an electronic file representing the document imaged in the picked-up image received by the receiving means; and a transmitting means for transmitting a projection instruction instructing the projection onto the document of a specific image, specifying the portion of the document controlled to be displayed by the control means and an annotated image for annotating the portion of the document. Thus, it becomes possible to instruct the projection, not only of the annotated image for annotating the portion of the document picked up in the received image, but also of the image specifying the portion of the displayed document to be annotated by the annotated image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a control device, a control program, an instruction device, an instruction program, and a projection system.

  Conventionally, not only can an image representing an annotation to be projected on an object to be projected an image be indicated from a remote location, but also the object and the object using an image obtained by capturing an image of the object on which the image is projected. A method for confirming an image projected on the camera is known (for example, Patent Document 1).

  The method is a method of annotating an object placed at a second position from a first position, wherein the second camera includes a video camera that images the object and a projection device that projects the image onto the object. Capturing a video of an object at a position and projecting an annotation image onto the object at a second position by a video camera and a projection device.

JP 2005-33756 A

  An object of the present invention is to provide an instruction device and an instruction program capable of instructing to project a specific image for specifying a part to be annotated, a control device and control program capable of controlling to project a specific image according to the instruction, and a specific image according to the instruction. It is to provide a projection system.

  An instruction apparatus according to the present invention includes a receiving unit that receives a captured image obtained by capturing a document, and an electronic file that represents the document captured in the captured image received by the receiving unit. Control means for controlling the display device to display, a projection instruction for instructing to project a specific image for identifying a part of the document controlled to be displayed by the control means and an annotation image for annotating the part of the document onto the document. And transmitting means for transmitting.

  In the above configuration, the projection instruction can employ a configuration in which an auxiliary image that assists annotation of the annotation image is projected to be deflected from the portion of the document that the annotation image annotates.

  In the above configuration, the control unit displays the display to change to the part of the other document to be annotated based on an electronic file representing another document different from the document captured in the captured image received by the receiving unit. It is possible to adopt a configuration in which the apparatus is controlled and the transmission means transmits a notification instruction instructing to notify that the document to be annotated has been changed.

  In the above configuration, the transmission unit may be configured to transmit an image representing another document to be annotated displayed on the display device.

  In the above configuration, the receiving unit receives information for identifying the document captured in the captured image, and the control unit displays the notification image for notifying the change of the document based on the information received by the receiving unit. A configuration can be employed in which the display device is controlled so as to display the portion of the document identified by the information while controlling the device.

  In the above configuration, the transmission unit may be configured to transmit a registration instruction instructing to register a writing image representing writing performed on the document and an electronic file representing the document in association with each other in the storage unit.

  The control device according to the present invention includes a transmission unit that transmits a captured image obtained by capturing a document, an annotation image that annotates a portion of the document captured in the captured image transmitted by the transmission unit, and a portion of the document that is annotated by the annotation image. Receiving means for instructing to project a specific image for specifying the image onto the document, and control for controlling the projection apparatus to project the annotation image and the specific image on the document according to the projection instruction received by the receiving means Means.

  In the above configuration, the projection instruction instructs to project the auxiliary image that assists the annotation of the annotation image by deflecting the part of the document that the annotation image annotates, and the control unit specifies the document specified by the specific image according to the projection instruction. It is possible to adopt a configuration in which the projection apparatus is controlled so as to project the auxiliary image onto the document by deflecting the portion.

  In the above configuration, the receiving unit receives a notification instruction for instructing that the document to be annotated has been changed, and the control unit selects a document to be annotated according to the notification instruction received by the receiving unit. It is possible to employ a configuration in which the projection apparatus is controlled so as to project a notification image that notifies the change.

  In the above configuration, the receiving unit can receive an image representing another document changed as an annotation target, and the control unit can control the projection apparatus to project the image received by the receiving unit.

  In the above configuration, the image processing apparatus further includes detection means for detecting a change in the position of the document imaged in the captured image, and an annotation image and an annotation image for annotating a document portion based on the change in the position of the document detected by the detection means. It is possible to employ a configuration in which the projection apparatus is controlled so as to change the position at which the specific image that specifies the part of the document to be annotated is projected.

  In the above configuration, it is possible to employ a configuration that further includes detection means for detecting information for identifying the document from the image of the document represented by the captured image, and the transmission means transmits the information detected by the detection means.

  In the above configuration, an extraction unit that extracts a writing image representing writing performed on the document based on an image of the document represented by the captured image and an image of the document generated based on the electronic file representing the document; A registration means for associating the written image extracted by the means with the electronic file representing the document written by the written image and registering it in the storage unit; and the receiving means comprises: the written image and the electronic file; It is possible to adopt a configuration in which a registration instruction for instructing registration in the storage unit is received and the control means controls the registration means to register in accordance with the registration instruction.

  The projection system according to the present invention includes an imaging unit that captures a document, an annotation image that annotates a captured document image on the captured image captured by the imaging unit, and a specific image that identifies the document portion that the annotation image annotates Receiving means for instructing to project the image onto the document, and projecting means for projecting the annotation image and the specific image onto the document in accordance with the projection instruction received by the receiving means.

  An instruction program according to the present invention is a document to be annotated based on a receiving unit that receives a captured image of a document and an electronic file that represents the document captured in the captured image received by the receiving unit. The control unit that controls the display device to display the part of the document, the specific image that specifies the part of the document controlled to be displayed by the control unit, and the annotation image that annotates the part of the document are instructed to be projected onto the document. It is characterized by functioning as a transmission means for transmitting a projection instruction.

  The control program according to the present invention includes a transmission unit that transmits a captured image obtained by capturing a document, an annotation image that annotates a portion of the document captured in the captured image transmitted by the transmission unit, and an annotation image that is annotated. A receiving unit that receives a projection instruction for instructing to project a specific image that specifies a part of the document onto the document, and a projection device that projects the annotation image and the specific image onto the document according to the projection instruction received by the receiving unit. It is characterized by functioning as control means for controlling.

  According to the configuration of the first aspect, it is possible to instruct to project not only an annotation image for annotating a captured document portion on the received image but also an image for specifying the displayed document portion to be annotated by the annotation image.

  According to the configuration of claim 2, since the projection instruction that directs to project the auxiliary image that assists the annotation of the annotation image by deflecting the portion of the annotation image to be annotated is transmitted, the annotation can be performed without interfering with the annotation of the annotation image. Can be instructed to assist.

  According to the configuration of claim 3, it is possible to instruct to notify that the document to be annotated has been changed.

  According to the configuration of claim 4, it is possible to transmit an image representing a document that has been notified that it has been changed as an annotation target.

  According to the configuration of claim 5, not only a change that has occurred in a document that is to be imaged and to which an annotation image is to be projected can be notified, but also a document that is to be an annotation that is displayed on the display device can be an annotation image. Can be matched with the document to be projected.

  According to the configuration of the sixth aspect, it is possible to instruct to register the writing image representing writing performed on the document and the electronic file representing the document in association with each other.

  According to the configuration of claim 7, in order to control not only an annotation image for annotating a part of the document but also a specific image for specifying the part to be annotated by the annotation image, the document to be annotated by projecting the annotation image Can be easily identified.

  According to the configuration of claim 8, the annotation image, which is the portion specified by the specific image, is controlled to project the auxiliary image that assists the annotation of the annotation image by deflecting the portion to be annotated, thereby preventing the annotation image from being annotated You can assist annotations.

  According to the configuration of the ninth aspect, in accordance with the notification instruction, control is performed so as to project an image for notifying that the document to be annotated has been changed. Therefore, it is possible to notify that the document to be annotated has been changed.

  According to the configuration of the tenth aspect, since control is performed to project an image representing a document that has been changed as an annotation target, it is possible to easily confirm the document that has notified the change.

  According to the configuration of the eleventh aspect, even when the position of the document changes, it can be controlled to change and project the position where the annotation image and the specific image are projected based on the change of the position of the document.

  According to the configuration of the twelfth aspect, it is possible to transmit information for identifying a document to be imaged and to which an annotation image is to be projected.

  According to the configuration of the thirteenth aspect, since the writing image representing writing performed on the document in accordance with the registration instruction and the electronic file representing the document are registered in the storage unit in association with each other, the contents written on the document can be easily known. Can do.

  According to the structure of claim 14, in order to project not only the annotation image for annotating a part of the document but also the specific image for specifying the part to be annotated by the annotation image, the part of the document to be annotated by projecting the annotation image is selected. Easy to identify.

  According to the configuration of the fifteenth aspect, it is possible to instruct the received image to project not only the annotation image for annotating the captured document portion but also the image for specifying the displayed document portion to be annotated by the annotation image.

  According to the configuration of claim 16, in order to control not only an annotation image for annotating a part of the document but also a specific image for specifying the part to be annotated by the annotation image, the document to be annotated by projecting the annotation image Can be easily identified.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Embodiments of the invention will be described below with reference to the accompanying drawings.

  FIG. 1 is a configuration diagram showing an embodiment of a projection instruction system including a projection system of the present invention having a control apparatus of the present invention and an instruction system having an instruction apparatus of the present invention.

  A projection instruction system 1 shown in FIG. 1 includes a projection system 10, an instruction system 20, and a communication network 100.

  The projection instruction system 1 instructs to project an annotation image that represents an annotation (hereinafter also referred to as an annotation image) and a specific image PS that identifies a portion that is an object of the annotation, and in accordance with the instruction, The specific image PS is projected.

  The projection system 10 includes a control device 1000, a projection device 1100, a first imaging device 1200, a dividing device 1300 (hereinafter also referred to as a half mirror), a second imaging device 1400, a sound output device 1500, a sound collection device 1600, a document 1910, An object 1920 and a projection screen 1990 are provided.

  The projection system 10 is connected to the instruction system 20 via the communication network 100. The projection system 10 transmits a captured image obtained by capturing the document 1910 to the instruction system 20 via the communication network 100, and then an annotation image for annotating the portion of the document 1910 captured in the transmitted captured image, and the annotation image is annotated. A projection instruction (hereinafter also referred to as a projection command) for instructing to project a specific image PS for identifying a portion of the document 1910 to be projected onto the document 1910 from the instruction system 20 via the communication network 100 and the received projection instruction Accordingly, the annotation image and the specific image PS are projected onto the document 1910.

  The control device 1000 will be described later. Therefore, the projection device 1100, the first imaging device 1200, and the dividing device 1300 will be described with reference to FIG. FIG. 2 is a diagram for explaining the arrangement positions of the projection device 1100, the first imaging device 1200, and the dividing device 1300.

  The projection device 1100 is constituted by a liquid crystal projector, for example, and is connected to the control device 1000. The projection apparatus 1100 is arranged such that the optical principal axis PO1 of the projection apparatus 1100 via the dividing apparatus 1300 and the optical principal axis PO2 of the first imaging apparatus 1200 via the dividing apparatus 1300 substantially coincide. Yes.

  The projection device 1100 is controlled by the control device 1000 according to the projection instruction, and projects the annotation image and the specific image PS onto the document 1910 via the dividing device 1300.

  The annotation image projected onto the document 1910 by the projection apparatus 1100 is an image that annotates a part of the document 1910, and the specific image projected onto the document 1910 by the projection apparatus 1100 identifies the part of the document 1910 that the annotation image annotates. It is an image to be.

  Further, the projection device 1100 is controlled by the control device 1000 to project the annotation image onto the object 1920 via the dividing device 1300. Note that the annotation image projected onto the object 1920 by the projection apparatus 1100 is an image for annotating the object 1920.

  Further, the projection apparatus 1100 projects the following various images onto the document 1910, details of which will be described later.

  The first imaging device 1200 is configured with, for example, a video camera or a Web camera, and is connected to the control device 1000. The first imaging device 1200 is installed at a position where the document 1910 and the object 1920 and an image projected on the document 1910 and the object 1920 can be imaged via the dividing device 1300.

  The first imaging device 1200 outputs a captured image obtained by capturing the document 1910 and the object 1920 to the control device 1000.

  The dividing device 1300 is configured by, for example, a beam splitter. In the present embodiment, the dividing device 1300 is configured by a half mirror that divides the light beam projected on the dividing device 1300 into a reflected light and a transmitted light at a ratio of approximately 1: 1.

  In the dividing device 1300, reflected light of light representing an image such as an annotation image projected by the projection device 1100 is irradiated on the document 1910 and the object 1920, and the first light is transmitted by the transmitted light reflected by the document 1910 and the object 1920. The imaging device 1200 is arranged so as to capture the image projected on the document 1910 and the object 1920 and the document 1910 and the object 1920.

  Here, returning to FIG. 1, the configuration of the projection instruction system 1 will be described.

  Similar to the first imaging device 1200, the second imaging device 1400 is configured with, for example, a video camera or a Web camera, and is connected to the control device 1000. The second imaging device 1400 is controlled by the control device 1000 in accordance with an imaging instruction for instructing imaging (hereinafter simply referred to as an imaging command), and images a part or the whole of the object 1920.

  In the present embodiment, the second imaging device 1400 will be described mainly as imaging a portion different from the portion of the object 1920 that the first imaging device 1200 images.

  In particular, in the present embodiment, the second imaging device 1400 is described as taking an image of an overhead view of the object 1920 or taking a close-up image from the first imaging device 1200, but the present invention is not limited thereto. It doesn't mean.

  Note that an image captured by the first captured image 1200 is hereinafter referred to as a first captured image. In addition, an image captured by the second captured image 1400 is hereinafter referred to as a second captured image.

  The sound output device 1500 is configured by a speaker, for example, and is connected to the control device 1000. The sound output device 1500 is controlled by the control device 1000 and outputs the transmission sound transmitted by the instruction system 20.

  The sound collection device 1600 is constituted by a microphone, for example, and is connected to the control device 1000. The sound collection device 1600 collects sound generated by a person who operates the control device 1000 or sound in a room where the devices constituting the projection system 10 are installed, and outputs the collected sound to the control device 1000.

  In the present embodiment, the object 1920 is illustrated as a cube, but is not limited thereto. For example, the object 1920 can employ a configuration that is a rectangular parallelepiped, a prism, a cylinder, a pyramid, a cone, a sphere, and the like, and a sheet-like planar shape.

  Next, the instruction system 20 will be described.

  The instruction system 20 includes an instruction device 2000, a display device 2100, an input device 2200, a sound output device 2500, and a sound collection device 2600. The instruction system 20 is connected to the projection system 10 via the communication network 100.

  The instruction system 20 displays the part of the document 1910 to be annotated based on the electronic file representing the document 1910 captured in the received first captured image after receiving the first captured image capturing the document 1910. The display device 2100 is controlled so as to project a specific image PS that identifies the part of the document 1910 that has been controlled to be displayed and an annotation image that annotates the part of the document 1910 are projected onto the document 1910. Send to system 10.

  The instruction device 2000 will be described after the control device 1000 is described.

  The display device 2100 is composed of, for example, a liquid crystal display, a plasma display, or a CRT (Cathode Ray Tube), and is connected to the pointing device 2000.

  The display device 2100 is controlled by the instruction device 2000 and displays a screen for displaying the following various images. The screen displayed by the display device 2100 will be described later.

  The input device 2200 is configured with, for example, a mouse, a keyboard, or a touch panel, and is connected to the instruction device 2000. The input device 2200 is operated by an instructor who gives an instruction, and inputs various types of information and signals representing the instruction.

  Similar to the sound output device 1500, the sound output device 2500 is configured with a speaker, for example, and is connected to the instruction device 2000. The sound output device 2500 is controlled by the instruction device 2000 and outputs the received sound received from the projection system 10.

  Similar to the sound collection device 1600, the sound collection device 2600 is composed of, for example, a microphone and is connected to the instruction device 2000. The sound collection device 2600 collects voices uttered by an instructor who operates the pointing device 2000, or voices in a room where the devices constituting the pointing system 20 are installed, and outputs the collected voices to the pointing device 2000.

  Before describing the control device 1000 and the instruction device 2000, referring to FIGS. 3 to 6, the projection device 1100 based on the display screen displayed by the display device 2100 and the projection instruction instructed by the instructor based on the display screen. The relationship with the image projected by will be described.

  FIG. 3 is a diagram illustrating an example of an image display screen displayed by the display device 2100 in order to display the first captured image and the second captured image.

  The image display screen FP shown in FIGS. 3A and 3B displays an annotation instruction part PCA, a document instruction part PCD, a first display part FS1, a second display part FS2, and a pointer PT1.

  The annotation instruction unit PCA displays buttons BT11 to BT13. For example, the annotation instruction unit PCA allows the instructor to operate the input device 2200, for example, so that the instructor performs an operation such as clicking after the pointer PT1 is superimposed on any of the buttons BT11 to BT13. This indicates that drawing of a desired annotation image instructing to project can be started.

  Buttons BT11 and BT are button displays indicating that drawing of an annotation image to be annotated using a straight line or curve drawn with a pen and a text character string can be started.

  When drawing of the annotation image is started, the input device 2200 can input information representing the annotation image and position information representing a position to be annotated by the annotation image. The information representing the annotation image refers to information representing the position of the start point and the end point of the line segment, for example, when the annotation image is a line segment.

  The button BT13 is a button display indicating that the input device 2200 can input a cancel instruction (hereinafter simply referred to as a cancel command) for canceling an instruction to project an already drawn annotation image.

  The document instruction unit PCD displays a button BT14. The button BT14 includes icons IC1 to 4 and a button BT15 (hereinafter also referred to as a determination button) used to indicate the range of the image ID of the document 1910 represented by the first captured image displayed on the first display unit FS1 described later. This is a button display indicating that a range instruction (hereinafter also referred to as a range instruction command) for instructing display can be input.

  As shown in FIG. 3B, for example, when the instructor operates the input device 2200, the arrowhead portions of the arrow-shaped icons IC <b> 1 to IC <b> 4 are aligned with the four vertices of the image ID of the document 1910. After the pointer PT1 is set to the determination button BT15, the input device 2200 performs a determination instruction (hereinafter, also referred to as a determination command) for determining the range of the image ID of the document 1910 by performing an operation such as clicking, for example. ) Can be entered.

  The first display unit FS1 displays a first captured image representing the image ID of the document 1910 and the image IT1 of the object 1920 captured by the first imaging device 1200.

  The second display unit FS2 displays a second captured image representing the image IT2 of the object 1920 captured by the second imaging device 1400.

  Next, the document display screen FD used for the display device 2100 to display the document 1910 representing an image in the first captured image displayed on the image display screen FP will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a document display screen FD used for displaying the document 1910.

  The document display screen FD shown in FIG. 4 displays the annotation instruction unit PCA, the display change unit PAS, and the document display unit PSD.

  The annotation instruction unit PCA displayed on the document display screen FD represents that the input device 2200 can input a projection instruction that instructs to project a desired annotation image for annotating the part of the document 1910 displayed by the document display unit PSD.

  Note that the annotation instruction unit PCA displayed on the document display screen FD is the same as the annotation instruction unit PCA displayed on the image display screen FP described with reference to FIG.

  The display change unit PAS displays buttons BT26 and 27. For example, the display changing unit PAS is operated by, for example, the clicker operating the input device 2200 so that the pointer performs the operation such as clicking after the pointer PT1 is superimposed on one of the buttons BT26 and 27. This indicates that the input device 2200 can input a change instruction (hereinafter simply referred to as a change command) for instructing to change the display range of the document image representing the document 1910 in the document display unit PSD.

  Buttons BT 26 and 27 are button displays indicating that the input device 2200 can input a magnification change instruction (hereinafter also simply referred to as a magnification change command) for changing a display magnification for displaying a document image. The buttons BT26 and 27 are collectively referred to as magnification change buttons BT26 and 27 hereinafter.

  The magnification change button BT26 is a button display indicating that the input device 2200 can input an instruction to increase the display magnification, and the magnification change button BT27 is an instruction to increase the display magnification.

  The magnification change instruction is included in the change instruction. Further, the display range is changed with the change of the magnification.

  The document display unit PSD displays a part of the document image (hereinafter simply referred to as a partial image). Further, the document display unit PSD displays a palm-shaped pointer PT2.

  The pointer PT2 is a pointer display indicating that the input device 2200 can input a range change instruction (hereinafter also referred to as a range change command) for instructing to change the display range. The range change instruction is included in the change instruction.

  More specifically, the instructor performs, for example, a drag and drop operation. First, the open palm-shaped pointer PT2 is changed to a grasped palm-shaped pointer PT2 by a drag operation, and the display position of the grasped palm-shaped pointer PT2 is changed, and then the shape of the pointer PT2 is changed by a drop operation. Return to the open palm shape.

  By this operation, the input device 2200 inputs a range change instruction that instructs to change the display range based on the display position of the pointer PT2.

  Next, the relationship between the display of the display device 2100 and the projection of the projection device 1100 will be described with reference to FIG. FIG. 5 is a diagram for explaining an example of the relationship between the display of the display device 2100 and the projection of the projection device 1100.

  First, another example of the document display screen FD displayed on the display device 2100 will be described with reference to FIG. FIG. 5A is a diagram illustrating another example of the document display screen FD displayed on the display device 2100.

  The display on the document display screen FD shown in FIG. 5A is almost the same as the display on the document display screen FD described with reference to FIG. 4, so the differences will be mainly described below.

  The document display unit PSD of the document display screen FD shown in FIG. 5A displays the annotation images PA31 and 32. The annotation images PA31 and PA32 are images that instruct the projection instruction input by the input device 2200 to be projected by the operation of the instructor.

  Next, an example of an image projected by the projection apparatus 1100 based on the projection instruction transmitted by the instruction apparatus 2000 that controls to display the screen illustrated in FIG. 5A will be described with reference to FIG. To do. FIG. 5B is a diagram illustrating an example of an image projected by the projection device 1100.

  As shown in FIG. 5B, the projection device 1100 projects the specific image PS1, the annotation images PA11 and PA12, the instructor image PP1, the auxiliary images PD11 and 12, and the guide images PG11 and 12.

  The specific image PS1 is an image representing the display range of the document 1910 displayed on the display device 2100. The specific image PS1 is also an image that specifies the portion of the document 1910 that the annotation images PA11 and 12 to project annotate.

  The annotation images PA11 and PA12 are images that annotate the part of the document 1910 specified by the specific image PS1.

  The instructor image PP1 is an image that represents information about the instructor who instructs to project the annotation images PA11 and PA12 that annotate the part of the document 1910 specified by the specific image PS1.

  In this embodiment, the information about the instructor includes, for example, information for identifying the instructor such as a name (hereinafter simply referred to as instructor identification information), and the display magnification of the display device 2100 that performs display for the instructor. Although described as including the information to represent, it is not necessarily limited to this.

  For example, the information regarding the instructor may include a configuration including information for identifying the instruction device 2000 operated by the instructor, such as an IP address (Internet Protocol Address), a machine name, or a domain name.

  The auxiliary images PD11 and 12 are images that assist the annotation of the annotation images PA11 and PA12. The auxiliary images PD11 and 12 are portions of the document 1910 that the annotation images PA11 and 12 annotate. The auxiliary images PD11 and 12 are projected on the projection screen 1990, for example, by deviating from the portion specified by the specific image PS1.

  In the present embodiment, the auxiliary images PD11 and 12 are described as images obtained by enlarging the annotation images PA11 and 12 based on the display magnification of the display device 2100. However, the present invention is not limited to this. The auxiliary images PD11 and 12 can employ a configuration that is an image representing supplementary explanation for assisting the annotation of the annotation images PA11 and PA12.

  The reason that the images obtained by enlarging the annotation images PA11 and 12 based on the display magnification of the display device 2100 are used as the auxiliary images PD11 and 12 is that the display device 2100 increases the display magnification and displays the document 1910. Since the instructed annotation images PA11 and 12 tend to be smaller, the projection device 1100 can assist the annotations of the annotation images PA11 and 12 by projecting the auxiliary images PD11 and 12 that are enlarged from the annotation images PA11 and 12. It is.

  The guide image PG11 is an image that guides the viewing position of the person viewing the specific image PS1 from the specific image PS1 to the instructor image PP1 related to the specific image PS1. In addition, the guidance image PG12 is an image that guides the viewing position of the person viewing the annotation images PA11 and PA12 from the annotation images PA11 and PA12 to the auxiliary images PD11 and PD12 that assist the annotation of the annotation images PA11 and PA12.

  The guide images PG11 and 12 are the parts of the document 1910 that are annotated by the annotation images PA11 and 12 projected to the visual position before the guide, similarly to the auxiliary images PD11 and 12, and the part specified by the specific image PS1. For example, it is projected onto the projection screen 1990.

  Next, the projection position of the projection apparatus 1100 that changes as the display range of the display apparatus 2100 changes will be described with reference to FIG. FIG. 6 is a diagram illustrating an example of a change in position of an image projected by the projection apparatus 1100.

  FIG. 6A is a diagram for describing an image projected by the projection apparatus 1100 when the display magnification is 300% and the display apparatus 2100 sets the upper left portion of the document 1910 as the display range.

  As shown in FIG. 6A, the projection device 1100 projects the specific image PS1 on the upper left part on the surface of the document 1910. 6A to 6C, the projection apparatus 1100 deflects the projected specific images PS1 to 3, and projects the instructor images PP1 to 3 and the guide images PG11 to 31 onto the projection screen 1990.

  FIG. 6B is a diagram for explaining an image projected by the projection device 1100 when the display magnification is enlarged to 600%, which is twice, and the display device 2100 uses the upper left portion of the document 1910 as the display range. It is.

  As shown in FIG. 6B, the projection device 1100 projects the specific image PS2 on the upper left as in FIG. 6A. Unlike FIG. 6B, the projection range of the specific image PS2 is Compared to the specific image PS1, it has only a quarter of the area.

  FIG. 6C is a diagram for explaining an image projected by the projection device 1100 when the display magnification is 300% and the display device 2100 uses the lower right portion of the document 1910 as the display range.

  As shown in FIG. 6C, the projection device 1100 projects the specific image PS3 on the lower right, unlike FIG. Note that the projection range of the specific image PS3 has the same area as that of the specific image PS1 shown in FIG. 6A because the display magnification is the same as in the case of FIG.

  Here, returning to FIG. 1, the configuration of the projection instruction instruction system 1 will be described.

  The control device 1000 is configured by, for example, a personal computer, and is connected to the communication network 100, the projection device 1100, the first imaging device 1200, the second imaging device 1400, the sound output device 1500, and the sound collection device 1600.

  After transmitting the first captured image obtained by capturing the document 1910, the control device 1000 annotates the part of the document 1910 captured in the transmitted first captured image, and the document 1910 annotated by the annotation image PA. A projection instruction for instructing to project a specific image PS for specifying a part onto the document 1910 is received, and the projection apparatus 1100 is controlled to project the annotation image PA and the specific image PS onto the document 1910 according to the received projection instruction. To do.

  Here, the configuration of the control apparatus 1000 will be described with reference to FIG. FIG. 7 is a functional block diagram illustrating a configuration example of the control device 1000.

  The control apparatus 1000 includes a reception unit 1010, a transmission unit 1020, a storage unit 1030, a detection unit 1040, and a control unit 1090. The reception unit 1010, the transmission unit 1020, the storage unit 1030, the detection unit 1040, and the control unit 1090 are connected so as to be able to exchange information with each other via the bus 1005.

  The functions of the detection unit 1040 and the control unit 1090 are realized by software control executed by the control device 1000.

  Here, with reference to FIG. 8, the hardware configuration of the control apparatus 1000 used for executing the software control will be described. FIG. 8 is a diagram illustrating an example of a hardware configuration of the control device 1000 used for executing software control.

  The control apparatus 1000 includes, for example, a calculation unit 1001 such as a CPU (Central Processing Unit), a ROM 1002 (Read-Only Memory) such as an EP (Erasable Programmable Read-Only Memory) or an EEPROM (Electrically Erasable Programmable Read-Only Memory). RAM 1003 (Random Access Memory) composed of volatile memory such as Only Memory), DRAM (Dynamic RAM) or SRAM (Static RAM) and nonvolatile memory such as NVRAM (Non Volatile RAM), and hard disk The storage unit 1004 includes an arithmetic unit 1001, a ROM 1002, a RAM 1003, and an external storage unit 1004 that are connected to each other via a bus 1005.

  Software control is realized by the calculation unit 1001 reading a program stored in the ROM 1002 or the external storage unit 1004 and performing a calculation according to the read program. Note that operation result data is written in the RAM 1003. In particular, the NVRAM stores data that needs to be backed up when the power is turned off.

  The receiving unit 1010 is configured with a network adapter, for example, and is connected to the communication network 100. The receiving unit 1010 executes a reception process, thereby annotating an image PA that annotates the portion of the document 1910 captured in a first captured image transmitted by the transmitting unit 1020 described below, and an annotation image PA annotated by the annotation image PA. A projection instruction for instructing to project the specific image PS for specifying the portion onto the document 1910 is received.

  The receiving unit 1010 receives the following various information, instructions, notifications, and images, and outputs the received information and the like to the control unit 1090. Therefore, unless otherwise specified, information output from the reception unit 1010 to the control unit 1090 is information received by the reception unit 1010.

  Similar to the receiving unit 1010, the transmitting unit 1020 is configured with, for example, a network adapter and is connected to the communication network 100. The transmission unit 1010 transmits a first captured image obtained by capturing the document 1910 by executing a transmission process.

  The transmission unit 1020 acquires the following various other information, instructions, notifications, and images from the control unit 1090, and is controlled by the control unit 1090 to transmit the acquired information and the like via the communication network 100. Send to.

  The storage unit 1030 includes, for example, a RAM 1003 or an external storage unit 1004. The storage unit 1030 stores the annotation image PA and information related to the annotation image PA.

  The information about the annotation image PA includes, for example, identification information for identifying the annotation image PA such as a file name of an electronic file representing the annotation image PA, position information indicating a position where the annotation image PA is projected, and the annotation image PA. The target information indicating whether or not the target to be processed is the document 1910 is included.

  In the present embodiment, the position information is described as being expressed using coordinate values, but the present invention is not limited to this. Further, the target information represents a case where the target on which the annotation image PA is projected using the value “TRUE” is the document 1910, but is not limited thereto.

  Here, with reference to Table 1, information regarding the annotation image PA stored in the storage unit 1030 will be described. Table 1 is a table showing an example of information related to the annotation image PA stored in the storage unit 1030.

  Table 1 has an annotation image identification information field, a position information field, and a target information field. The annotation image identification information field stores identification information for identifying the annotation image PA.

  The position information field and the target information field store the position information and target information of the annotation image PA identified by the information stored in the annotation image identification information field of the same record.

  That is, the storage unit 1030 associates and stores identification information for identifying the annotation image PA, position information indicating a position where the annotation image PA is projected, and target information indicating a target on which the annotation image PA is projected.

  The storage unit 1030 stores the specific image PS and information related to the specific image PS.

  Note that the information related to the specific image PS includes, for example, identification information for identifying the specific image PS such as a file name of an electronic file representing the specific image PS, and range information indicating a range in which the specific image PS is projected.

  In the present embodiment, the range information will be described using two different coordinate values, but the present invention is not limited to this.

  Here, with reference to Table 2, information regarding the specific image PS stored in the storage unit 1030 will be described. Table 2 is a table showing an example of information related to the specific image PS stored in the storage unit 1030.

  Table 2 has a specific image identification information field and a range information field. The annotation image identification information field stores identification information for identifying the specific image PS.

  The range information field stores the range information of the specific image PS identified by the information stored in the specific image identification information field of the same record.

  That is, the storage unit 1030 stores identification information for identifying the specific image PS and range information indicating a range in which the specific image PS is projected in association with each other.

  Still further, the storage unit 1030 associates the specific image SP with the instructor image PP that represents information about the instructor who instructs to project the annotation image PA that annotates the part of the document 1910 specified by the specific image SP. Remember.

  Here, with reference to Table 3, information relating the specific image PS and the instructor image PP stored in the storage unit 1030 will be described. Table 3 is a table showing an example of information in which the specific image PS and the instructor image PP stored in the storage unit 1030 are associated with each other.

  Table 3 has a specific image identification information field and an instructor identification information field. The annotation image identification information field stores identification information for identifying the specific image PS.

  The instructor identification information field stores information for identifying the instructor image PP related to the specific image PS identified by the information stored in the specific image identification information field of the same record.

  Furthermore, the storage unit 1030 stores the specific image SP, the annotation image PA for annotating the part of the document 1910 specified by the specific image SP, and the auxiliary image PD for assisting the annotation of the annotation image PA in association with each other.

  Here, with reference to Table 4, information relating the specific image PS, the annotation image PA, and the auxiliary image PD stored in the storage unit 1030 will be described. Table 4 is a table illustrating an example of information in which the specific image PS, the annotation image PA, and the auxiliary image PD stored in the storage unit 1030 are associated with each other.

  Table 4 has a specific image identification information field, an annotation image identification information field, and an auxiliary image identification information field. The annotation image identification information field stores identification information for identifying the specific image PS.

  The specific image identification information field stores information for identifying the annotation image PA that annotates the part specified by the specific image PS identified by the information stored in the specific image identification information field of the same record.

  The auxiliary image identification information field stores information for identifying the auxiliary image PD that assists the annotation of the annotation image PA identified by the information stored in the annotation image identification information field of the same record.

  Note that the storage unit 1030 stores other various types of information and images described below, but other various types of information and images stored by the storage unit 1030 will be described later.

  The detection unit 1040 detects the image ID of the document 1910 from the first captured image obtained by capturing the document 1910 by executing a detection process described later.

  Here, with reference to FIG. 9, the detection process which the detection part 1040 performs is demonstrated. FIG. 9 is a flowchart illustrating an example of detection processing executed by the detection unit 1040.

  First, the detection unit 1040 acquires a first captured image from the first imaging device 1200 (step ST0001). Next, the detection unit 1040 detects an image of the document 1910 based on the color of each pixel of the first captured image (step ST0002).

  As a specific example, when the document 1910 is a document in which a sentence is printed on white paper, the detection unit 1040 detects a portion of the first captured image in which a predetermined number of white pixels are surrounded by a portion of the document 1910. Detect as an image.

  Thereafter, detection unit 1040 outputs the detection result to control unit 1090 (step ST0003). Next, the detection unit 1040 ends the execution of the detection process.

  Next, returning to FIG. 7, the configuration of the control device 1000 will be described.

  The control unit 1090 is connected not only to each unit included in the control device 1000 but also to the projection device 1100, the first imaging device 1200, the second imaging device 1400, the sound output device 1500, and the sound collection device 1600. The control unit 1090 controls each unit and each device connected to the control unit 1090 by executing a control process described later.

  In particular, the control unit 1090 controls the projection device 1100 to project the annotation image PA and the specific image PS onto the document 1910 in accordance with the projection instruction received by the reception unit 1010 by executing control processing.

  Further, the control unit 1090 controls the projection apparatus 1100 to project the auxiliary image PD onto the document 1910 by deflecting the part of the document 1910 specified by the specific image PS according to the projection instruction by executing the control process.

  Here, the control process executed by the control unit 1090 will be described with reference to FIG. FIG. 10 is a flowchart illustrating an example of a control process executed by the control unit 1090.

  First, the control unit 1090 executes an initial control process for controlling the projection apparatus 1100 in an initial state before obtaining a projection instruction (step ST0101). The initial control process will be described later.

  Next, control unit 1090 starts processing to save, in storage unit 1030, voice information that is collected by sound collection device 1600 and voice information that is received by reception unit 1010 (step ST0102).

  Note that the control unit 1090 stores the information indicating the time when the storage is started and the voice information in association with each other and stores them in the storage unit 1030. In addition, the control unit 1090 executes the processing for storing the audio information in parallel with the control processing by executing the processing using, for example, multithreading or multiprocessing.

  Thereafter, control unit 1090 determines whether or not an end instruction (hereinafter also referred to as an end command) for instructing execution of processing has been acquired from receiving unit 1010 (step ST0103). If it is determined that an end instruction has been acquired, control unit 1090 executes the process of step ST0113, and otherwise, executes the process of step ST0104. The end instruction is transmitted by the instruction device 2000.

  In Step ST0103, when it is determined that the end instruction has not been acquired, the control unit 1090 instructs the receiving unit 1010 to capture an image of the object 1900 from the desired position, or sets the desired position of the object 1900. It is determined whether or not an imaging instruction (that is, an imaging command) for instructing imaging has been acquired (step ST0104). If it is determined that the imaging command has been acquired, control unit 1090 performs the process of step ST0105, and otherwise, executes the process of step ST0107.

  When determining in step ST0104 that the imaging instruction has been acquired, the control unit 1090 acquires the imaging position from the imaging instruction (step ST0105). Next, the control unit 1090 controls the second imaging device 1400 to image the object 1920 from the imaging position, or to image the imaging position of the object 1920 (step ST0106). Thereafter, control unit 1090 returns to step ST0103 and repeats the above processing.

  As a specific example, the control unit 1090 controls the imaging position of the second imaging device 1400 by outputting a control command for controlling the second imaging device 1400.

  In step ST0104, when it is determined that the imaging instruction is not acquired, the control unit 1090 determines whether the projection instruction is acquired from the receiving unit 1010 (step ST0107). If it is determined that the projection instruction has been acquired, control unit 1090 performs the process of step ST0108, and otherwise, executes the process of step ST0110.

  In step ST0107, when it is determined that the imaging instruction has been acquired, control unit 1090 executes a projection image generation process for generating a projection image to be projected by projection apparatus 1100 (step ST0108). The projection image generation process will be described later.

  Next, control unit 1090 controls projection apparatus 1100 to project the projection image generated in step ST0108 (step ST0109). Thereafter, control unit 1090 returns to step ST0103 and repeats the above processing.

  In step ST0107, when it is determined that the imaging instruction is not acquired, the control unit 1090 executes an information acquisition process of acquiring information transmitted to the instruction device 2000 and information representing the sound output by the sound output device 1500. (Step ST0110). The information acquisition process will be described later.

  Next, control section 1090 executes information transmission processing for transmitting the information acquired in step ST0110 to instruction device 2000 (step ST0111). The information transmission process will be described later.

  After that, the control unit 1090 controls the sound output device 1500 to output sound based on the sound information acquired in step ST0110 (step ST0112).

  Note that the voice information acquired in step ST0110 is voice information received by the receiving unit 1010 (hereinafter referred to as received voice information), which is voice representing the voice of the instructor who operates the pointing device 2000.

  Thereafter, control unit 1090 returns to step ST0103 and repeats the above processing.

  When determining in step ST0103 that the control unit 1090 has acquired the end instruction, the control unit 1090 ends the execution of the process of saving the audio information in the storage unit 1030 (step ST0113). Thereafter, the control unit 1090 ends the execution of the control process.

  Note that the control unit 1090 can control the thread or process generated in step ST0102 to finish storing the audio information by outputting a signal such as SIGTERM or SIGKILL, for example.

  Here, an initial control process executed by the control unit 1090 will be described with reference to FIG. FIG. 11 is a flowchart illustrating an example of an initial control process executed by the control unit 1090.

  First, control unit 1090 clears a buffer composed of a predetermined area secured in storage unit 1030 (step ST0201). Next, control unit 1090 generates an initial image (step ST0202). Note that the initial image can be configured to be an image composed of all white pixels, for example.

  Thereafter, control unit 1090 stores the generated initial image in the buffer as a projection image (step ST0203). Next, control unit 1090 controls projection apparatus 1100 to project a projection image that is an initial image (step ST0204). Thereafter, the control unit 1090 ends the execution of the initial control process.

  Here, with reference to FIG. 12, a projection image generation process executed by the control unit 1090 will be described. FIG. 12 is a flowchart illustrating an example of a projection image generation process executed by the control unit 1090.

  First, the control unit 1090 executes a drawing process for drawing an image such as the specific image PS to be superimposed on the projection image (step ST0301). The drawing process will be described later.

  Next, control section 1090 determines whether or not storage section 1030 stores the target annotation image generated in step ST0301 (step ST0302). Note that the target annotation image refers to an annotation image PA that annotates the object 1920.

  Specifically, the control unit 1090 determines whether or not there is a record in which the value of the target information field in Table 1 stored in the storage unit 1030 is “FALSE”, so that the storage unit 1030 performs the target annotation image. Can be determined.

  When it is determined that storage unit 1030 stores the target annotation image, control unit 1090 executes the process of step ST0303, and otherwise, executes the process of step ST0304.

  In step ST0302, when the storage unit 1030 determines that the storage unit 1030 stores the target annotation image, the control unit 1090 executes target annotation image superimposition processing for superimposing the target annotation image on the projection image (step ST0303). The target annotation image superimposing process will be described later.

  In step ST0302, the control unit 1090 determines whether or not the document annotation projection mode is set when the storage unit 1030 determines not to store the target annotation image or after executing step ST0303 (step ST0304).

  The document annotation projection mode refers to an execution form in which an image such as the specific image PS is projected onto the document 1910. In the present embodiment, the control unit 1090 is described as determining whether or not the document annotation projection mode is set by referring to a setting file or a registry representing settings, for example. However, the present invention is not limited to this. .

  For example, when a mode change instruction (hereinafter also referred to as a mode change command) for instructing to change the execution form to the document annotation projection mode is acquired from the instruction device 2000, the control unit 1090 is in the document annotation projection mode. The structure to judge can be employ | adopted.

  If it is determined that the mode is the document annotation projection mode, the control unit 1090 ends the process of step ST0305, and otherwise ends the projection image generation process.

  In step ST0304, when determining that the document annotation projection mode is set, the control unit 1090 determines whether the detection unit 1040 has detected the image ID of the document 1910 from the first captured image (step ST0305). . If it is determined that an image of document 1910 has been detected, control unit 1090 ends the process of step ST0306, and otherwise ends the projection image generation process.

  When determining in step ST0305 that the image of the document 1910 has been detected, the control unit 1090 determines whether the storage unit 1030 stores a document annotation image (step ST0306). The document annotation image is an annotation image PA that annotates the document 1910.

  Specifically, the control unit 1090 determines whether or not there is a record in which the value of the target information field in Table 1 stored in the storage unit 1030 is “TRUE”, so that the storage unit 1030 has the document annotation image. Can be determined.

  If it is determined that storage unit 1030 stores the document annotation image, control unit 1090 performs the process of step ST0307, and otherwise the process of step ST0308.

  In Step ST0306, when the storage unit 1030 determines that the document annotation image is stored, the control unit 1090 executes document annotation image superimposing processing for superimposing the document annotation image on the projection image (Step ST0307). The document annotation image superimposing process will be described later.

  In step ST0306, when the storage unit 1030 determines that the document annotation image is not stored, or after executing step ST0307, the control unit 1090 executes a specific image superimposing process for superimposing the specific image PS on the projection image ( Step ST0308). The specific image superimposing process will be described later.

  Next, control unit 1090 determines whether or not storage unit 1030 stores auxiliary image PD (step ST0309). The control unit 1090 executes the process of step ST0310 when the storage unit 1030 determines that the auxiliary image PD is stored, and executes the process of step ST0311 otherwise.

  Specifically, the control unit 1090 determines whether there is a record storing information in the auxiliary image identification information field of Table 4 stored in the storage unit 1030, so that the storage unit 1030 stores the auxiliary image PD. Whether or not to memorize can be determined.

  When determining that the storage unit 1030 stores the auxiliary image PD in step ST0309, the control unit 1090 executes auxiliary image superimposing processing for superimposing the auxiliary image PD on the projection image (step ST0310). The auxiliary image superimposing process will be described later.

  In Step ST0309, the control unit 1090 performs an instructor image superimposing process for superimposing the instructor image PP on the projection image when the storage unit 1030 determines that the auxiliary image PD is not stored or after executing Step ST0310. (Step ST0311). The instructor image superimposition process will be described later.

  Next, the control unit 1090 executes a guidance image superimposing process for superimposing the guidance image PG on the projection image (step ST0312). The guidance image superimposing process will be described later. Thereafter, the control unit 1090 ends the execution of the projection image generation process.

  Here, with reference to FIG. 13, a drawing process executed by the control unit 1090 will be described. FIG. 13 is a flowchart illustrating an example of a drawing process executed by the control unit 1090.

  First, control unit 1090 determines whether or not the acquired projection instruction is a projection instruction for instructing to project a document annotation image (step ST0401). If it is determined that the instruction is to project a document annotation image, control unit 1090 executes the process of step ST0402; otherwise, it executes the process of step ST0406.

  In step ST0401, if the control unit 1090 determines that the acquired projection instruction is a projection instruction for instructing to project the document annotation image, the control unit 1090 executes document annotation image drawing processing for drawing the document annotation image (step ST0401). ST0402). The document annotation image drawing process will be described later.

  Next, control unit 1090 executes a specific image drawing process for drawing specific image PS (step ST0403). The specific image drawing process will be described later. Thereafter, control unit 1090 executes an instructor image drawing process for drawing instructor image PP (step ST0404). The instructor image drawing process will be described later.

  Next, control section 1090 executes auxiliary image drawing processing for drawing auxiliary image PD (step ST0405). The auxiliary image drawing process will be described later. Next, the control unit 1090 ends the execution of the drawing process.

  In step ST0401, when it is determined that the acquired projection instruction is not a projection instruction for instructing to project the document annotation image, the control unit 1090 executes target annotation image drawing processing for drawing the target annotation image (step ST0406). ). The target annotation image drawing process will be described later. Next, the control unit 1090 ends the execution of the drawing process.

  Next, a target annotation image drawing process, a document annotation image drawing process, and a specific image drawing process executed by the control unit 1090 will be described with reference to FIG. FIG. 14 is a flowchart illustrating an example of target annotation image drawing processing, document annotation image drawing processing, and specific image drawing processing executed by the control unit 1090.

  First, the target annotation image drawing process executed by the control unit 1090 will be described with reference to FIG.

  First, the control unit 1090 acquires information representing the target annotation image from the projection instruction (step ST0501). Since the information representing the target annotation image is the same as the information representing the annotation image, description thereof will be omitted.

  Next, control unit 1090 generates a target annotation image based on the information representing the acquired target annotation image (step ST0502). Thereafter, control unit 1090 obtains position information representing the projection position of the target annotation image from the projection instruction (step ST0503).

  Thereafter, control unit 1090 obtains target information representing the projection target from the projection instruction (step ST0504). Next, the control unit 1090 stores the target annotation image in the storage unit 1030, and associates the identification information of the target annotation image with the position information and target information of the target annotation image, and stores the buffer configured by the storage unit 1030. Is stored in Table 1 (step ST0505). Thereafter, the control unit 1090 ends the execution of the target annotation image drawing process.

  Next, the document annotation image drawing process executed by the control unit 1090 will be described with reference to FIG.

  First, control unit 1090 obtains information representing a document annotation image from a projection instruction (step ST0601). The information representing the document annotation image is the same as the information representing the target annotation image, and a description thereof will be omitted.

  Next, control section 1090 generates a document annotation image based on the information representing the acquired document annotation image (step ST0602). Thereafter, control unit 1090 obtains position information indicating the projection position of the document annotation image from the projection instruction (step ST0603). Thereafter, control unit 1090 obtains target information representing the projection target from the projection instruction (step ST0604). Next, the control unit 1090 saves the document annotation image in the storage unit 1030, associates the identification information of the document annotation image with the position information and target information of the document annotation image, and saves them in Table 1 stored in the buffer ( Step ST0605). Thereafter, the control unit 1090 ends the execution of the document annotation image drawing process.

  Finally, the specific image drawing process executed by the control unit 1090 will be described with reference to FIG.

  First, the control unit 1090 clears the identification information and range information of the specific image PS from Table 2 stored in the buffer (step ST0701).

  Next, control unit 1090 obtains range information representing the range of specific image PS from the projection instruction (step ST0702). Thereafter, control unit 1090 generates specific image PS based on the acquired range information (step ST0703).

  Thereafter, control unit 1090 saves specific image PS in storage unit 1030, associates the identification information of specific image PS with the range information of specific image PS, and saves them in Table 2 stored in the buffer (step ST0704). . Thereafter, the control unit 1090 ends the execution of the specific image drawing process.

  Next, the instructor image drawing process and the auxiliary image drawing process executed by the control unit 1090 will be described with reference to FIG. FIG. 15 is a flowchart illustrating an example of the instructor image drawing process and the auxiliary image drawing process executed by the control unit 1090.

  First, the instructor image drawing process executed by the control unit 1090 will be described with reference to FIG.

  First, the control unit 1090 clears the identification information of the instructor image PP and the identification information of the specific image PS from Table 3 stored in the buffer configured by the storage unit 1030 (step ST0801).

  Next, control unit 1090 acquires instructor identification information from the projection instruction (step ST0802). Thereafter, the display magnification is acquired from the projection instruction (step ST0803).

  Next, control unit 1090 generates an instructor image based on the instructor identification information and the display magnification (step ST0804). Thereafter, the control unit 1090 stores the instructor image in the storage unit 1030, and also associates the identification information of the instructor image with the identification information of the specific image PS drawn by the above processing, and is configured by the storage unit 1030. Is stored in the table 3 stored in step ST0805. Thereafter, the control unit 1090 ends the execution of the instructor image drawing process.

  Next, an auxiliary image drawing process executed by the control unit 1090 will be described with reference to FIG.

  First, the control unit 1090 clears the identification information of the auxiliary image PD, the identification information of the annotation image PA, and the identification information of the specific image PS from Table 4 stored in the buffer (step ST0901).

  Next, based on the range information of the specific image PS stored in Table 2 and the position information of the annotation image PA stored in Table 1, the control unit 1090 projects the annotation to be projected into the specific image PS stored in Table 1. Image PA is acquired (step ST0902).

  After that, the control unit 1090 enlarges the annotation image PA based on the display magnification to generate the auxiliary image PD (step ST0903). Next, the control unit 1090 saves the generated auxiliary image PD in the storage unit 1030, and specifies the identification information of the saved auxiliary image PD, the identification information of the annotation image PA to be auxiliary, and the annotation range. The identification information of the image PS is stored in association with the table 4 stored in the buffer (step ST0904). Thereafter, the control unit 1090 ends the execution of the auxiliary image drawing process.

  Next, the target annotation image superimposing process, document annotation image superimposing process, and specific image superimposing process executed by the control unit 1090 will be described with reference to FIG. FIG. 16 is a flowchart illustrating an example of target annotation image superimposition processing, document annotation image superimposition processing, and specific image superimposition processing executed by the control unit 1090.

  First, the target annotation image superimposing process executed by the control unit 1090 will be described with reference to FIG.

  First, the control unit 1090 acquires identification information for identifying the annotation image PA (that is, the target annotation image) associated with the target information having the value “FALSE” representing the target object 1920 from the table 1 stored in the buffer, and identifies the identification information. Based on the information, the target annotation image is acquired from the storage unit 1030 (step ST1001).

  Next, control unit 1090 acquires position information associated with the acquired annotation image PA from Table 1 stored in the buffer (step ST1002). Thereafter, control unit 1090 superimposes the acquired annotation image PA on the projection image based on the acquired position information (step ST1003). Thereafter, the control unit 1090 ends the execution of the target annotation image superimposition process.

  Next, with reference to FIG. 16B, the document annotation image superimposing process executed by the control unit 1090 will be described.

  First, the control unit 1090 acquires identification information for identifying the annotation image PA (that is, the document annotation image) associated with the target information having the value “TRUE” from the table 1 stored in the buffer, and stores the identification information based on the identification information. A document annotation image is acquired from unit 1030 (step ST1101). Next, control unit 1090 acquires position information associated with the acquired annotation image PA from Table 1 stored in the buffer (step ST1102). Thereafter, control unit 1090 superimposes the acquired annotation image PA on the projection image based on the acquired position information (step ST1103). Thereafter, the control unit 1090 ends the execution of the document annotation image superimposing process.

  Finally, the specific image superimposing process executed by the control unit 1090 will be described with reference to FIG.

  First, the control unit 1090 acquires identification information for identifying the specific image PS from Table 2 stored in the buffer, and acquires the specific image PS from the storage unit 1030 based on the identification information (step ST1201). Next, control unit 1090 acquires range information associated with the acquired specific image PS from Table 2 stored in the buffer (step ST1202). Thereafter, control unit 1090 superimposes the acquired specific image PS on the projection image based on the acquired range information (step ST1203). Thereafter, the control unit 1090 ends the execution of the specific image superimposing process.

  Next, the instructor image superimposing process and the auxiliary image superimposing process executed by the control unit 1090 will be described with reference to FIG. FIG. 17 is a flowchart illustrating an example of the instructor image superimposing process and the auxiliary image superimposing process executed by the control unit 1090.

  First, the instructor image superimposing process executed by the control unit 1090 will be described with reference to FIG.

  First, the control unit 1090 acquires identification information for identifying the instructor image PP from Table 3 stored in the buffer, and acquires an instructor image from the storage unit 1030 based on the identification information (step ST1301).

Next, control unit 1090 specifies a specific image PS stored in association with the acquired instructor image PP from Table 3 stored in the buffer (step ST1302).
Thereafter, control unit 1090 determines a position outside a predetermined amount from the projection range of identified specific image PS as a projection position (step ST1303).

  Next, the control unit 1090 superimposes the instructor image PP on the projection image based on the determined projection position (step ST1304). Thereafter, the control unit 1090 ends the execution of the instructor image superimposition process.

  Next, with reference to FIG. 17B, the auxiliary image superimposing process executed by the control unit 1090 will be described.

  First, control unit 1090 acquires identification information for identifying auxiliary image PD from Table 4 stored in the buffer, and acquires an auxiliary image from storage unit 1030 based on the identification information (step ST1401).

Next, control unit 1090 specifies a specific image PS stored in association with the acquired auxiliary image PD from Table 4 stored in the buffer (step ST1402).
Thereafter, control unit 1090 determines a position outside the predetermined amount from the projection range of identified specific image PS as the projection position (step ST1403).

  Next, control unit 1090 superimposes auxiliary image PD on the projected image based on the determined projection position (step ST1404). Thereafter, the control unit 1090 ends the execution of the auxiliary image superimposing process.

  In the present embodiment, the control unit 1090 has been described as determining the projection position of the instructor image PP and the auxiliary image PD at a position outside the predetermined amount from the projection range of the specific image PS, but there is no limitation to this. .

  For example, the control unit 1090 employs a configuration in which the position at which the specific image PS is determined not to be projected by performing image processing on the first captured image is determined as the projection position of the instructor image PP and the auxiliary image PD. it can.

  In addition, for example, the control unit 1090 performs image processing on the first captured image, determines that it is a margin of a sentence printed on the document 1910, and the part of the document 1910 on which the specific image PS is projected. It is possible to adopt a configuration in which the position that is determined to be absent is determined as the projection position of the instructor image PP and auxiliary image PD.

  Next, with reference to FIG. 18, the guidance image superimposing process executed by the control unit 1090 will be described. FIG. 18 is a flowchart illustrating an example of the guidance image superimposing process executed by the control unit 1090.

  First, control unit 1090 obtains instructor image PP from Table 3 stored in the buffer (step ST1501). Next, control unit 1090 obtains specific image PS associated with the obtained instructor image PP from Table 3 stored in the buffer (step ST1502).

  After that, the control unit 1090 generates a guide image PG that guides the line of sight to the projection position of the acquired instructor image PP from the projection range of the acquired specific image PS (step ST1503). Next, control unit 1090 superimposes guidance image PG based on the obtained projection position of instructor image PP (step ST1504).

  Thereafter, control unit 1090 obtains auxiliary image PD from Table 4 stored in the buffer (step ST1505). Next, control unit 1090 obtains annotation image PA associated with the obtained auxiliary image PD from Table 4 stored in the buffer (step ST1506).

  After that, the control unit 1090 generates a guide image PG that guides the line of sight from the acquired projection position of the annotation image PA to the acquired projection position of the auxiliary image PD (step ST1507). Next, control unit 1090 superimposes guidance image PG based on the obtained projection position of auxiliary image PD (step ST1508). Thereafter, the control unit 1090 ends the execution of the guidance image superimposition process.

  Next, an information acquisition process and an information transmission control process executed by the control unit 1090 will be described with reference to FIG. FIG. 19 is a flowchart illustrating an example of an information acquisition process and an information transmission control process executed by the control unit 1090.

  First, an information acquisition process executed by the control unit 1090 will be described with reference to FIG.

  First, control unit 1090 acquires a first captured image from first imaging device 1200 (step ST1601). Next, control unit 1090 obtains a second captured image from second imaging device 1400 (step ST1602).

  Thereafter, control unit 1090 obtains audio information representing the transmission audio to be transmitted to instruction device 2000 from sound collection device 1600 (step ST1603). Next, control unit 1090 obtains audio information representing the received audio received from instruction device 2000 from receiving unit 1010 (step ST1604). Thereafter, the control unit 1090 ends the execution of the information acquisition process.

  Next, an information transmission control process executed by the control unit 1090 will be described with reference to FIG.

  First, control unit 1090 controls transmission unit 1020 to transmit the first captured image to instruction device 2000 (step ST1701). Next, control section 1090 controls transmission section 1020 to transmit the second captured image to pointing device 2000 (step ST1702).

  Thereafter, control unit 1090 controls transmission unit 1020 to transmit audio information representing transmission audio to instruction apparatus 2000 (step ST1703). Thereafter, the control unit 1090 ends the execution of the information transmission control process.

  Here, returning to FIG. 1, the configuration of the projection instruction system 1 will be described.

  The instruction device 2000 is composed of, for example, a personal computer, and is connected to the communication network 100, the display device 2100, the input device 2200, the sound output device 2500, and the sound collection device 2600.

  The instruction device 2000 displays the portion of the document 1910 to be annotated based on the electronic file representing the document 1910 captured in the received first captured image after receiving the first captured image capturing the document 1910. The display apparatus 2100 is controlled to perform a projection instruction for instructing to project onto the document 1910 the specific image PS that identifies the part of the document 1910 that is controlled to be displayed and the annotation image PA that annotates the part of the document 1910. Send.

  Here, with reference to FIG. 20, the configuration of the pointing device 2000 will be described. FIG. 20 is a functional block diagram illustrating a configuration example of the instruction device 2000.

  The instruction device 2000 includes a reception unit 2010, a transmission unit 2020, a storage unit 2030, and a control unit 2090. The reception unit 2010, the transmission unit 2020, the storage unit 2030, and the control unit 2090 are connected via a bus 2005 so that information can be exchanged.

  Further, the function of the control unit 2090 is realized by software control executed by the instruction device 2000. The hardware configuration of the instruction device 2000 used for executing the software control is the same as the hardware configuration of the control device 1000 described with reference to FIG.

  The receiving unit 2010 is configured by, for example, a network adapter and is connected to the communication network 100. The receiving unit 2010 receives the first captured image obtained by capturing the document 1910 by executing reception processing.

  In addition, the reception unit 2010 receives the following various types of information and notifications by executing reception processing, and outputs the received information to the control unit 2090. Therefore, unless otherwise specified, information output by the receiving unit 2010 to the control unit 2090 is information received by the receiving unit 2010.

  The transmission unit 2020 is configured by, for example, a network adapter and is connected to the communication network 100. The transmission unit 2020 executes a transmission process, whereby the specific image PS that identifies the part of the document 1910 that is controlled to be displayed by the control unit, which will be described later, and the annotation image PA that annotates the part of the document 1910 are sent to the document 1910. A projection instruction for instructing projection is transmitted.

  The transmission unit 1020 acquires the following various information, instructions, notifications, and images from the control unit 2090 and is controlled by the control unit 2090 to transmit the acquired information and the like via the communication network 100. Send to.

  The storage unit 2030 is configured by an external storage device such as a RAM or a hard disk, for example. The storage unit 2030 stores the following various types of information or images.

  The control unit 2090 is connected to the display device 2100, the input device 2200, the sound output device 2500, and the sound collection device 2600. The control unit 2090 controls each unit configuring the instruction device 2000 and each device connected to the instruction device 2000 by executing a control process described later.

  In particular, the control processing executed by the control unit 2090 includes the image display control processing executed to control the display device 2100 to display the image display screen FP described with reference to FIG. 3, and FIG. Document display control processing executed to control the display device 2100 to display the described document display screen FD.

  Here, an image display control process executed by the control unit 2090 will be described with reference to FIG. FIG. 21 is a flowchart illustrating an example of an image display control process executed by the control unit 2090.

  First, control unit 2090 determines whether or not an end instruction (that is, an end command) has been acquired from input device 2200 (step ST1801). If it is determined that an end instruction has been acquired, control unit 2090 ends the execution of the image display control process, and if not, executes the process of step ST1802.

  Although illustration is omitted, the control unit 2090 controls the transmission unit 2020 to transmit the acquired end instruction to the control device 1000 before ending the execution of the image display control process.

  In step ST1801, when it is determined that the end instruction is not acquired, the control unit 2090 determines whether an imaging instruction (that is, an imaging command) is acquired from the input device 2200 (step ST1802).

  In this embodiment, a description of a GUI (Graphical User Interface) for inputting an imaging instruction is omitted, but, for example, in accordance with the annotation instruction unit PCA for inputting a projection instruction described with reference to FIG. The input device 2200 can input an imaging instruction via the GUI for inputting the imaging instruction.

  If it is determined that an imaging instruction has been acquired, control unit 2090 performs the process of step ST1803, and otherwise does the process of step ST1804.

  In step ST1802, when determining that the imaging instruction has been acquired, the control unit 2090 controls the transmission unit 2020 to transmit the acquired imaging instruction to the control apparatus 1000 (step ST1803). Thereafter, control unit 2090 returns to step ST1801 and repeats the above processing.

  In step ST1802, when determining that the imaging instruction has not been acquired, control unit 2090 determines whether or not a range instruction (that is, a range command) has been acquired from input device 2200 (step ST1804). When it is determined that the range instruction has been acquired, control unit 2090 executes the process of step ST1805, and otherwise, executes the process of step ST1806.

  When determining in step ST1804 that the range instruction has been acquired, the control unit 2090 executes range acquisition processing for acquiring the range of the image ID of the document 1910 in the first captured image (step ST1805). Thereafter, control unit 2090 returns to step ST1801 and repeats the above processing. The range acquisition process will be described later.

  In step ST1804, when it is determined that the range instruction has not been acquired, control unit 2090 determines whether or not a projection instruction (that is, a projection command) has been acquired from input device 2200 (step ST1806). If it is determined that a projection instruction has been acquired, control unit 2090 performs the process of step ST1807; otherwise, it executes the process of step ST1808.

  In step ST1806, when the control unit 2090 determines that the projection instruction has been acquired, the control unit 2090 executes a first instruction transmission control process for controlling the transmission unit 2020 to transmit the projection instruction related to the document annotation image or the target annotation image. (Step ST1807). Thereafter, control unit 2090 returns to step ST1801 and repeats the above processing. The first instruction transmission control process will be described later.

  In step ST1806, when it is determined that the projection instruction has not been acquired, the control unit 2090 acquires information indicating information representing an image displayed on the display device 2100 and information representing sound output from the sound output device 2500. Processing is executed (step ST1808). The information acquisition process will be described later.

  Next, the control unit 2090 controls the display device 2100 to display an image represented by the acquired information, and executes a device control process for controlling the sound output device 2500 to output the sound represented by the acquired information ( Step ST1809). The device control process will be described later.

  After that, the control unit 2090 controls the transmission unit 2020 to transmit, to the control device 1000, information representing the transmission voice to be transmitted to the control device 1000 from the sound collecting device 2600. ST1810). Thereafter, control unit 2090 returns to step ST1801 and repeats the above processing.

  Here, the range acquisition process executed by the control unit 2090 will be described with reference to FIG. FIG. 22 is a flowchart illustrating an example of a range acquisition process executed by the control unit 2090.

  First, the control unit 2090 controls the display device 2100 to start displaying the four icons IC1 to IC4 described with reference to FIG. 3 and the determination button BT15 (step ST1901).

  Next, control unit 2090 acquires position information representing the initial display position of each icon IC1 to IC4 (step ST1902). Next, control unit 2090 determines whether or not a determination instruction (that is, a determination command) has been acquired from input device 2200 (step ST1903). When it is determined that the determination instruction has been acquired, control unit 2090 executes the process of step ST1906, and otherwise, executes the process of step ST1904.

  When determining in step ST1903 that the determination instruction is not acquired, the control unit 2090 acquires the amount of movement of the icons IC1 to 4 from the input device 2200 (step ST1904).

  Next, the control unit 2090 changes the position information of the icons IC1 to 4 based on the acquired movement amount, and controls the display device 2100 to display the icons IC1 to 4 on the changed position information (step ST1905). . Thereafter, control unit 2090 returns to step ST1903 and repeats the above processing.

  In step ST1903, when the control unit 2090 determines that the determination instruction has been acquired, the range of the image ID of the document 1910 in the first captured image (hereinafter simply referred to as the document range) based on the position information of the icons IC1 to IC4. (Step ST1906).

  Thereafter, control unit 2090 controls display device 2100 to end the display of four icons IC1 to IC4 and determination button BT15 (step ST1907). Thereafter, the control unit 2090 ends the execution of the range acquisition process.

  Next, the first instruction transmission control process executed by the control unit 2090 will be described with reference to FIG. FIG. 23 is a flowchart illustrating an example of a first instruction transmission control process executed by the control unit 2090.

  First, the first instruction transmission control process executed by the control unit 2090 will be described with reference to FIG.

  First, control unit 2090 executes annotation image drawing processing for drawing annotation image PA instructed to be projected by the projection instruction (step ST2001). The annotation image drawing process will be described later with reference to FIG.

  Next, control unit 2090 superimposes annotation image PA on the first captured image based on the position information representing the annotation position acquired when the annotation image drawing process is executed in step ST2001 (step ST2002).

  Thereafter, control unit 2090 controls display device 2100 to display the superimposed first captured image (step ST2003). Next, control unit 2090 determines whether or not to transmit a projection instruction (step ST2004).

  As a specific example, a case where the input device 2200 is configured with a keyboard will be described. The control unit 2090 determines to transmit a projection instruction when it acquires a signal output from the input device 2200 when the instructor presses the “Enter” key. Conversely, when the control unit 2090 obtains a signal indicating a cancel instruction output from the input device 2200 when the instructor presses the “Esc” key, the control unit 2090 determines that the projection instruction is not transmitted.

  As another specific example, a case where the input device 2200 is configured with a mouse will be described. When the control unit 2090 obtains a signal indicating a cancellation instruction output by the input device 2200 when the pointer performs a click operation or the like after the pointer PT1 described with reference to FIG. It is determined not to send the instruction.

  If it is determined that a projection instruction is to be transmitted, control unit 2090 performs the process of step ST2005, and otherwise, executes the process of step ST2007.

  If it is determined in step ST2004 that a projection instruction is to be transmitted, control unit 2090 executes a projection instruction generation process for generating a projection instruction (step ST2005). The projection instruction generation process will be described later. Next, control unit 2090 controls transmission unit 2020 to transmit a projection instruction (step ST2006).

  In step ST2004, when it is determined that the projection instruction is not transmitted, or after executing step ST2006, control unit 2090 controls display device 2100 to display the first captured image before superimposition (step ST2007). Thereafter, the control unit 2090 ends the execution of the first instruction transmission control process.

  Next, an annotation image drawing process executed by the control unit 2090 will be described with reference to FIG.

  First, the control unit 2090 acquires information representing the annotation image PA from the input device 2200 (step ST2101). Next, control unit 2090 obtains position information representing the annotation position from input device 2200 (step ST2102).

Thereafter, control unit 2090 draws annotation image PA based on the information representing annotation image PA (step ST2103). Next, control unit 2090 determines whether or not the document range has been acquired (step ST2104). If it is determined that the document range has been acquired, control unit 2090 executes the process of step ST2105, and otherwise, executes the process of step ST2106.

  If it is determined in step ST2104 that the document range has been acquired, the control unit 2090 generates target information based on the information indicating the annotation position and the information indicating the document range (step ST2105).

  Specifically, the control unit 2090 generates target information indicating that the annotation target is the document 1910 when the annotation position is included in the document range. If not, the annotation target is the target object. Object information representing 1920 is generated.

  After executing Step ST2105, the control unit 2090 ends the execution of the annotation image generation process.

  If it is determined in step ST2104 that the document range has not been acquired, the control unit 2090 sets the target information as information indicating that the annotation target is the target object 1920 (step ST2106). Thereafter, the control unit 2090 ends the execution of the annotation image generation process.

  Next, a projection instruction generation process executed by the control unit 2090 will be described with reference to FIG. FIG. 24 is a flowchart illustrating an example of a projection instruction generation process executed by the control unit 2090.

  First, control unit 2090 determines whether or not information representing annotation image PA has been acquired from input device 2200 (step ST2201). The control unit 2090 executes the process of step ST2202 when determining that the information representing the annotation image PA has been acquired, and otherwise executes the process of step ST2204.

  When determining in step ST2201 that information representing the annotation image PA has been acquired, the control unit 2090 adds information representing the annotation image PA to the projection instruction (step ST2202).

  Next, control unit 2090 adds position information representing the annotation position to the projection instruction as position information representing the projection position (step ST2203). Thereafter, control unit 2090 adds the target information to the projection instruction (step ST2204).

  If it is determined in step ST2201 that information representing the annotation image PA is not acquired, or after executing the process of step ST2204, the control unit 2090 determines whether the document range has already been acquired (step ST2205). ). If it is determined that the document range has already been acquired, control unit 2090 ends the process of step ST2206, and otherwise ends the projection instruction generation process.

  If it is determined in step ST2205 that the document range has already been acquired, the control unit 2090 determines the projection range of the specific image PS based on the document range and the display range, and represents the determined projection range. The range information is added to the projection instruction (step ST2206).

  Next, control unit 2090 adds the instructor identification information to the projection instruction (step ST2207). Thereafter, control unit 2090 adds the display magnification to the projection instruction (step ST2208). Thereafter, the control unit 2090 ends the execution of the projection instruction generation process.

  Next, with reference to FIG. 25, information acquisition processing and device control processing executed by the control unit 2090 will be described. FIG. 25 is a flowchart illustrating an example of information acquisition processing and device control processing executed by the control unit 2090.

  First, an information acquisition process executed by the control unit 2090 will be described with reference to FIG.

  First, control unit 2090 obtains the first captured image from receiving unit 2010 (step ST2301). Next, control unit 2090 obtains the second captured image from receiving unit 2010 (step ST2302).

  Thereafter, control unit 2090 obtains voice information representing the voice or the like uttered by the instructor from transmission device 2600 and representing the transmission voice to be transmitted to control apparatus 1000 (step ST2303).

  Next, control unit 2090 obtains voice information representing the received voice received from control apparatus 1000 and received from control section 1000 from reception section 2010 (step ST2304). Thereafter, the control unit 2090 ends the execution of the information acquisition process.

  Next, with reference to FIG. 25B, a device control process executed by the control unit 2090 will be described.

  First, control unit 2090 controls display device 2100 to display the first captured image (step ST2401). Next, control unit 2090 controls display device 2100 to display the second captured image (step ST2402).

  Thereafter, control unit 2090 controls sound output device 2500 to output the received voice (step ST2403). Thereafter, the control unit 2090 ends the execution of the device control process.

  Next, a document display control process executed by the control unit 2090 to display the document display screen FD described with reference to FIG. 4 will be described with reference to FIG. FIG. 26 is a flowchart illustrating an example of a document display control process executed by the control unit 2090.

  First, control unit 2090 executes an initialization process in which information used in the initial execution of the document display process is predetermined information (step ST2501). The initialization process will be described later.

  Next, control unit 2090 acquires identification information (hereinafter simply referred to as file identification information) for identifying an electronic file (hereinafter simply referred to as a document file) representing document 1910 to be processed (step ST2502).

  In the present embodiment, the control unit 2090 is described as acquiring file identification information from the input device 2200, but the present invention is not limited to this.

  In this embodiment, the file identification information is described as a URL (Uniform Resource Locator) that describes the location of the file. However, the present invention is not limited to this, and the file identification information includes the file name. A configuration that is information to be represented can be employed.

  Thereafter, control unit 2090 determines whether or not an end instruction has been acquired from input device 2200 (step ST2503). If it is determined that an end instruction has been acquired, control unit 2090 ends the execution of the document display control process, and if not, executes the process of step ST2504.

  In step ST2503, when it is determined that the end instruction has not been acquired, control unit 2090 determines whether or not a range change instruction (that is, a range change command) has been acquired from input device 2200 (step ST2504). . If it is determined that a range change instruction has been acquired, control unit 2090 performs the process of step ST2505, and otherwise does the process of step ST2506.

  If it is determined in step ST2504 that a range change instruction has been acquired, the control unit 2090 changes the portion of the document 1910 displayed on the document display screen FD and also instructs to change the projection range of the specific image PS. A range change control process for controlling to transmit a projection instruction is executed (step ST2505).

  The range change control process will be described later. Thereafter, control unit 2090 returns to step ST2503 and repeats the above processing.

  If it is determined in step ST2504 that the range change instruction has not been acquired, control unit 2090 determines whether or not a magnification change instruction (that is, a magnification change command) has been acquired from input device 2200 (step ST2506). ). If it is determined that the magnification change instruction has been acquired, control unit 2090 executes the process of step ST2507, and otherwise, executes the process of step ST2508.

  If it is determined in step ST2506 that the magnification change instruction has been acquired, the control unit 2090 changes the display magnification of the document display screen FD and changes the projection range of the specific image PS in accordance with the change of the display magnification. A magnification change control process for controlling to transmit a projection instruction for instructing to execute is executed (step ST2507).

  The magnification change control process will be described later. Thereafter, control unit 2090 returns to step ST2503 and repeats the above processing.

  If it is determined in step ST2506 that a magnification change instruction has not been acquired, control unit 2090 determines whether or not a projection instruction (that is, a projection command) has been acquired from input device 2200 (step ST2508). If it is determined that a projection instruction has been acquired, control unit 2090 performs the process of step ST2509, and otherwise does the process of step ST2510.

  If it is determined in step ST2508 that a projection instruction has been acquired, the control unit 2090 displays the document annotation image on the document display screen FD and controls the transmission unit 2020 to transmit the projection instruction related to the document annotation image. The second instruction transmission control process is executed (step ST2509).

  The second instruction transmission control process will be described later. Thereafter, control unit 2090 returns to step ST2503 and repeats the above processing.

  If it is determined in step ST2508 that a projection instruction has not been acquired, control unit 2090 executes partial image display control processing for controlling display device 2100 to display a partial image (step ST2510).

  The partial image display control process will be described later. Thereafter, control unit 2090 returns to step ST2503 and repeats the above processing.

  Next, an initialization process executed by the control unit 2090 will be described with reference to FIG. FIG. 27 is a flowchart illustrating an example of an initialization process executed by the control unit 2090.

  First, the control unit 2090 acquires the instructor identification information from the input device 2200 or the storage unit 2030 that stores environment variables used by, for example, an OS (Operating System) (step ST2601). Next, control unit 2090 sets the display magnification to a predetermined magnification (step ST2602).

  In the present embodiment, the predetermined magnification is described as “100%”, but the present invention is not limited to this. For example, a configuration in which the predetermined magnification is a magnification that can display the entire document 1910 can be employed.

  Thereafter, control unit 2090 sets the display range of document 1910 as a predetermined range (step ST2603).

  In the present embodiment, the predetermined range is described as being a part of the upper left of the document 1910, but is not limited thereto. For example, a configuration in which a predetermined range is the entire range of the document 1910 can be employed.

  Next, control unit 2090 deletes annotation image PA stored in storage unit 2030 and position information representing the annotation position (step ST2604). Thereafter, control unit 2090 executes partial image display processing based on the display magnification and display range set in steps ST2602 and ST2603 (step ST2605). Thereafter, the control unit 2090 ends the initialization process.

  Next, the partial image display control process executed by the control unit 2090 will be described with reference to FIG. FIG. 28 is a flowchart illustrating an example of the partial image display control process executed by the control unit 2090.

  First, control unit 2090 obtains an electronic file representing document 1910 (step ST2701).

  In the present embodiment, the control unit 2090 can adopt a configuration in which an electronic file is acquired by downloading based on file identification information. Specifically, the control unit 2090 acquires information such as an IP address for identifying the device storing the electronic file on the communication network 100 from the file identification information such as URL.

  Next, the control unit 2090 controls the transmission unit 2020 to transmit a request for transmitting the electronic file to a device that stores the electronic file based on the acquired information. Thereafter, the control unit 2090 obtains the requested electronic file from the receiving unit 2010 that has received the transmitted electronic file.

  In the present embodiment, the storage unit 2030 can store an electronic file identified by file identification information in advance, and the control unit 2090 can employ a configuration in which the electronic file is acquired from the storage unit 2030 based on the file identification information.

  After executing step ST2701, the control unit 2090 generates a document image of the document 1910 to be processed based on the acquired electronic file (step ST2702). In particular, when a page number is designated, a document image of a document corresponding to the designated page is generated.

Next, control unit 2090 obtains a partial image corresponding to the display range from the generated document image (step ST2703). Next, control unit 2090 determines whether or not there is an annotation image PA for annotating the display range (step ST2704). If it is determined that there is an annotation image PA for annotating the display range, control unit 2090 performs the process of step ST2705, and otherwise, executes the process of step ST2706.

  If it is determined in step ST2704 that there is an annotation image PA that annotates the display range, the control unit 2090 superimposes the annotation image PA on the partial image based on the annotation position stored in the storage unit 2030 (step ST2705). ).

  In step ST2704, when it is determined that there is no annotation image PA for annotating the display range, or after executing step ST2705, the control unit 2090 controls the display device 2100 to display the partial image (step ST2706). . Thereafter, the control unit 2090 ends the execution of the partial image display control process.

  Next, the range change control process and the magnification change control process executed by the control unit 2090 will be described with reference to FIG. FIG. 29 is a flowchart illustrating an example of a range change control process and a magnification change control process executed by the control unit 2090.

  First, the range change control process executed by the control unit 2090 will be described with reference to FIG.

  First, control unit 2090 obtains the changed display range from the range change instruction (step ST2801). Next, control unit 2090 executes the partial image display control process based on the acquired display range (step ST2802).

  Thereafter, control unit 2090 executes the projection instruction generation process described above in order to generate a projection instruction for instructing to project specific image PS that specifies the changed display range (step ST2803).

  Next, control unit 2090 controls transmission unit 2020 to transmit the generated projection instruction (step ST2804). Thereafter, the control unit 2090 ends the execution of the range change control process.

  Next, with reference to FIG. 29B, the magnification change control process executed by the control unit 2090 will be described.

  First, control unit 2090 obtains the changed display magnification from the magnification change instruction (step ST2901). Next, control unit 2090 changes the display range based on the changed display magnification (step ST2902). Next, control unit 2090 executes the processing from step ST2903 to step ST2905 (steps ST2903 to 2905). Thereafter, the control unit 2090 ends the execution of the magnification change control process.

  Note that the processing in steps ST2903 to 2905 is the same as the processing in steps ST2802 to 2804 described with reference to FIG.

  Next, the second instruction transmission control process executed by the control unit 2090 will be described with reference to FIG. FIG. 30 is a flowchart illustrating an example of the second instruction transmission control process executed by the control unit 2090.

  First, the control unit 2090 executes the above-described annotation image drawing process for drawing the annotation image PA instructed to be projected by the projection instruction (step ST3001). Next, control unit 2090 associates the drawn annotation image PA with the position information indicating the annotation position, and stores them in storage unit 2030 (step ST3002).

  Thereafter, control unit 2090 executes the partial image display control process for controlling to display the partial image on which annotation image PA stored in storage unit 2030 is superimposed (step ST3003).

  Next, control unit 2090 determines whether or not to transmit a projection instruction (step ST3004). If it is determined that a projection instruction is to be transmitted, control unit 2090 performs the process of step ST3005, and otherwise does the process of step ST3007.

  When determining in step ST3004 that the projection instruction is to be transmitted, the control unit 2090 executes the projection instruction generation process for generating the projection instruction (step ST3005).

  Thereafter, control unit 2090 controls transmission unit 2020 to transmit a projection instruction (step ST3006). Next, the control unit 2090 ends the execution of the second instruction transmission control process.

  If it is determined in step ST3004 that a projection instruction is not transmitted, control unit 2090 deletes annotation image PA drawn in step ST3001 and position information of annotation image PA from storage unit 1030 (step ST3007).

  Next, after deleting annotation image PA, control unit 2090 executes the above partial image display control process for controlling to display the partial image on which annotation image PA stored in storage unit 1030 is superimposed (step ST3008). . Thereafter, the control unit 2090 ends the execution of the second instruction transmission control process.

  Here, returning to FIG. 1, the configuration of the projection instruction system 1 will be described.

  The communication network 100 is configured by, for example, a LAN (Local Area Network), a WAN (Wide Area Network), a MAN (Metropolitan Area Network), or a public line network, and connects the control device 1000 and the instruction device 2000 so that they can communicate with each other. .

  In the present embodiment, the projection device 1100 is controlled by the control device 1000, and the only specific image PS for specifying the portion to be annotated in the annotation image PA for which the only instructor instructs the projection, and the only instruction. However, the present invention is not limited to this.

  For example, the projection device 1100 may be configured to project a plurality of specific images PS related to a plurality of instructions and a plurality of instruction images related to a plurality of instructions under the control of the control device 1000.

  Here, an embodiment in which a plurality of specific images PS are projected will be described with reference to FIG. FIG. 31 is a diagram illustrating an embodiment in which a plurality of specific images PS are projected.

  The document 1910 and the projection screen 1990 shown in FIG. 31 project the instructor image PP1, the specific image PS1, the guidance image PG11, and other images described with reference to FIG.

  Further, the document 1910 and the projection screen 1990 shown in FIG. 31 are projected with the instructor image PP4, the specific image PS4, and the guide image PG41.

  Note that the instructor image PP4 is an image representing information related to an instructor different from the instructor related to the instructor image PP1. The specific image PS4 is an image that specifies a part of the document 1910 to be displayed to different instructors and also specifies a part to be annotated by the annotation image PA instructed to be projected by a different instructor. Further, the auxiliary image PD41 is an image that guides the line of sight from the specific image PS4 to the instructor image PP1.

  In this embodiment, the receiving unit 1010 corresponds to a receiving unit included in the control device and the control system, the transmitting unit 1020 corresponds to a transmitting unit included in the control device, and the control unit 1090 corresponds to a control unit included in the control device. The receiving unit 2010 corresponds to the receiving unit included in the pointing device, the transmitting unit 2020 corresponds to the transmitting unit included in the pointing device, the control unit 2090 corresponds to the control unit included in the pointing device, and the projection device 1100 corresponds to the control system. The first imaging device 1200 corresponds to the imaging means included in the control system.

  In the second embodiment, an instruction device that transmits a notification instruction to notify that a document to be annotated has been changed, and notification that the document to be annotated has been changed in accordance with the notification instruction A projection instruction system including a control device that controls the projection device to project a notification image will be described.

  The instruction device described in the second embodiment is also a device that transmits an image representing another document to be annotated displayed on the display device, and the control device described in the second embodiment is an annotation device. It is also an apparatus that controls the projection apparatus to project an image representing another document that has been changed as a target of the above.

  Since the projection instruction system described in the second embodiment has substantially the same configuration and function as the projection instruction system described in the first embodiment, the differences will be mainly described below and the differences will not be described. Omitted. Further, for convenience of explanation, the device numbers and the like will be described using the same numbers as the device numbers used in the first embodiment.

  First, a document display screen FD displayed on the display device 2100 in the second embodiment will be described with reference to FIG. FIG. 32 is a diagram illustrating an example of a document display screen FD displayed on the display device 2100 in the second embodiment.

  Unlike the first embodiment, the display change unit PAS displayed on the document display screen FD displays the document change buttons BT28 and 29 instead of displaying the magnification change buttons BT26 and 27.

  The document change buttons BT28 and 29 are a document 1910 represented by an image displayed on the document display unit PSD, and input a document change instruction (hereinafter also simply referred to as a document change command) for changing the document 1910 to be annotated. This is a button display indicating that 2200 can be input.

  The button BT26 is a button display indicating that the input device 2200 can input an instruction to change to another document corresponding to the previous page, and the button BT27 can input an instruction to change to another document corresponding to the previous page. The document change instruction is included in the change instruction.

  In the present embodiment, the display device 2100 will be described as performing button display indicating that an instruction to change to another document corresponding to the previous page or the next page can be input. However, the present invention is not limited to this. 2100 can employ a configuration for displaying a button indicating that an instruction to change to another document represented by another electronic file can be input.

  Next, a notification image projected by the projection apparatus 1100 and an image representing another document changed as an annotation target will be described with reference to FIG. FIG. 33 is a diagram illustrating an example of a notification image projected by the projection apparatus 1100 and an image representing another document.

  As shown in FIG. 33, the projection device 1100 projects a notification image PAN1 that notifies that a document to be annotated has been changed. The projection device 1100 projects an image (hereinafter also referred to as a thumbnail image) PSM representing another document that has been changed as an annotation target.

  In the present embodiment, the image PSM is described as a thumbnail representing another changed document in a reduced scale, but the present invention is not limited to this. For example, the image PSM is an image that represents an enlarged view of another document, a structure that is an image that represents all or part of a sentence or an image included in another document, and other documents such as a document name or a title. A configuration that is an image representing document identification information to be identified or a configuration that is an image representing file identification information of an electronic file representing another document can be adopted.

  Hereinafter, the receiving unit 1010 and the control unit 1090 constituting the control device 1000 according to the second embodiment will be described.

  Unlike the first embodiment, the receiving unit 1010 further receives a notification instruction instructing to notify that the document 1910 to be annotated has been changed by executing the reception process, and has changed the object to be annotated. An image PSM representing another document is further received. Note that, unlike the first embodiment, the receiving unit 1010 does not receive an imaging instruction.

  Unlike the first embodiment, the control unit 1090 executes the control process to project the notification image PAN1 that notifies that the document 1910 to be annotated has been changed according to the notification instruction received by the reception unit 1010. The projector 1100 is further controlled and the projector 1100 is further controlled to project the received image PSM. Note that, unlike the first embodiment, the control unit 1090 does not control the imaging position of the second imaging device based on the imaging instruction.

  Here, with reference to FIG. 34, a control process executed by the control unit 1090 in the second embodiment will be described. FIG. 34 is a flowchart illustrating an example of a control process executed by the control unit 1090 in the second embodiment.

  First, control section 1090 executes the processing of steps ST3101 to ST3103 (steps ST3101 to ST3103). Note that the processing of steps ST3101 to ST3103 is the same as the processing of steps ST0101 to ST0103 described with reference to FIG.

  When determining in step ST3103 that the end instruction has not been acquired, control section 1090 determines whether or not a notification instruction (hereinafter also referred to as a notification command) has been acquired from receiving section 1010 (step ST3104). If it is determined that the notification instruction has been acquired, control section 1090 executes the process of step ST3105, and otherwise, executes the process of step ST3109.

  When determining in step ST3104 that the notification instruction has been acquired, the control unit 1090 superimposes the notification image PAN1 on the projection image (step ST3105).

  In this embodiment, the storage unit 1030 stores the notification image PAN1 in advance, and the control unit 1090 is described as acquiring the notification image PAN1 from the storage unit 1030, but is not limited thereto. The control unit 1090 can employ a configuration that generates a notification image PAN1 representing a character string representing the content notified by the notification instruction.

  Next, control unit 1090 obtains thumbnail image PSM of another changed document from receiving unit 1010 (step ST3106). Thereafter, control unit 1090 superimposes the acquired thumbnail image on the projection image (step ST3107).

  Next, control section 1090 controls projection apparatus 1100 to project the superimposed projection image (step ST3108). Thereafter, control unit 1090 returns to step ST3103 and repeats the above processing.

  In Step ST3104, when it is determined that the notification instruction has not been acquired, the control unit 1090 executes the processing from Step ST3109 to Step ST3114 (Step ST3109 to Step ST3114). Note that the processing from step ST3109 to step ST3115 is the same as the processing from step ST0107 to ST0112 described with reference to FIG.

  In this embodiment, the detection unit 1040 detects a document range representing the range of the document 1910 in the first captured image, and the control unit 1090 determines that the notification image PAN1 is the document 1910 based on the document range detected by the detection unit 1040. It is possible to employ a configuration in which the notification image PAN1 is superimposed on a position in the projected image projected onto the image and the thumbnail image PSM is superimposed on a position projected off the document 1910.

  Hereinafter, the transmission unit 2020 and the control unit 2090 constituting the instruction device 2000 according to the second embodiment will be described.

  Unlike the first embodiment, the transmission unit 2020 further transmits a notification instruction instructing that the document 1910 to be annotated has been changed by executing a transmission process, and the display device 2100 displays the notification instruction. An image PSM representing another document to be annotated is transmitted.

  Unlike the first embodiment, the control unit 2090 executes the document display control process, and based on an electronic file representing another document different from the document 1910 captured in the first captured image received by the reception unit 2010. The display device 2100 is controlled to change the display to the part of another document to be annotated.

  Here, with reference to FIG. 35, a document display control process executed by the control unit 2090 in the second embodiment will be described. FIG. 35 is a flowchart illustrating an example of a document display control process executed by the control unit 2090 in the second embodiment.

  First, control unit 2090 executes the processes of steps ST3201 to ST3205 (steps ST3201 to ST3205). The processing of steps ST3201 to ST3205 is the same as the processing of steps ST2501 to ST2505 described with reference to FIG.

  When determining in step ST3204 that the range change instruction has not been acquired, the control unit 2090 determines whether or not a document change instruction (that is, a document change command) has been acquired from the input device 2200 (step ST3206). If it is determined that the document change instruction has been acquired, control unit 2090 performs the process of step ST3207, and otherwise, executes the process of step ST3209.

  If it is determined in step ST3206 that a document change instruction has been acquired, the control unit 2090 executes a document change control process for controlling to change the document to be displayed to another document (step ST3207). The document change control process will be described later.

  Next, control section 2090 executes notification transmission control processing for controlling transmission section 2020 to transmit a notification instruction (step ST3208). The notification transmission control process will be described later. Thereafter, control unit 2090 returns to step ST3203 and repeats the above processing.

  If it is determined in step ST3206 that the document change instruction has not been acquired, control section 2090 executes the processing of steps ST3209 to 3211 (steps ST3209 to 3211). Note that the processing in steps ST3209 to 3211 is the same as the processing in steps ST2508 to ST2510 described with reference to FIG.

  Here, the document change control process and the notification transmission control process executed by the control unit 2090 will be described with reference to FIG. FIG. 36 is a flowchart illustrating an example of document change control processing and notification transmission control processing executed by the control unit 2090.

  First, a document change control process executed by the control unit 2090 will be described with reference to FIG.

  First, control unit 2090 obtains file identification information of an electronic file representing another document after change (previous or subsequent page) (step ST3301). The file identification information does not change before and after the change, but if only the page number changes, the page number is also acquired.

  Thereafter, control unit 2090 executes the above-described initialization process for controlling to initially display the document identified by the acquired file identification information (and page number) (step ST3202). Next, the control unit 2090 ends the execution of the document change control process.

  Next, the notification transmission control process executed by the control unit 2090 will be described with reference to FIG.

  First, control unit 2090 generates a notification instruction for instructing notification of a document change (step ST3401). Next, control unit 2090 controls transmission unit 2020 to transmit the generated notification instruction (step ST3402).

  Thereafter, control unit 2090 generates thumbnail image SNM based on the electronic file identified by the file identification information (step ST3403). Next, control unit 2090 controls transmission unit 2020 to transmit the generated thumbnail image (step ST3404). Thereafter, the control unit 2090 ends the execution of the notification transmission control process.

  In this embodiment, the receiving unit 1010 corresponds to a receiving unit included in the control device and the control system, the transmitting unit 1020 corresponds to a transmitting unit included in the control device, and the control unit 1090 corresponds to a control unit included in the control device. The receiving unit 2010 corresponds to the receiving unit included in the pointing device, the transmitting unit 2020 corresponds to the transmitting unit included in the pointing device, the control unit 2090 corresponds to the control unit included in the pointing device, and the projection device 1100 corresponds to the control system. The first imaging device 1200 corresponds to the imaging means included in the control system.

  In the third embodiment, a change in the position of the document imaged in the first captured image is detected, and an annotation image and an annotation image for annotating a part of the document are annotated based on the detected change in the position of the document. A projection instruction system including a control device that controls the projection device so as to change the position of projecting a specific image that identifies a portion of a document to be described will be described.

  The projection instruction system described in the third embodiment has almost the same configuration and function as the projection instruction system described in the first embodiment. Therefore, the differences will be mainly described below, and the differences will not be described. Omitted. Further, for convenience of explanation, the device numbers and the like will be described using the same numbers as the device numbers used in the first embodiment.

  First, with reference to FIG. 37, an image projected by the projection apparatus 1100 in the third embodiment will be described. FIG. 32 is a diagram illustrating an example of an image projected by the projection device 1100 according to the third embodiment.

  First, an image projected by the projection apparatus 1100 before the position of the document 1910 changes will be described with reference to FIG.

  FIG. 37A shows an image projected by the projection apparatus 1100 before the position of the document 1910 is changed. Note that the image and document 1910 shown in FIG. 37A are the same as the image and document 1910 described with reference to FIG.

  Next, an image projected by the projection apparatus 1100 after the position of the document 1910 has changed will be described with reference to FIG.

  The position of the document 1910 shown in FIG. 37B is moved by “x0” in the X-axis direction and rotated counterclockwise by an angle “θ” from the position of the document 1910 shown in FIG. Position.

  The specific image PS3 and the annotation images PA31 and 32 shown in FIG. 37B are moved by “x0” in the X-axis direction based on the change in the position of the document 1910, and only the angle “θ” is counterclockwise. Projected to the rotated position.

  That is, the relative projection positions of the specific image PS3 and the annotation images PA31 and 32 shown in FIG. 37B with respect to the document 1910 do not change before and after the position of the document 1910 changes.

  Therefore, even after the position of the document 1910 has changed, the relative projection position with respect to the document 1910 instructed to be projected by the projection instruction does not change.

  Hereinafter, the detection unit 1040 and the control unit 1090 constituting the control device 1000 according to the third embodiment will be described.

  Unlike the first embodiment, the detection unit 1040 detects a change in the position of the document 1910 captured in the first captured image by executing a detection process.

  Here, with reference to FIG. 38, the detection process which the detection part 1040 performs in Example 3 is demonstrated. FIG. 38 is a flowchart illustrating an example of detection processing executed by the detection unit 1040 in the third embodiment.

  First, the detection unit 1040 executes the processes of steps ST3501 and ST3502 (steps ST3501 and ST3502). The processes in steps ST3501 and ST3502 are the same as the processes in steps ST3501 and ST3502 described with reference to FIG.

  After executing step ST3502, the detection unit 1040 determines whether an image of the document 1910 has been detected (step ST3503). If the detection unit 1040 determines that an image of the document 1910 has been detected, the detection unit 1040 ends the process of step ST3504, and otherwise ends the detection process.

  In step ST3503, the detection unit 1040 detects position information representing the position of the vertex image of the document 1910 in the first captured image (step ST3504). Next, detection section 1040 outputs the detected position information to control section 1090 (step ST3505).

  After that, the detection unit 1040 calculates a difference from the vertex position of the document 1910 represented by the previously detected position information (step ST3506). Next, detection unit 1040 outputs the calculated amount of change in position to control unit 1090 (step ST3507). Thereafter, the detection unit 1040 ends the execution of the detection process.

  Unlike the first embodiment, the control unit 1090 executes control processing, so that an annotation image PA and an annotation image PA for annotating a part of the document 1910 are displayed based on a change in the position of the document 1910 detected by the detection unit 1040. The projection apparatus 1100 is controlled so as to change the position where the specific image PS that specifies the part of the document 1910 to be annotated is projected.

  In the third embodiment, the control process executed by the control unit 1090 is the same as the control process described with reference to FIG. 10 in the first embodiment.

  However, in the third embodiment, the projection image generation process executed in step ST0108 of the control process includes the process of determining the coefficient of the coordinate conversion function used for converting the position coordinates of the specific image PS or the like. 1 is different from the projection image generation processing described with reference to FIG.

  Here, with reference to FIG. 39, a projection image generation process executed by the control unit 1090 in the third embodiment will be described. FIG. 39 is a flowchart illustrating an example of a projection image generation process executed by the control unit 1090 in the third embodiment.

  First, control unit 1090 executes the processes of steps ST3601 to 3605 (steps ST3601 to 3605). Note that the processing in steps ST3601 to 3605 is the same as the processing in steps ST0301 to 0305 described with reference to FIG.

  In step ST3605, when it is determined that the detection unit 1040 has detected the image ID of the document 1910, the control unit 1090 acquires the amount of change in the position of the document 1910 from the detection unit 1040 (step ST3606).

  Next, control section 1090 determines the coefficient of the coordinate conversion function based on the obtained change amount (step ST3607). In the present embodiment, the coordinate transformation function is described as an affine transformation function, but is not limited to this.

  After that, control section 1090 executes the processing of steps ST3608 to 3614 (steps ST3608 to 3614). Thereafter, the execution of the projection image generation process is terminated. Note that the processing in steps ST3608 to 3614 is substantially the same as the processing in steps ST0306 to 0312, and thus detailed description thereof is omitted.

  In addition, the document annotation image superimposing process and the specific image superimposing process executed by the control unit 1090 in steps ST3609 and ST3610 superimpose the annotation image PA and the specific image PS on the position converted using the coordinate conversion function. In Example 1, it differs from the process demonstrated with reference to FIG.

  Therefore, with reference to FIG. 40, the document annotation image superimposing process and the specific image superimposing process executed by the control unit 1090 in the third embodiment will be described. FIG. 40 is a flowchart illustrating an example of the document annotation image superimposing process and the specific image superimposing process executed by the control unit 1090 in the third embodiment.

  First, with reference to FIG. 40A, a document annotation image superimposing process executed by the control unit 1090 in the third embodiment will be described.

  First, control unit 1090 executes the processes of steps ST3701 and ST3702 (steps ST3701 and ST3702). Note that the processing in steps ST3701 and ST3702 is the same as the processing in steps ST1101 and ST1102 described with reference to FIG.

  After executing step ST3702, the control unit 1090 converts the document annotation image and the position information of the document annotation image using the coordinate conversion function for which the coefficient has been determined (step ST3703).

  Thereafter, control unit 1090 executes the process of step ST3704 (step ST3704). Next, the control unit 1090 ends the execution of the document annotation image superimposition process. Note that the processing in step ST3704 is the same as the processing in step ST1103 described with reference to FIG.

  Next, with reference to FIG. 40B, the specific image superimposing process executed by the control unit 1090 in the third embodiment will be described.

  First, control unit 1090 executes the processes of steps ST3801 and ST3802 (steps ST3801 and ST3802). Note that the processing in steps ST3801 and ST3802 is the same as the processing in steps ST1201 and ST1202 described with reference to FIG.

  After executing step ST3802, the control unit 1090 converts the specific image PS and the range information of the specific image PS using the coordinate conversion function for which the coefficient has been determined (step ST3803).

  Thereafter, control unit 1090 executes the process of step ST3804 (step ST3804). Next, the control unit 1090 ends the execution of the specific image superimposing process. Note that the processing in step ST3804 is the same as the processing in step ST1203 described with reference to FIG.

  In the present embodiment, the receiving unit 1010 corresponds to a receiving unit included in the control device and the control system, the transmitting unit 1020 corresponds to a transmitting unit included in the control device, and the detecting unit 1040 corresponds to a detecting unit included in the control device. The control unit 1090 corresponds to the control unit included in the control device, the reception unit 2010 corresponds to the reception unit included in the instruction device, the transmission unit 2020 corresponds to the transmission unit included in the instruction device, and the control unit 2090 corresponds to the instruction device. The projection apparatus 1100 corresponds to the projection means included in the control system, and the first imaging apparatus 1200 corresponds to the imaging means included in the control system.

  In the fourth embodiment, a control device that transmits a notification that notifies a change in the image of a document in a captured image, and a display device that controls display of a notification image that notifies a change in the image of the document based on the notification. A projection instruction system including an instruction device that performs the above will be described.

  In the present embodiment, the change in the image of the document in the captured image is described as a typical change in which the document image disappears from the captured image. However, the present invention is not limited to this. It is possible to adopt a configuration in which it is a change in the position of the image of the document.

  Since the projection instruction system described in the fourth embodiment has substantially the same configuration and function as the projection instruction system described in the first embodiment, the differences will be mainly described below and the differences will not be described. Omitted. Further, for convenience of explanation, the device numbers and the like will be described using the same numbers as the device numbers used in the first embodiment.

  First, a document display screen FD displayed on the display device 2100 in the fourth embodiment will be described with reference to FIG. FIG. 41 is a diagram illustrating an example of a document display screen FD displayed on the display device 2100 in the fourth embodiment.

  Unlike the first embodiment, the document display unit PSD displayed on the document display screen FD displays a notification image PAN2 that notifies that the image ID of the document 1910 has disappeared in the first captured image.

  The change that the image ID of the document 1910 disappears from the first captured image occurs when the document 1910 is removed from the imaging range of the first imaging device 1200. Therefore, the notification that notifies the change that the image ID of the document 1910 has disappeared is simply referred to as a removal notification.

  In order to simplify the description, the display change unit PAS displayed on the document display screen FD does not display the magnification change buttons BT26 and 27 unlike the first embodiment.

  Hereinafter, the reception unit 1010, the transmission unit 1020, and the control unit 1090 constituting the control device 1000 according to the fourth embodiment will be described.

  For convenience of explanation, the receiving unit 1010 does not receive an imaging instruction unlike the first embodiment.

  Unlike the first embodiment, the transmission unit 1020 transmits a removal notification by executing a transmission process.

  Unlike the first embodiment, the control unit 1090 controls the transmission unit 1020 to transmit a removal notification by executing a control process.

  Here, with reference to FIG. 42, a control process executed by the control unit 1090 in the fourth embodiment will be described. FIG. 42 is a flowchart illustrating an example of a control process executed by the control unit 1090 in the fourth embodiment.

  First, control unit 1090 executes the processes of steps ST3901 to ST3903 (steps ST3901 to ST3903). Note that the processing of steps ST3901 to ST3903 is the same as the processing of steps ST0101 to ST0103 described with reference to FIG.

  When determining in step ST3903 that the end instruction has not been acquired, control unit 1090 determines whether or not detection unit 1040 has detected the image ID of document 1910 (step ST3904). If it is determined that the image ID of document 1910 has been detected, control unit 1090 executes the process of step ST3907, and otherwise the process of step ST3905.

  When determining in step ST3904 that the image ID of the document 1910 has not been detected, the control unit 1090 generates a removal notification (step ST3905). Next, control section 1090 controls transmission section 1020 to transmit the generated notification (step ST3906). Thereafter, control unit 1090 returns to step ST3903 and repeats the above processing.

  When determining in step ST3904 that the image ID of the document 1910 has been detected, the control unit 1090 executes processing from step ST3907 to step ST3913 (step ST3907 to step ST3912). Note that the processing from step ST3907 to step ST3912 is the same as the processing from step ST0107 to ST0112 described with reference to FIG.

  Hereinafter, the reception unit 2010, the transmission unit 2020, and the control unit 2090 that configure the instruction device 2000 according to the fourth embodiment will be described.

  Unlike the first embodiment, the reception unit 2010 also receives a removal notification by executing a reception process.

  Unlike the first embodiment, the transmission unit 2020 does not transmit an imaging instruction for convenience of explanation.

  Unlike the first embodiment, the control unit 2090 controls the display device to display the notification image PAN2 that notifies the change of the image of the document according to the removal notification received by the reception unit 2010 by executing the document display control process. To do.

  Here, a document display control process executed by the control unit 2090 in the fourth embodiment will be described with reference to FIG. FIG. 43 is a flowchart illustrating an example of a document display control process executed by the control unit 2090 in the fourth embodiment.

  First, control unit 2090 executes the processes of steps ST4001 to ST4005 (steps ST4001 to ST4005). Note that the processing of steps ST4001 to ST4004 is the same as the processing of steps ST2501 to ST2505 described with reference to FIG.

  When determining in step ST4004 that the range change instruction has not been acquired, control section 2090 determines whether or not a removal notification has been acquired from receiving section 2010 (step ST4006). If it is determined that the document change instruction has been acquired, control unit 2090 performs the process of step ST4007, and otherwise, executes the process of step ST4008.

  If it is determined in step ST4006 that a removal notification has been acquired, control unit 2090 executes notification control processing for controlling display of notification image PAN2 in accordance with the removal notification (step ST4007). Thereafter, control unit 2090 returns to step ST4003 and repeats the above processing.

  Note that the notification control process includes a step of generating the notification image PAN2 based on the notification content and a step of controlling the display device 2100 to display the notification image PAN2 generated in the previous step.

  If it is determined in step ST4004 that a range change instruction has been acquired, control section 2090 executes the processing of steps ST4008 to ST4010 (steps ST4008 to ST4010).

  Note that the processing of steps ST4008 to ST4010 is the same as the processing of steps ST2508 to ST2510 described with reference to FIG.

  In the present embodiment, the receiving unit 1010 corresponds to a receiving unit included in the control device and the control system, the transmitting unit 1020 corresponds to a transmitting unit included in the control device, and the detecting unit 1040 corresponds to a detecting unit included in the control device. The control unit 1090 corresponds to the control unit included in the control device, the reception unit 2010 corresponds to the reception unit included in the instruction device, the transmission unit 2020 corresponds to the transmission unit included in the instruction device, and the control unit 2090 corresponds to the instruction device. The projection apparatus 1100 corresponds to the projection means included in the control system, and the first imaging apparatus 1200 corresponds to the imaging means included in the control system.

  In the fifth embodiment, as in the fourth embodiment, a control device that transmits a notification that notifies a change in the image of a document in the captured image, and a notification that notifies a change in the image of the document based on the notification. A projection instruction system including an instruction device that controls a display device to display an image will be described.

  However, in this embodiment, the description will be made assuming that the change in the image of the document in the captured image refers to a change in which the image of one document is changed to the image of another document.

  The projection instruction system described in the fifth embodiment has substantially the same configuration and function as the projection instruction system described in the first embodiment. Therefore, the differences will be mainly described below, and the differences will not be described. Omitted. Further, for convenience of explanation, the device numbers and the like will be described using the same numbers as the device numbers used in the first embodiment.

  First, referring to FIG. 44, a document display screen FD displayed on the display device 2100 in the fifth embodiment will be described. FIG. 44 is a diagram illustrating an example of a document display screen FD displayed on the display device 2100 in the fifth embodiment.

  Unlike the first embodiment, the document display unit PSD displayed on the document display screen FD displays a notification image PAN3 that notifies that the image ID of the document 1910 in the first captured image has changed to an image of another document. .

  Note that the case where the image ID of the document 1910 in the first captured image changes to an image of another document usually occurs when the document 1910 captured by the first imaging device 1200 is changed to another document. Therefore, the notification for notifying that the image ID of the document 1910 in the first captured image has changed to an image of another document is hereinafter simply referred to as a change notification.

  In order to simplify the description, the display change unit PAS displayed on the document display screen FD does not display the magnification change buttons BT26 and 27 unlike the first embodiment.

  Hereinafter, the reception unit 1010, the transmission unit 1020, the detection unit 1040, and the control unit 1090 that configure the control device 1000 according to the fifth embodiment will be described.

  For convenience of explanation, the receiving unit 1010 does not receive an imaging instruction unlike the first embodiment.

  Unlike the first embodiment, the transmission unit 1020 also transmits a change notification by executing a transmission process.

  Unlike the first embodiment, the detection unit 1040 detects a change in the image ID of the document 1910 in the first captured image by executing a detection process.

  Here, with reference to FIG. 45, the detection process which the detection part 1040 performs in Example 5 is demonstrated. FIG. 45 is a flowchart illustrating an example of detection processing executed by the detection unit 1040 in the fifth embodiment.

  First, the detection unit 1040 executes processing from step ST4101 to ST4105 (steps ST4101 to ST4105). Note that the processing of steps ST4101 to ST4105 is the same as the processing of steps ST3501 to ST3505 described with reference to FIG.

  After executing Step ST4105, the detection unit 1040 determines whether or not the document 1910 represented by the first captured image as the image ID has been changed based on the color of the pixels constituting the image ID of the document 1910 (Step ST4106). ).

  Next, detection unit 1040 outputs the determination result to control unit 1090 (step ST4107). Thereafter, the detection unit 1040 ends the execution of the detection process.

  In the present embodiment, the detection unit 1040 has described the case where the change of the document 1910 represented as the image ID is detected based on the color of the pixel constituting the image ID of the document 1910. However, the present invention is not limited to this. For example, a configuration in which a change in the document 1910 is detected based on characters acquired by performing character identification processing on the image ID of the document 1910 can be employed.

  Unlike the first embodiment, the control unit 1090 controls the transmission unit 1020 to transmit a change notification by executing a control process.

  Here, with reference to FIG. 46, a control process executed by the control unit 1090 in the fifth embodiment will be described. FIG. 46 is a flowchart illustrating an example of a control process executed by the control unit 1090 in the fifth embodiment.

  First, control section 1090 executes the processing of steps ST4201 to ST4203 (steps ST4201 to ST4203). Note that the processing of steps ST4201 to ST4203 is the same as the processing of steps ST0101 to ST0103 described with reference to FIG.

  When determining in step ST4203 that the end instruction has not been acquired, control section 1090 determines whether or not detection section 1040 has detected a change in document 1910 represented by the image ID (step ST4204). If it is determined that a change in document 1910 has been detected, control unit 1090 executes the process of step ST4205; otherwise, it executes the process of step ST4207.

  When determining in step ST4204 that the change of the document 1910 has been detected, the control unit 1090 generates a change notification for notifying the change of the document (step ST4205). Next, control section 1090 controls transmission section 1020 to transmit the generated notification (step ST4206). Thereafter, control unit 1090 returns to step ST4203 and repeats the above processing.

  If it is determined in step ST4203 that an end instruction has been acquired, control section 1090 executes the process of step ST4213 (step ST4203). Next, the control unit 1090 ends the execution of the control process. The process in step ST4203 is the same as the process in step ST0113 described with reference to FIG.

  When determining in step ST4204 that the change of the document 1910 has not been detected, the control unit 1090 executes processing from step ST4207 to step ST4212 (step ST4207 to step ST4212).

  The processing from step ST4207 to step ST4212 is the same as the processing from step ST0107 to ST0112 described with reference to FIG.

  Note that the function of the instruction device 2000 in the fifth embodiment is such that the change notification is received instead of the removal notification, and the notification image PAN4 is displayed instead of the notification image PAN3 according to the received notification. Although different from the function of the device 2000, other functions are almost the same. Therefore, in order to realize the function of the instruction device 2000 according to the fifth embodiment, description of processing executed by the instruction device 2000 is omitted.

  In the present embodiment, the receiving unit 1010 corresponds to a receiving unit included in the control device and the control system, the transmitting unit 1020 corresponds to a transmitting unit included in the control device, and the detecting unit 1040 corresponds to a detecting unit included in the control device. The control unit 1090 corresponds to the control unit included in the control device, the reception unit 2010 corresponds to the reception unit included in the instruction device, the transmission unit 2020 corresponds to the transmission unit included in the instruction device, and the control unit 2090 corresponds to the instruction device. The projection apparatus 1100 corresponds to the projection means included in the control system, and the first imaging apparatus 1200 corresponds to the imaging means included in the control system.

  In the sixth embodiment, after detecting information identifying the document from the image of the document represented by the captured image, the control device that transmits the detected information and the information identifying the document captured in the captured image are received. Based on the received information, the display device is controlled to display a notification image for notifying the change of the document, and is configured by an instruction device that controls the display device to display a part of the document identified by the information. The projection instruction system will be described.

  Since the projection instruction system described in the sixth embodiment has substantially the same configuration and function as the projection instruction system described in the first embodiment, the differences will be mainly described below and the differences will not be described. Omitted. Further, for convenience of explanation, the device numbers and the like will be described using the same numbers as the device numbers used in the first embodiment.

  First, the configuration of the document 1910 in the sixth embodiment and the document display screen FD displayed on the display device 2100 to display the document 1910 in the sixth embodiment will be described with reference to FIG. FIG. 47 is a diagram illustrating an example of a document display screen FD displayed on the display device 2100 in the sixth embodiment.

  First, the configuration of the document 1910 according to the sixth embodiment will be described with reference to FIG.

  A document 1910 shown in FIG. 47A has a document identification image QR that identifies the document 1910. The document identification image QR is an image representing a QR code or a barcode, for example. The document identification image QR represents, for example, document identification information such as a document name or a document title.

  In the present embodiment, the QR code included in the document 1910 has been described as representing the document identification information. However, the present invention is not limited to this. For example, a configuration representing the file identification information of the electronic file representing the document 1910 can be adopted. . The file identification information includes information indicating the location of an electronic file such as a URL (Uniform Resource Locator).

  Next, with reference to FIG. 47B, a document display screen FD displayed on the display device 2100 in order to display the document 1910 in the sixth embodiment will be described.

  Unlike the first embodiment, the document display unit PSD displayed on the document display screen FD illustrated in FIG. 47B displays a notification image PAN4, a permission button BT30, and a rejection button BT31.

  In order to simplify the description, the display change unit PAS displayed on the document display screen FD does not display the magnification change buttons BT26 and 27 unlike the first embodiment.

  The notification image PAN4 notifies that the image ID of the document 1910 in the first captured image has changed to an image of another document, and permits the display of another document portion on the document display unit PSD. Alternatively, it is an image that prompts the user to input a rejection signal for rejection.

  The permission button BT30 and the rejection button BT31 are, for example, that the instructor performs an operation such as clicking after the pointer PT is overlaid on the permission button BT30 or the rejection button BT31 by operating the input device 2200, for example. The input device 2200 is a button display indicating that a permission signal or a rejection signal can be input.

  Hereinafter, the reception unit 1010, the transmission unit 1020, the detection unit 1040, and the control unit 1090 that configure the control device 1000 according to the sixth embodiment will be described.

  For convenience of explanation, the receiving unit 1010 does not receive an imaging instruction unlike the first embodiment.

  Unlike the first embodiment, the transmission unit 1020 also transmits the document identification information detected by the detection unit 1040 by executing a transmission process.

  Unlike the first embodiment, the detection unit 1040 executes detection processing to detect document identification information that identifies the document 1910 from the image ID of the document 1910 represented by the first captured image.

  Here, with reference to FIG. 48, a detection process executed by the detection unit 1040 in the sixth embodiment will be described. FIG. 48 is a flowchart illustrating an example of detection processing executed by the detection unit 1040 in the sixth embodiment.

  First, the detection unit 1040 executes processing from steps ST4301 to ST4305 (steps ST4301 to ST4305). Note that the processing of steps ST4301 to ST4305 is the same as the processing of steps ST3501 to ST3505 described with reference to FIG.

  After executing step ST4305, the detection unit 1040 detects the document identification image QR from the image ID of the document 1910 (step ST4306). Next, the detection unit 1040 acquires document identification information from the document identification image QR (step ST4307).

  After that, the detection unit 1040 detects the change of the document 1910 by comparing the previously detected document identification information with the currently detected document identification information (step ST4307).

  Next, detection section 1040 outputs the change detection result and document identification information to control section 1090 (step ST4308). Thereafter, the detection unit 1040 ends the execution of the detection process.

  Unlike the first embodiment, the control unit 1090 controls the transmission unit 1020 to transmit the document identification information by executing a control process.

  Here, with reference to FIG. 49, a control process executed by the control unit 1090 in the sixth embodiment will be described. FIG. 49 is a flowchart illustrating an example of a control process executed by the control unit 1090 in the sixth embodiment.

  First, control unit 1090 executes the processes of steps ST4401 to ST4403 (steps ST4401 to ST4403). Note that the processing of steps ST4401 to ST4403 is the same as the processing of steps ST0101 to ST0103 described with reference to FIG.

  When determining in step ST4403 that the end instruction has not been acquired, control section 1090 determines whether or not detection section 1040 has detected a change in document 1910 from the image ID (step ST4404). If it is determined that a change in document 1910 has been detected, control unit 1090 executes the process of step ST4405, and otherwise the process of step ST4407 is executed.

  When determining in step ST4404 that the change of the document 1910 has been detected, the control unit 1090 acquires document identification information from the detection unit 1040 (step ST4405). Next, control section 1090 controls transmission section 1020 to transmit the acquired document identification information (step ST4406). Thereafter, control unit 1090 returns to step ST4403 and repeats the above processing.

  If it is determined in step ST4403 that an end instruction has been acquired, control section 1090 executes the process of step ST4413 (step ST4403). Next, the control unit 1090 ends the execution of the control process. The process in step ST4403 is the same as the process in step ST0113 described with reference to FIG.

  When determining in step ST4404 that the change of the document 1910 has not been detected, the control unit 1090 executes processing from step ST4407 to step ST4413 (step ST4407 to step ST4413).

  Note that the processing from step ST4407 to step ST4413 is the same as the processing from step ST0107 to ST0113 described with reference to FIG.

  Hereinafter, the reception unit 2010, the transmission unit 2020, and the control unit 2090 that configure the instruction device 2000 according to the sixth embodiment will be described.

  Unlike the first embodiment, the receiving unit 2010 also receives document identification information by executing a receiving process.

  In the present embodiment, the receiving unit 2010 is described as receiving document identification information as well, but the present invention is not limited to this. For example, a configuration for receiving file identification information can be adopted.

  Unlike the first embodiment, the transmission unit 2020 does not transmit an imaging instruction for convenience of explanation.

  Unlike the first embodiment, the control unit 2090 executes a document display control process, and thereby displays a notification image PAN4 that notifies the change of the document 1910 based on the document identification information received by the reception unit 2010. And the display device 2100 is controlled to display the portion of the document identified by the document identification information.

  Now, with reference to FIG. 50, a document display control process executed by the control unit 2090 in the sixth embodiment will be described. FIG. 50 is a flowchart illustrating an example of a document display control process executed by the control unit 2090 in the sixth embodiment.

  First, control unit 2090 executes the processes of steps ST4501 to ST4503. The processing of steps ST4501 to ST4503 is the same as the processing of steps ST2501 to ST2503 described with reference to FIG.

  When determining in step ST4503 that the end instruction has not been acquired, control unit 2090 determines whether or not an permission signal has been acquired from input device 2200 (step ST4504). Control unit 2090 executes the process of step ST4505 when determining that the permission signal has been acquired, and executes the process of step ST4506 otherwise.

  In Step ST4504, when the control unit 2090 determines that the permission signal has been acquired, the document to be controlled to change the document to be displayed to the document 1910 identified by the document identification information acquired from the reception unit 1010. A change control process is executed (step ST4505). Thereafter, control unit 2090 returns to step ST4503 and repeats the above processing.

  In this embodiment, the control unit 2090 controls to change the document to be displayed to the document 1910 represented by the downloaded electronic file after downloading the electronic file identified by the file identification information acquired from the receiving unit 1010. Can be adopted.

  In step ST4504, when it is determined that the permission signal has not been acquired, control unit 2090 determines whether or not document identification information has been acquired from receiving unit 1010 (step ST4506). If it is determined that the document identification information has been acquired, control unit 2090 performs the process of step ST4507, and otherwise, executes the process of step ST4508.

  If it is determined in step ST4506 that the document identification information has been acquired, the control unit 2090 notifies the change to the document identified by the document identification information and prompts the user to input a permission signal or a rejection signal to be rejected. A notification control process for controlling to display PAN4 is executed (step ST4507). Thereafter, control unit 2090 returns to step ST4503 and repeats the above processing.

  The notification control process first controls the display device 2100 to display the step of generating the notification image PAN4 based on the document identification information, the notification image PAN4 generated in the previous step, and the allow button BT30 and the reject button BT31. And a step of performing.

  When determining in step ST4506 that the document identification information has not been acquired, the control unit 1090 executes the processing from step ST4508 to step ST4510 (step ST4508 to step ST4510).

  Note that the processing from step ST4508 to step ST4510 is the same as the processing from step ST2508 to ST2510 described with reference to FIG.

  In the present embodiment, the receiving unit 1010 corresponds to a receiving unit included in the control device and the control system, the transmitting unit 1020 corresponds to a transmitting unit included in the control device, and the detecting unit 1040 corresponds to a detecting unit included in the control device. The control unit 1090 corresponds to the control unit included in the control device, the reception unit 2010 corresponds to the reception unit included in the instruction device, the transmission unit 2020 corresponds to the transmission unit included in the instruction device, and the control unit 2090 corresponds to the instruction device. The projection apparatus 1100 corresponds to the projection means included in the control system, and the first imaging apparatus 1200 corresponds to the imaging means included in the control system.

  In the seventh embodiment, an instruction device that transmits a registration instruction instructing to register a writing image representing writing performed on a document and an electronic file representing the document in association with each other in the storage unit is performed on the document. After receiving a registration instruction instructing to register the writing image representing writing and the electronic file representing the document in association with each other in the storage unit, the image of the document represented by the captured image and the electronic representing the document are received in accordance with the received registration instruction. Projection composed of a control device that extracts a written image based on an image of a document generated based on a file, and controls the extracted written image and an electronic file to be associated and registered in the storage unit The instruction system will be described.

  The projection instruction system described in the seventh embodiment has substantially the same configuration and function as the projection instruction system described in the first embodiment. Therefore, the differences will be mainly described below, and the differences will not be described. Omitted. Further, for convenience of explanation, the device numbers and the like will be described using the same numbers as the device numbers used in the first embodiment.

  First, a document display screen FD displayed on the display device 2100 in the seventh embodiment will be described with reference to FIG. FIG. 51 is a diagram illustrating an example of a document display screen FD displayed on the display device 2100 in the seventh embodiment.

  Unlike the first embodiment, the document display screen FD shown in FIG. 51 displays the registration instruction unit PCS.

  The registration instruction unit PCS displays a button BT40. The button BT40 indicates that a registration instruction (hereinafter also referred to as a registration command) for instructing to register a writing image representing writing performed on the document 1910 and an electronic file representing the document 1910 in association with each other can be input. Button display.

  In order to simplify the description, the display change unit PAS displayed on the document display screen FD does not display the magnification change buttons BT26 and 27 unlike the first embodiment.

  Here, with reference to FIG. 52, an image display screen that displays a written document and an image obtained by capturing the written document will be described. FIG. 52 is a diagram illustrating an example of a written document.

  First, a written document will be described with reference to FIG.

  A document 1910 illustrated in FIG. 52A includes, for example, writing images PW1 and PW2 representing writing performed by a person who operates the control apparatus 1000 using, for example, a pencil, a ballpoint pen, a fountain pen, or magic. Note that the document 1910 shown in FIG. 52A is projected with the annotation image PA61.

  Next, an image display screen that displays an image obtained by capturing the document shown in FIG. 52A will be described with reference to FIG.

  The image display screen FP illustrated in FIG. 52B displays a first captured image obtained by capturing the document 1910 illustrated in FIG. 52A on the first display unit FS1.

  The first captured image displayed by the first display unit FS1 represents the image ID of the document 1910 shown in FIG. Further, the image ID of the document 1910 represents the written images PW1 and PW2 and the annotation image PA61.

  Here, with reference to FIG. 53, the structure of the control apparatus 1000 in Example 7 is demonstrated. FIG. 53 is a functional block diagram illustrating a configuration example of the control device 1000 according to the seventh embodiment.

  Unlike the first embodiment, the control device 1000 illustrated in FIG. 53 further includes an extraction unit 1050 and a registration unit 1060. Note that the functions of the detection unit 1040 and the control unit 1090 are realized by software control executed by the control device 1000.

  Before describing the extraction unit 1050 and the registration unit 1060, information stored in the reception unit 1010 and the storage unit 1030 in the seventh embodiment will be described.

  Unlike the first embodiment, the receiving unit 1010 receives a registration instruction that instructs the storage unit 1030 to register the write image PW and the electronic file in association with each other by executing reception processing. Note that, for convenience of explanation, the receiving unit 1010 does not receive an imaging instruction unlike the first embodiment.

  Unlike the first embodiment, the storage unit 1030 stores the write image PW and information related to the write image PW.

  Note that the information related to the written image PW includes, for example, identification information for identifying the written image PW and position information indicating the writing position of the written image PW. The identification information for identifying the write image PW includes, for example, the file name of the electronic file representing the write image PW.

  Here, with reference to Table 5, information regarding the write image PW stored in the storage unit 1030 will be described. Table 5 is a table showing an example of information related to the write image PW stored in the storage unit 1030.

  Table 5 has a written image identification information field and a position information field. The written image identification information field stores identification information for identifying the written image PW.

  The position information field stores the position information of the write image PW identified by the information stored in the write image identification information field of the same record.

  That is, the storage unit 1030 stores the identification information for identifying the write image PW and the position information indicating the write position of the write image PW in association with each other.

  In the present embodiment, the information related to the write image PW has been described as including, for example, identification information for identifying the write image PW and position information indicating the write position of the write image PW. However, the present invention is not limited to this. Instead, for example, a configuration including information for identifying the document 1910 in which the writing has been performed, a time at which the written image PW is registered, and information for identifying the instructor who has instructed projection of the image onto the document 1910 at the registered time. Can be adopted.

  Here, with reference to Table 6, another example of information related to the write image PW stored in the storage unit 1030 will be described. Table 6 is a table showing another example of information related to the write image PW stored in the storage unit 1030.

  Table 6 has a document identification information field, a written image identification information field, a position information field, a time information field, and an instructor identification information field.

  The document identification information field stores information for identifying the document 1910. The written image identification information field stores identification information for identifying the written image PW written in the document 1910 identified by the information stored in the document identification information field of the same record.

  The position information field stores the position information of the write image PW identified by the information stored in the write image identification information field of the same record.

  The time information field stores information indicating the registration time of information registered in the same record. The time information field can be configured to store information indicating the time of writing.

  The instructor field stores information for identifying an instructor who has instructed the document 1910 to project an image at the registration time. The instructor field can be configured to store information for identifying an instructor who has instructed writing. Further, the display 6 has a writer field, and the writer field can adopt a configuration for storing information for identifying a person who has written.

  Further, the storage unit 1030 includes information for identifying the document 1910 to be annotated by the annotation image PA, identification information for identifying the annotation image PA, position information indicating the projection position of the annotation image PA, time when the annotation image PA is registered or projected, In addition, information identifying the instructor who has instructed the document 1910 to project the annotation image PA at the time of registration or projection is stored in association with each other.

  Here, with reference to Table 7, in the seventh embodiment, an example of information related to the annotation image PA stored in the storage unit 1030 will be described. Table 7 is a table showing an example of information related to the annotation image PA stored in the storage unit 1030.

  Table 7 has a document identification information field, an annotation image identification information field, a position information field, a time information field, and an instructor identification information field. The information stored in each field is the same type as the information stored in each field in Table 6, and thus description thereof is omitted.

  In the present embodiment, the storage unit 1030 registers information for identifying the document 1910 for which the specific image PS specifies a part, identification information for identifying the specific image PS, range information indicating the projection range of the specific image PS, and the specific image PS. Alternatively, it is possible to adopt a configuration in which information for identifying the instructor who has instructed the document 1910 to project the specific image PS at the time of projection and the time of registration or projection is stored in association.

  Here, returning to FIG. 53, the configuration of the control device 1000 will be described.

  The extraction unit 1050 is connected to the first imaging device 1200. The extraction unit 1050 performs extraction processing on the document 1910 based on the image of the document 1910 represented by the first captured image and the image of the document 1910 generated based on the electronic file representing the document 1910. A written image PW representing the written data is extracted.

  Here, with reference to FIG. 54, the extraction process executed by the extraction unit 1050 will be described. FIG. 54 is a flowchart illustrating an example of the extraction process performed by the extraction unit 1050.

  First, the extraction unit 1050 acquires a first captured image from the first imaging device 1200 (step ST4601). Next, extraction section 1050 acquires image ID of document 1910 from the first captured image (step ST4602).

  After that, the extraction unit 1050 generates an image of the document 1910 based on the electronic file representing the document 1910 (step ST4603). Next, the extraction unit 1050 extracts a difference image between the acquired image ID of the document 1910 and the generated image of the document 1910 (step ST4604).

  After that, the extraction unit 1050 sets the extracted difference image as the write image PW, and outputs the extracted position to the registration unit 1060 as the written write position (step ST4605). Thereafter, the extraction unit 1050 ends the execution of the extraction process.

  Here, returning to FIG. 53, the configuration of the control device 1000 will be described.

  The registration unit 1060 performs registration processing, and associates the writing image PW extracted by the extraction unit 1050 with the electronic file representing the document 1910 to which the writing represented by the writing image PW has been registered in the storage unit 1030. To do.

  Here, an example of the registration process executed by the registration unit 1060 will be described with reference to FIG. FIG. 55 is a flowchart illustrating an example of registration processing executed by the registration unit 1060.

  First, registration unit 1060 obtains write image PW and a write position from extraction unit 1040 (step ST4701). Next, the registration unit 1060 acquires the instructor identification information and the electronic file identification information from the control unit 1090, acquires the registration time with reference to the system time, and then writes the writing image PW, the writing position, and the electronic file. The identification information, the registration time, and the instructor identification information are associated with each other and registered in Table 6 stored in the storage unit 1030 (step ST4702).

  Thereafter, registration unit 1060 acquires annotation image PA being projected and the projection position from storage unit 1030 (step ST4703). Next, the registration unit 1060 obtains the registration time by referring to the system time, and the storage unit 1030 stores the annotation image PA, the projection position, the electronic file identification information, the registration time, and the instructor identification information in association with each other. It registers in Table 7 (step ST4704). Thereafter, the registration unit 1060 ends the registration process.

  Unlike the first embodiment, the control unit 1090 controls the registration unit 1060 to perform registration according to a registration instruction by executing a control process.

  Now, with reference to FIG. 56, a control process executed by the control unit 1090 in the seventh embodiment will be described. FIG. 56 is a flowchart illustrating an example of a control process executed by the control unit 1090 in the seventh embodiment.

  First, control unit 1090 executes the processes of steps ST4801 to ST4803 (steps ST4801 to ST4803). Note that the processing of steps ST4801 to ST4803 is the same as the processing of steps ST1101 to ST1103 described with reference to FIG.

  When determining in step ST4803 that the end instruction has not been acquired, control section 1090 determines whether or not a registration instruction (hereinafter also referred to as a registration command) has been acquired from receiving section 1010 (step ST4804). If it is determined that a registration instruction has been acquired, control section 1090 executes the process of step ST4805, and otherwise, executes the process of step ST4809.

  In step ST4804, when it is judged that the control part 1090 acquired the registration instruction | indication, the extraction part 1050 is controlled to perform said extraction process (step ST4805).

  Next, control section 1090 controls registration section 1060 to execute the above registration process (step ST4806). Thereafter, control section 1090 returns to step ST4803 and repeats the above processing.

  When determining in step ST4803 that the end instruction has been acquired, control section 1090 executes the process of step ST4813 (step ST4803). Next, the control unit 1090 ends the execution of the control process. Note that the processing in step ST4803 is the same as the processing in step ST0113 described with reference to FIG.

  When determining in step ST4804 that the registration instruction has not been acquired, the control unit 1090 executes processing from step ST4807 to step ST4812 (step ST4807 to step ST4812).

  Note that the processing from step ST4807 to step ST4812 is the same as the processing from step ST0107 to ST0112 described with reference to FIG.

  Hereinafter, the transmission unit 2020 and the control unit 2090 constituting the instruction device 2000 according to the seventh embodiment will be described.

  Unlike the first embodiment, the transmission unit 2020 associates the writing image PW representing the writing performed on the document 1910 with the electronic file representing the document 1910 and registers them in the storage unit 1030 by executing transmission processing. Send a registration instruction to instruct. Note that, for convenience of explanation, the transmission unit 2020 does not transmit an imaging instruction unlike the first embodiment.

  Unlike the first embodiment, the control unit 2090 controls the transmission unit 2020 to transmit a registration instruction by executing a document display control process.

  Here, with reference to FIG. 57, a document display control process executed by the control unit 2090 in the seventh embodiment will be described. FIG. 57 is a flowchart illustrating an example of a document display control process executed by the control unit 2090 in the seventh embodiment.

  First, control unit 2090 executes the processes of steps ST4901 to ST4905. The processing of steps ST4901 to ST4905 is the same as the processing of steps ST2501 to ST2505 described with reference to FIG.

  When determining in step ST4904 that the range change instruction has not been acquired, control unit 2090 determines whether or not a registration instruction (that is, a registration command) has been acquired from input device 2200 (step ST4906). If it is determined that a registration instruction has been acquired, control unit 2090 performs the process of step ST4907, and otherwise does the process of step ST4909.

  When determining in step ST4906 that the registration instruction has been acquired, control section 2090 controls transmission section 2020 to transmit the registration instruction (step ST4907). Thereafter, control unit 2090 returns to step ST4903 and repeats the above processing.

  If it is determined in step ST4906 that the registration instruction has not been acquired, control unit 2090 executes the processes of steps ST4908 to ST4910 (steps ST4908 to ST4910). Note that the processing in steps ST4908 to ST4910 is the same as the processing in steps ST2508 to ST2510 described with reference to FIG.

  In this embodiment, the receiving unit 1010 corresponds to a receiving unit included in the control device and the control system, the transmitting unit 1020 corresponds to a transmitting unit included in the control device, and the extracting unit 1050 corresponds to an extracting unit included in the control device. The registration unit 1060 corresponds to the registration unit included in the control device, the control unit 1090 corresponds to the control unit included in the control device, the reception unit 2010 corresponds to the reception unit included in the instruction device, and the transmission unit 2020 corresponds to the instruction device. The control unit 2090 corresponds to the control unit included in the instruction device, the projection device 1100 corresponds to the projection unit included in the control system, and the first imaging device 1200 corresponds to the imaging unit included in the control system. .

  The control device 1000 and the instruction device 2000 are realized when the arithmetic unit executes a program stored in at least one of the ROM and the RAM. Further, this program can be provided by being stored and distributed in a magnetic disk, an optical disk, a semiconductor memory, or other recording medium, or distributed via a network.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

It is a block diagram which shows one Embodiment of the projection instruction | indication system comprised with the control apparatus and instruction | indication system of this invention. It is a figure for demonstrating the arrangement position of a projection apparatus, a 1st imaging device, and a dividing device. It is a figure showing an example of the image display screen which a display apparatus displays in order to display a 1st captured image and a 2nd captured image. It is a figure showing an example of the document display screen used in order to display a document. It is a figure for demonstrating an example of the relationship between the display of a display apparatus, and the projection of a projection apparatus. It is a figure showing an example of the position change of the image which a projection device projects. It is a functional block diagram showing the example of 1 structure of a control apparatus. It is a figure showing an example of the hardware constitutions of the control apparatus used in order to perform software control. It is a flowchart showing an example of the detection process which a detection part performs. It is a flowchart showing an example of the control process which the control part with which a control apparatus is provided performs. It is a flowchart showing an example of the initial control process which the control part with which a control apparatus is provided performs. It is a flowchart showing an example of the projection image generation process which the control part with which a control apparatus is provided performs. It is a flowchart showing an example of the drawing process which the control part with which a control apparatus is provided performs. It is a flowchart showing an example of the target annotation image drawing process, document annotation image drawing process, and specific image drawing process which the control part with which a control apparatus is provided performs. It is a flowchart showing an example of the instructor image drawing process and auxiliary image drawing process which the control part with which a control apparatus is provided performs. It is a flowchart showing an example of the target annotation image superimposition processing, document annotation image superimposition processing, and specific image superimposition processing which the control part with which a control apparatus is provided performs. It is a flowchart showing an example of the instructor image superimposition process and auxiliary image superimposition process which the control part with which a control apparatus is provided performs. It is a flowchart showing an example of the guidance image superimposition process which the control part with which a control apparatus is provided performs. It is a flowchart showing an example of the information acquisition process and information transmission control process which the control part with which a control apparatus is provided performs. It is a functional block diagram showing the example of 1 structure of an instruction | indication apparatus. It is a flowchart showing an example of the image display control process which the control part with which an instruction | indication apparatus is provided performs. It is a flowchart showing an example of the range acquisition process which the control part with which an instruction | indication apparatus is provided performs. It is the flowchart showing an example of the 1st directions transmission control processing etc. which the control part with which a directions device is provided performs. It is a flowchart showing an example of the projection instruction | indication production | generation process which the control part with which an instruction | indication apparatus is provided performs. It is a flowchart showing an example of the information acquisition process and apparatus control process which the control part with which an instruction | indication apparatus is provided performs. It is a flowchart showing an example of the document display control process which the control part with which an instruction | indication apparatus is provided performs. It is a flowchart showing an example of the initialization process which the control part with which an instruction | indication apparatus is provided performs. It is a flowchart showing an example of the partial image display control process which the control part with which an instruction | indication apparatus is provided performs. It is a flowchart showing an example of the range change control process and magnification change control process which the control part with which an instruction | indication apparatus is provided performs. It is a flowchart showing an example of the 2nd instruction | indication transmission control process which the control part with which an instruction | indication apparatus is provided performs. It is a figure showing one Embodiment which projects a some specific image. In Example 2, it is a figure showing an example of the document display screen which a display apparatus displays. It is a figure showing an example of an image showing a notice image and other documents which a projection device projects. In Example 2, it is a flowchart showing an example of the control processing which the control part with which a control apparatus is provided performs. 9 is a flowchart illustrating an example of a document display control process executed by a control unit included in an instruction device according to a second embodiment. 9 is a flowchart illustrating an example of a document change control process and a notification transmission control process executed by a control unit included in an instruction device according to a second embodiment. In Example 3, it is a figure showing an example of the image which a projection device projects. In Example 3, it is a flowchart showing an example of the detection process which a detection part performs. In Example 3, it is a flowchart showing an example of the projection image generation process which the control part with which a control apparatus is provided performs. In Example 3, it is a flowchart showing an example of the document annotation image superimposition process and specific image superimposition process which the control part with which a control apparatus is provided performs. In Example 4, it is a figure showing an example of the document display screen which a display apparatus displays. In Example 4, it is a flowchart showing an example of the control processing which the control part with which a control apparatus is provided performs. In Example 4, it is a flowchart showing an example of the document display control process which the control part with which an instruction | indication apparatus is provided performs. In Example 5, it is a figure showing an example of the document display screen which a display apparatus displays. In Example 5, it is a flowchart showing an example of the detection process which a detection part performs. In Example 5, it is a flowchart showing an example of the control processing which the control part with which a control apparatus is provided performs. In Example 6, it is a figure showing an example of the document display screen which a display apparatus displays. In Example 6, it is a flowchart showing an example of the detection process which a detection part performs. In Example 6, it is a flowchart showing an example of the control processing which the control part with which a control apparatus is provided performs. In Example 6, it is a flowchart showing an example of the document display control process which the control part with which an instruction | indication apparatus is provided performs. In Example 7, it is a figure showing an example of the document display screen which a display apparatus displays. It is a figure showing an example of the document in which writing was performed. FIG. 10 is a functional block diagram illustrating a configuration example of a control device included in a control device according to a seventh embodiment. It is a flowchart showing an example of the extraction process which an extraction part performs. It is a flowchart showing an example of the registration process which a registration part performs. In Example 7, it is a flowchart showing an example of the control processing which the control part with which a control apparatus is provided performs. In Example 7, it is a flowchart showing an example of the document display control process which the control part with which an instruction | indication apparatus is provided performs.

Explanation of symbols

1 ... Projection instruction system 10 ... Projection system
DESCRIPTION OF SYMBOLS 100 ... Communication network 1000 ... Control apparatus
1001 ... Calculation unit 1002 ... ROM
1003 ... RAM 1004 ... External storage unit
1005 ... Bus 1010 ... Receiving unit (receiving means)
1020: Transmission means (transmission unit) 1030 ... Storage unit
1040: Detection unit (detection unit) 1050 ... Extraction unit (extraction unit)
1090: Control unit (control unit) 1100 ... Projection device (projection unit)
1200 ... 1st imaging device (imaging means)
1300: Splitting device (half mirror)
1400 ... Second imaging device 1500 ... Sound collector
1500 ... Sound output device 1600 ... Sound collection device
1910 ... Document 1920 ... Subject
1990 ... Projection screen (screen) 20 ... Instruction system
2000 ... Indicator 2005 ... Bus
2010: receiving unit (receiving unit) 2020 ... transmitting unit (transmitting unit)
2030 ... Storage unit 2090 ... Control unit (control means)
2100 ... Display device 2200 ... Input device
2500 ... sound generator 2600 ... sound collector
BT11-15, 21-31, 40 ... button BT11-15 ... button FD ... Document display screen
FP ... Image display screen FS1 ... First display section
FS2 ... 2nd display part IC1-4 ... Icon
ID: Document image IT1: Front image of the object
IT2 ... Image of the top surface of the object
PA11, 12, 31, 32, 61 ... Annotation image (annotation image)
PAN1 to 3 ... Notification image PAS ... Display change section
PCA ... Annotation instruction part PCD ... Document instruction part
PCS ... Registration instruction part PD11, 12, 32, 32 ... Auxiliary image PG11, 12, 21, 31, 32, 41 ... Guide image PO1, ... Optical main axis PP1-4 ... Instructor image
PS1-3 ... Specific image PSD ... Document display part
PSM ...... Thumbnail PT1, 2 of the two-page document (other document) PW1,2 ... Writing image
QR1,2 ... Document identification image

Claims (16)

Receiving means for receiving a captured image of a document;
Control means for controlling the display device to display a portion of the document to be annotated based on an electronic file representing the document imaged in the captured image received by the receiving means;
Transmitting means for transmitting a projection instruction for instructing to project a specific image for specifying a part of the document controlled to be displayed by the control means and an annotation image for annotating the part of the document to the document; A pointing device characterized by.   The instruction apparatus according to claim 1, wherein the projection instruction instructs to project an auxiliary image assisting annotation of the annotation image by deflecting a part of the document to be annotated by the annotation image. The control means changes the display to a part of the other document to be annotated based on an electronic file representing another document different from the document imaged in the captured image received by the receiving means. Control the display device,
The instruction device according to claim 1, wherein the transmission unit transmits a notification instruction for instructing that a document to be annotated has been changed.   The instruction device according to claim 3, wherein the transmission unit transmits an image representing the other document to be an annotation displayed on the display device. The receiving means receives information identifying the document imaged in the captured image;
The control unit controls the display device to display a notification image for notifying the change of the document based on the information received by the receiving unit, and displays a part of the document identified by the information. The pointing device according to claim 1, wherein the display device is controlled.   The transmission unit transmits a registration instruction instructing to register a writing image representing writing performed on the document and an electronic file representing the document in association with each other in a storage unit. The indicating device according to any one of 5. Transmitting means for transmitting a captured image of a document;
A projection instruction for instructing to project an annotation image for annotating a portion of the document captured in the captured image transmitted by the transmission unit and a specific image for identifying the portion of the document to be annotated by the annotation image onto the document. Receiving means for receiving;
A control apparatus comprising: control means for controlling the projection apparatus to project the annotation image and the specific image onto the document according to a projection instruction received by the reception means. The projection instruction instructs to project an auxiliary image that assists annotation of the annotation image by deflecting a part of the document that the annotation image annotates,
The control unit according to claim 7, wherein the control unit controls the projection device to project the auxiliary image onto the document by deflecting a part of the document specified by the specific image according to the projection instruction. Control device. The receiving means receives a notification instruction that instructs to notify that a document to be annotated has been changed,
9. The control unit according to claim 7, wherein the control unit controls the projection device to project a notification image notifying that a document to be annotated has been changed in accordance with the notification instruction received by the reception unit. The control device described in 1. The receiving means receives an image representing another document changed as an annotation target,
The control device according to claim 9, wherein the control unit controls the projection device to project an image received by the reception unit. A detection unit for detecting a change in the position of the document imaged in the captured image;
Based on the change in the position of the document detected by the detection means, the annotation image for annotating the part of the document and the position for projecting the specific image for identifying the part of the document to be annotated by the annotation image are changed. The control device according to claim 7, wherein the control device controls a projection device. Detection means for detecting information for identifying the document from the image of the document represented by the captured image;
The control device according to claim 7, wherein the transmission unit transmits information detected by the detection unit. Extraction means for extracting a writing image representing writing performed on the document based on an image of the document represented by the captured image and an image of the document generated based on an electronic file representing the document;
A registration means for associating the writing image extracted by the extraction means with an electronic file representing a document written by the writing image in a storage unit;
The receiving means receives a registration instruction that instructs the storage unit to register the written image and the electronic file in association with each other,
The control device according to claim 7, wherein the control unit controls the registration unit to register in accordance with the registration instruction. Imaging means for imaging a document;
Receiving means for receiving a projection instruction for instructing to project an annotation image for annotating a part of the document imaged by the imaging means and a specific image for specifying the part of the document to be annotated by the annotation image;
A projection system comprising: a projection unit that projects the annotation image and the specific image onto the document according to a projection instruction received by the reception unit. Computer
Receiving means for receiving a captured image of a document;
Control means for controlling the display device to display a portion of the document to be annotated based on an electronic file representing the document imaged in the captured image received by the receiving means;
Causing the control unit to function as a transmission unit that transmits a projection instruction for instructing to project a specific image that specifies a part of the document controlled to be displayed and an annotation image that annotates the part of the document onto the document. An instruction program characterized by. Computer
Transmitting means for transmitting a captured image of a document;
A projection instruction for instructing to project onto the document an annotation image for annotating the captured document part in the captured image transmitted by the transmission unit and a specific image for identifying the document part to be annotated by the annotation image. Receiving means for receiving;
A control program for causing a projection device to function so as to project the annotation image and the specific image onto the document in accordance with a projection instruction received by the reception unit.

Images