JP2016157201A - Information processing apparatus, operation cancelling method, computer program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a mechanism for performing processing of cancelling a user's operations, including a writing operation and an erasing operation.SOLUTION: An information processing apparatus 111 is connected to a projector 112 that projects an image on a board surface 300 on which a user performs writing using a writing instrument and erasing using an eraser, and a camera 113 that photographs the board surface 300. When receiving an instruction to cancel an operation, the information processing apparatus 111 creates a difference image between a photographed image to be acquired according to the instruction to cancel an operation and a photographed image acquired before receiving the instruction to cancel an operation. The information processing apparatus 111 causes the projector 112 to project the difference image on the board surface 300 to clearly indicate a portion where the writing or erasing has been performed.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for canceling a writing operation and an erasing operation of a character, a figure, a picture or the like handwritten on a whiteboard or the like by a user.

  Some application software executed by an information processing apparatus such as a personal computer has an operation cancel function, that is, a so-called Undo function. For example, Patent Document 1 discloses an image processing command based on a difference between an image generated by an image processing command (drawing, erasing, enlargement, movement, etc.) and an image being displayed in response to an operation cancel instruction. An image editing method for displaying an image before input is disclosed.

JP-A-6-76032

In recent years, a projector with a camera has been proposed in which an image (projected image) is projected using a surface on which a whiteboard is written (hereinafter referred to as a “board surface”) as a projection surface, and the description of the whiteboard is captured together with the projected image. Has been. By using the whiteboard and the projector with the camera, the user can write the projected image by handwriting.
The projection image is generated by the information processing apparatus. For this purpose, the operation cancellation processing for the projected image is performed by using, for example, the technique disclosed in Patent Document 1. However, since the description handwritten by the user cannot be handled as electronic data, the operation cannot be canceled. For example, when a handwritten character or figure is erased, the erased description cannot be restored by the operation cancel function.

  The main object of the present invention is to provide a mechanism for performing processing for canceling operations such as writing operation and erasing operation by handwriting by a user in order to solve the above-described problems.

  An information processing apparatus of the present invention that solves the above-described problems includes a display control means for projecting an image by a predetermined projection means on a board surface on which writing by a user using a writing instrument and erasing using an eraser are performed, The display control unit projects a difference image representing a difference between two captured images captured at different timings with the imaging control unit configured to capture the image by the imaging unit. And a control means for clearly indicating the erased portion.

  According to the present invention, by projecting a difference image representing a difference between two captured images onto a board surface, it is possible to specify a portion where writing and erasing are performed and to cancel the writing operation and the erasing operation.

(A), (b) is explanatory drawing of a whole structure. (A), (b) is a block diagram of information processing apparatus. The illustration figure of the captured image preserve | saved at a preservation | save area | region. The flowchart showing the process at the time of writing and erasing. The flowchart showing operation cancellation processing. Explanatory drawing of the specific example of the operation cancellation process of deletion operation. Explanatory drawing of the specific example of the operation cancellation process of write-in operation. Explanatory drawing of the specific example in the case of performing the operation cancellation process of deletion operation continuously. The whole block diagram of the system of 2nd Embodiment. The functional block diagram of information processing apparatus. The flowchart showing the acquisition process of a captured image. The flowchart showing the preservation | save process of a captured image. The illustration figure of the captured image preserve | saved at a preservation | save area | region. Explanatory drawing of the process which updates a part of captured image. Explanatory drawing of the specific example of the operation cancellation process of write-in operation.

  Hereinafter, embodiments will be described in detail with reference to the drawings.

[First Embodiment]
<Configuration>
FIG. 1 is an explanatory diagram of the overall configuration of the system of this embodiment. FIG. 1A is a diagram illustrating an example of the system 110, and FIG. 1B is an overall configuration diagram of the system 110. The system 110 includes a projector 112 that projects an image onto the board surface 300 of the whiteboard 114, a camera 113 that images the board surface 300 of the whiteboard 114, and an information processing device 111. The board surface 300 of the whiteboard 114 is a surface on which writing with a writing tool such as a pen, and paper or the like on which characters, figures, pictures or the like are written are pasted. Writing with a writing instrument can be erased with an eraser.

  The information processing apparatus 111 generates a projection image projected from the projector 112 and instructs the projection to control the operation of the projector 112. The information processing apparatus 111 transmits an imaging instruction for causing the camera 113 to perform imaging, and acquires a captured image from the camera 113. The information processing apparatus 111 can communicate with other information processing apparatuses and the like via a network 130 such as a LAN (Local Area Network) or a WAN (Wide Area Network). In the present embodiment, the information processing device 111, the projector 112, and the camera 113 are described as independent devices, but these may be configured integrally. Further, any two devices may be integrated.

  The positions of the projector 112 and the whiteboard 114 are determined so that the projector 112 projects an image having the same size as the board 300. Further, the positions of the camera 113 and the whiteboard 114 are determined so that the camera 113 captures the entire area of the board surface 300 of the whiteboard 114. The camera 113 attaches a time stamp indicating the imaging time to the captured image and outputs it to the information processing apparatus 111.

  FIG. 2 is a configuration diagram of the information processing apparatus 111. FIG. 2A is a hardware configuration diagram of the information processing apparatus 111. FIG. 2B is a functional block diagram of the information processing apparatus 111. The information processing apparatus 111 includes a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, and a ROM (Read Only Memory) 203, and controls the operation of the entire system 110. The information processing apparatus 111 includes a keyboard I / F 205, a display I / F 206, an external memory I / F 207, an external I / F 208, a mouse I / F 211, and a network I / F 213 as interfaces with peripheral devices. Each component of the information processing apparatus 111 is connected by a system bus 204 and can communicate with each other.

A keyboard I / F 205 and a mouse I / F 211 are input interfaces. A keyboard 209 that is an input device is connected to the keyboard I / F 205. The mouse I / F 211 is connected to a mouse 212 which is an input device. The CPU 201 receives input signals from the keyboard 209 and the mouse 212 via the keyboard I / F 205 and the mouse I / F 211.
A display I / F 206 is an output interface to which the projector 112 is connected. The CPU 201 outputs a projection image to the projector 112 via the display I / F 206 and instructs the projection.
The external memory I / F 207 is an interface with an external memory 210 that is a large-capacity storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The external I / F 208 is connected to the camera 113. The CPU 201 transmits an imaging instruction to the camera 113 via the external I / F 208 and acquires a captured image captured by the camera 113 via the external I / F 208. The CPU 201 stores the acquired captured image in the external memory 210 via the external memory I / F 207. The CPU 201 transmits an imaging instruction to the camera 113 in accordance with a preset time interval or an input signal representing an imaging instruction from the input device.

  The CPU 201 reads the computer program stored in the ROM 203 and executes it using the RAM 202 as a work area, thereby realizing the functional blocks shown in FIG. 2B for controlling the operations of the projector 112 and the camera 113. The information processing apparatus 111 functions as an input unit 301, an imaging control unit 302, a captured image storage unit 303, a difference image processing unit 304, a display control unit 305, and a control unit 306.

  The input unit 301 receives input signals input from the keyboard 209 and the mouse 212. The imaging control unit 302 controls the imaging operation by the camera 113. The captured image storage unit 303 stores the captured image captured by the camera 113 in a storage area formed in the external memory 210. When the input unit 301 receives an operation cancellation instruction (undo instruction) as an input signal, the difference image processing unit 304 generates a difference image representing a difference between two captured images captured at different timings. The display control unit 305 causes the projector 112 to display an image. When the difference image processing unit 304 generates a difference image, the display control unit 305 causes the projector 112 to display a combined image obtained by combining the difference image with the previous projection image. The control unit 306 controls the operation of the information processing apparatus 111 by controlling processing by the above functional blocks.

  FIG. 3 is a view showing an example of a captured image stored in the storage area of the external memory 210. The ID, the captured image, and the captured time are stored in the storage area of the external memory 210 as one record. ID which is identification information is attached | subjected to a captured image in the acquisition order. The imaging time is represented by a time stamp. In the example of FIG. 3, captured images with ID “1” to ID “3” are stored. The captured image with the ID “1” includes a handwritten “Aiueokiki Kokusashi Suseso”. The captured image with ID “2” includes a handwritten “Aiue Okeki Keko”. The captured image with ID “3” includes handwritten “Aiueo”. These are the timing when the camera 113 handwritten “A line”, “K line”, “Sa line” by the user, the timing when the user erased “Sa line”, the timing when the user erased “Ka line”, FIG.

<Operation cancellation processing>
The system 110 configured as described above responds to an instruction to cancel an operation on characters and figures handwritten by a user using a writing instrument on the surface 300 of the whiteboard 114 and characters and figures erased by the user using an eraser. To restore the state before handwriting and before erasing. FIG. 4 is a flowchart showing processing of the information processing apparatus 111 at the time of writing and erasing on the board surface 300 by the user, including operation cancellation processing.

  The control unit 306 determines whether or not the input unit 301 has received an imaging instruction as an input signal from the keyboard 209 or the mouse 212 (S400). When the input unit 301 receives an imaging instruction (S400: Y), the control unit 306 transmits an imaging instruction to the camera 113 by the imaging control unit 302. The camera 113 receives the imaging instruction, images the board surface 300 of the whiteboard 114, and transmits the captured image to the control unit 306 via the imaging control unit 302. Thereby, the control unit 306 acquires a captured image (S401). The control unit 306 attaches an ID to the acquired captured image and stores it in the storage area by the captured image storage unit 303 (S402).

  After saving the captured image or when no imaging instruction has been received (S400: N), the control unit 306 determines whether the input unit 301 has received an operation cancellation instruction from the keyboard 209 or the mouse 212 as an input signal. (S403). When an operation cancellation instruction is received (S403: Y), the control unit 306 performs an operation cancellation process to be described later (S404). After completion of the operation cancellation process or when an operation cancellation instruction has not been received (S403: N), the control unit 306 ends the process if an instruction to end the process is given (S405: Y), and instructs the end of the process. If not, the processes after S400 are repeated (S405: N).

FIG. 5 is a flowchart showing the operation cancellation processing in S404.
In response to the operation cancellation instruction, the control unit 306 transmits an imaging instruction to the camera 113 by the imaging control unit 302. The camera 113 receives the imaging instruction, images the board surface 300, and transmits the captured image to the control unit 306 via the imaging control unit 302. As a result, the control unit 306 acquires a captured image corresponding to the operation cancellation instruction (S500). The control unit 306 stores the acquired captured image in the storage area by using the captured image storage unit 303 without attaching an ID (S501).

  The control unit 306 causes the difference image processing unit 304 to generate a difference image between the captured image acquired according to the operation cancellation instruction in S500 and the captured image acquired according to the imaging instruction before receiving the operation cancellation instruction ( S502). The difference image processing unit 304 designates the ID of the captured image acquired in accordance with the imaging instruction with the pointer N. The difference image processing unit 304 updates the pointer N so as to specify the ID of the captured image to be stored when the captured image is stored in S402 of FIG. The difference image processing unit 304 generates a difference image from the captured image acquired in response to the operation cancellation instruction and the captured image with the ID specified by the pointer N.

  The difference image processing unit 304 identifies the type of the generated difference image (S503). The types of difference images include “erase difference”, “additional difference”, and “no difference”. When an erasing operation such as erasing characters is performed on a temporally old captured image (ID = N) and a new captured image (captured image acquired in response to an operation cancellation instruction), these captured images are used. The type of the difference image to be generated is “erase difference”. When a writing operation such as addition of characters is performed between a temporally old captured image (ID = N) and a new captured image (captured image acquired in response to an operation cancellation instruction), these captured images The type of the difference image generated from is “additional difference”. When the operation is not performed between the old captured image (ID = N) and the new captured image (captured image acquired in response to the operation cancellation instruction), the type of difference image generated from these captured images Is “no difference”.

  The generation of the difference image and the specification of the type of the difference image can be performed at the same time. For example, the differential image processing unit 304 uses a captured image acquired in response to an operation cancellation instruction, which is used to generate a differential image, and a captured image with an ID (ID = ID), to specify the generation and type of the differential image. N) and the absolute value of the difference in pixel value for each pixel.

The difference image processing unit 304 determines whether the calculated absolute value is equal to or greater than a preset threshold value, and determines that a pixel that is equal to or greater than the threshold value is a pixel having a difference. When the number of pixels having a difference is less than the predetermined number, the difference image processing unit 304 specifies the type of the difference image as “no difference”.
When the number of pixels having a difference is greater than or equal to a predetermined number, the difference image processing unit 304 determines whether or not the color of the pixel having the difference in the captured image acquired in response to the operation cancellation instruction is a background color. The difference image processing unit 304 determines that an erasing operation has been performed when the number of pixels determined to be the background color is greater than or equal to a predetermined number. In this case, the difference image processing unit 304 specifies the type of the difference image as “erase difference”. In the case of “erase difference”, an image obtained by extracting only pixels having a difference in a captured image (ID = N) with an ID is a difference image.
When the number of pixels determined to be the background color is less than the predetermined number, the difference image processing unit 304 determines that the writing operation has been performed, and specifies the type of the difference image as “additional difference”. In this case, an image obtained by extracting only pixels having a difference in the captured image acquired in response to the operation cancellation instruction is a difference image.

The background color is the average value of the pixel values of the captured image obtained by imaging a whiteboard on which nothing is written in advance, or the average value of the background color obtained by imaging the whiteboard and using the background estimation method It is.
The difference image processing unit 304 may specify the type of the difference image with a pixel value for each rectangular area (for example, 8 pixels × 8 pixels) of the captured image instead of the pixel value for each pixel. In this case, the two captured images used for generating the difference image are each divided into rectangles, and the type of the difference image is specified by the average value of the pixel values in the rectangle. As another method for generating the difference image, the two captured images to be compared are binarized, and the region where the writing state has changed is extracted from the shape of the histogram generated in the vertical direction and the horizontal direction of the captured image. There is also a method of making a difference image.

  When the type of the difference image is “erase difference” (S504: erase difference), the control unit 306 generates a composite image by combining the difference image with the projection image (S505). When the type of the difference image is “additional difference” (S504: additional difference), the control unit 306 generates a composite image in which a message that prompts the deletion of writing as a difference is combined with the projection image (S506). When the type of the difference image is “no difference” (504: no difference), the control unit 306 subtracts “1” from the pointer N of the difference image processing unit 304 (S507), The difference image is generated again (S502). Thereby, the generation of the difference image again is performed by the captured image and the captured image (ID = N−1) according to the operation cancellation instruction. The difference image processing unit 304 subtracts the pointer N by “1” by “1” until the type of the difference image becomes “erase difference” or “additional difference”, and sequentially starts from the latest captured image before accepting the operation cancellation instruction. The generation of the difference image is repeated.

  The control unit 306 causes the display control unit 305 to project the composite image from the projector 112 onto the panel surface 300 (S508). As a result of the above processing, the portion of the board surface 300 that the user has written on the board surface 300 with a writing tool or the portion that has been erased by the eraser is clearly shown. A specific operation canceling process for the erasing operation and an operation canceling process for the writing operation will be described.

<Erase operation cancellation processing>
FIG. 6 is an explanatory diagram of a specific example of the operation cancellation process of the erasing operation. In FIG. 6, time elapses from time t1 to time t6.

At time t <b> 1, the user writes “Aiue Okeki Keko” on the surface 300 of the whiteboard 114. At this time, when the user inputs an imaging instruction, the camera 113 images the board surface 300 at time t2, and the captured image 601 is stored in the storage area (S400 to S402 in FIG. 4). The ID of the captured image 601 is “1”.
At time t <b> 3, the user erases “Kakiku Koko” from the board surface 300 of the whiteboard 114 using an eraser. At this time, when the user inputs an imaging instruction, the camera 113 images the board surface 300 at time t4, and the captured image 602 is stored in the storage area (S400 to S402 in FIG. 4). The ID of the captured image 602 is “2”.

  Next, when the user inputs an operation cancellation instruction, the camera 113 images the board surface 300 at time t5, and the captured image 603 is stored in the storage area (S403 in FIG. 4, S500 and S501 in FIG. 5). This operation cancellation instruction is an instruction for canceling the operation of deleting “Kakikakeko” performed by the user immediately before.

The difference image processing unit 304 sets the pointer N to “2”. The difference image processing unit 304 generates a difference image from the captured image 603 acquired in response to the operation cancellation instruction and the captured image 602 with the ID “N = 2”, and specifies the type (S502 in FIG. 5). S503). Since the captured image 603 and the captured image 602 are identified as having no difference, the pointer N of the differential image processing unit 304 is decremented by 1 and set to “1” (S504 and S507 in FIG. 5).
The difference image processing unit 304 generates a difference image from the captured image 603 acquired in response to the operation cancellation instruction and the captured image 601 with the ID “N = 1”, and identifies the type (S502 in FIG. 5). S503). Here, a difference image 604 is generated, and the type is specified as an erasure difference. The difference image 604 becomes the “Kakikukeko” that has been deleted. The difference image processing unit 304 causes the difference between the pixel values to be equal to or greater than a threshold value due to the characters “Kakikakeko”, and the difference image in the captured image 603 acquired in response to the operation cancellation instruction is the background color. The type of 604 is specified as “erase difference”.

  The control unit 306 generates a combined image by combining the generated difference image 605 with the projection image, and causes the projector 112 to project the combined image on the panel surface 300 (S505 and S508 in FIG. 5). As a result, at time t6, “AIUEO” handwritten by the user and “KAKIKAKEKO” projected by the projector 112 are displayed on the surface 300 of the whiteboard 114. “Kekikuko” erased by the user at time t3 is displayed by projector 112 at time t6. In this way, the operation cancellation process of the erasing operation of “Kakikukeko” by the user is performed, and the state of the panel surface 300 before erasing (the state at time t1) can be restored.

<Operation cancellation processing for write operations>
FIG. 7 is an explanatory diagram of a specific example of the operation cancellation processing for the write operation. In FIG. 7, time elapses from time t1 to time t7.

At time t1, the user writes “Aiueo” on the surface 300 of the whiteboard 114. At this time, when the user inputs an imaging instruction, the camera 113 images the board surface 300 at time t2, and the captured image 701 is stored in the storage area (S400 to S402 in FIG. 4). The ID of the captured image 701 is “1”.
At time t <b> 3, the user adds “Kakikukeko” to the board surface 300 of the whiteboard 114. At this time, when the user inputs an imaging instruction, the camera 113 images the board surface 300 at time t4, and the captured image 702 is stored in the storage area (S400 to S402 in FIG. 4). The ID of the captured image 702 is “2”.

  Next, when the user inputs an operation cancellation instruction, the camera 113 captures the board surface 300 at time t5, and the captured image 703 is stored in the storage area (S403 in FIG. 4 and S500 and S501 in FIG. 5). This operation cancellation instruction is an instruction for canceling the operation of writing “Kakikakeko” performed by the user immediately before.

The difference image processing unit 304 sets the pointer N to “2”. The difference image processing unit 304 generates a difference image from the captured image 703 acquired in response to the operation cancellation instruction and the captured image 702 whose ID is “N = 2”, and specifies the type (S502 in FIG. 5). S503). Since the captured image 703 and the captured image 702 are identified as having no difference, the pointer N of the differential image processing unit 304 is decremented by 1 and set to “2” (S504 and S507 in FIG. 5).
The difference image processing unit 304 generates a difference image from the captured image 703 acquired in response to the operation cancellation instruction and the captured image 701 with the ID “N = 1”, and specifies the type (S502 in FIG. 5). S503). Here, a difference image 704 is generated, and the type is specified as a writing difference. The difference image 704 becomes “Kakikukeko” written at time t3. The difference image processing unit 304 uses the character “Kakikakeko” to make the difference between the pixel values equal to or greater than the threshold, and the difference image in the captured image 703 acquired in response to the operation cancellation instruction is not a background color. The type of 704 is specified as “write difference”.

  The control unit 306 generates a composite image by combining a message for prompting deletion of the generated difference image 704 with the projection image, and causes the projector 112 to project the composite image on the board surface 300 (S506 and S508 in FIG. 5). As a result, at time t6, a frame surrounding the added “Kakikukeko” is displayed as a message prompting deletion. The frame line may be blinking or highlighted. At time t7, the user erases “Kakiku Kenko” with the eraser in response to the message. Thereby, the operation cancellation process is completed, and the state of the board surface 300 before the additional writing (the state at time t1) can be restored.

<Continuous operation cancellation processing>
FIG. 8 is an explanatory diagram of a specific example in the case where the operation cancellation process of the erasing operation is continuously performed. In FIG. 8, time elapses from time t1 to time t9.

At time t <b> 1, the user has written “Aiueokiki Kakusashi Suseso” on the surface 300 of the whiteboard 114. At this time, when the user inputs an imaging instruction, the camera 113 images the board surface 300 at time t2, and the captured image 801 is stored in the storage area (S400 to S402 in FIG. 4). The ID of the captured image 801 is “1”.
At time t <b> 3, the user erases “Sashisoseso” from the board surface 300 of the whiteboard 114 using an eraser. At this time, when the user inputs an imaging instruction, the camera 113 images the board surface 300 at time t4, and the captured image 802 is stored in the storage area (S400 to S402 in FIG. 4). The ID of the captured image 802 is “2”.

  At time t <b> 5, the user erases “Kakiku Koko” from the board surface 300 of the whiteboard 114 using an eraser. When the user inputs an operation cancel instruction, the camera 113 captures the board surface 300 at time t6, and the captured image 803 is stored in the storage area (S403 in FIG. 4, S500, S501 in FIG. 5). This operation cancellation instruction is an instruction for canceling the operation of deleting “Kakikakeko” performed by the user immediately before.

  The difference image processing unit 304 sets the pointer N to “2”. The difference image processing unit 304 generates a difference image from the captured image 803 acquired in response to the operation cancellation instruction and the captured image 802 with the ID “N = 2”, and specifies the type (S502 in FIG. 5). S503). Here, a difference image 804 is generated, and the type is specified as an erasure difference. The difference image 806 becomes “Kakikukeko” deleted at time t5. The difference image processing unit 304 causes the difference between the pixel values to be equal to or greater than the threshold value due to the characters “Kakikakeko”, and the difference image in the captured image 803 acquired in response to the operation cancellation instruction is the background color. The type of 804 is specified as “erase difference”.

  The control unit 306 generates a combined image by combining the generated difference image 804 with the projection image, and causes the projector 112 to project the combined image on the panel surface 300 (S505 and S508 in FIG. 5). As a result, at time t7, “AIUEO” handwritten by the user and “KAKIKAKEKO” projected by the projector 112 are displayed on the surface 300 of the whiteboard 114. “Kekikuko” erased by the user at time t5 is displayed by projector 112 at time t7. In this way, the operation cancellation processing of the erasing operation of “Kakikukeko” by the user is performed, and the state of the panel surface 300 before erasing (the state at time t3) can be restored. Even when the user inputs an operation cancellation instruction without inputting an imaging instruction after performing an erasing operation or a writing operation, the operation cancellation processing can be performed correctly.

  When the user continuously inputs an operation cancel instruction during the projection of the composite image, the camera 113 images the board surface 300 at time t8, and the captured image 805 is stored in the storage area (S403 in FIG. 4 and S500 in FIG. 5). , S501). This operation cancellation instruction is an instruction for canceling the operation for deleting “Sashisuseso” performed by the user at time t3. The captured image 805 is an image that includes handwritten “Aiueo” and projected “Kakikukeko”.

The difference image processing unit 304 sets the pointer N to “2”. The difference image processing unit 304 generates a difference image from the captured image 805 acquired in response to the operation cancellation instruction and the captured image 802 with the ID “N = 2”, and specifies the type (S502 in FIG. 5). S503). Since the captured image 805 and the captured image 802 are identified as having no difference, the pointer N of the difference image processing unit 304 is decremented by 1 and set to “1” (S504 and S507 in FIG. 5).
The difference image processing unit 304 generates a difference image from the captured image 805 acquired in response to the operation cancellation instruction and the captured image 801 with the ID “N = 1”, and specifies the type (S502 in FIG. 5). S503). Here, a difference image 806 is generated, and the type is specified as an erasure difference. The difference image 806 is the erased “Sashisuseso”. The difference image processing unit 304 causes the difference between the pixel values to be equal to or greater than the threshold due to the characters “Sashisuseso”, and the pixel that becomes the difference in the captured image 805 acquired in response to the operation cancellation instruction is the background color. The type is specified as “erase difference”.

  The control unit 306 combines the generated difference images 804 and 806 with the projection image to generate a combined image, and causes the projector 112 to project the combined image onto the board surface 300 (S505 and S508 in FIG. 5). As a result, at time t <b> 9, “Ai Ueo” handwritten by the user and “Kakikoku Kosashi Suseo” projected by the projector 112 are displayed on the surface 300 of the whiteboard 114. “Sashisuseso” erased by the user at time t3 is displayed by projector 112 at time t9. In this way, the operation cancellation processing of the “Sashisuseso” erasure operation performed at time t3 by the user is performed, and the state of the board surface 300 before erasure (the state at time t1) can be restored.

  As described above, when the operation cancellation process is continuously performed, the control unit 306 includes a captured image acquired after the last accepted operation cancellation instruction and a captured image acquired before the first accepted operation cancellation instruction. A difference image is generated and displayed by comparison. Further, at the time of projection, all the difference images generated while continuously accepting the operation cancellation instruction are included in the composite image.

  As described above, by using the projector 112 and the camera 113, it is possible to clearly indicate to the user the description added and erased by the user on the surface 300 of the whiteboard 114 by the writing operation. Accordingly, the system 110 can perform processing for canceling operations such as a writing operation and an erasing operation by the user's hand.

[Second Embodiment]
In the first embodiment, the board surface 300 is imaged in accordance with an imaging instruction from the user, but in the second embodiment, the board surface 300 is imaged according to the user's operation in front of the board surface 300. Therefore, the system of the second embodiment performs processing using a distance image acquired by a distance image sensor.

  FIG. 9 is an overall configuration diagram of a system 120 according to the second embodiment. The system 120 has a configuration in which a distance image sensor 115 is added to the system 110 of the first embodiment. The distance image sensor 115 acquires a distance image of the board surface 300 of the whiteboard 114. Since the projector 112 and the camera 113 have the same configuration as in the first embodiment, description thereof is omitted. The information processing apparatus 116 has a configuration in which an interface for acquiring a distance image from the distance image sensor 115 is added to the hardware configuration similar to that of the information processing apparatus 111 of the first embodiment. The positions of the distance image sensor 115 and the whiteboard 114 are determined so that the distance image sensor 115 captures the entire area of the board surface 300 of the whiteboard 114.

  FIG. 10 is a functional block diagram of the information processing apparatus 116. The CPU 201 reads a computer program stored in the ROM 203 and executes it using the RAM 202 as a work area, thereby realizing a function for controlling the operation of the entire system 120. The information processing apparatus 116 functions as an input unit 301, an imaging control unit 302, a captured image storage unit 303, a difference image processing unit 304, a display control unit 305, a control unit 306, a distance image processing unit 307, and a moving object region storage unit 308. To do. Since the input unit 301, the imaging control unit 302, the captured image storage unit 303, the difference image processing unit 304, the display control unit 305, and the control unit 306 have the same configuration as that of the information processing apparatus 111 of the first embodiment, a description will be given. Omitted.

  The distance image processing unit 307 detects an area where an object (for example, a user) has moved in front of the board surface 300 of the whiteboard 114 based on the distance image acquired from the distance image sensor 115. The distance image processing unit 307 acquires a distance image from the distance image sensor 115 at a predetermined time interval (for example, every 0.1 second). The distance image processing unit 307 can detect the region where the object has moved by detecting the object region (moving object region) from the acquired distance image and comparing it with the moving object region of the distance image acquired last time. The moving object region storage unit 308 stores the moving object region detected by the distance image processing unit 307 in a storage region formed in the external memory 210.

  Similar to the system 110 of the first embodiment, the system 120 configured as described above can cancel the user's write operation and erase operation in response to an operation cancel instruction. In the second embodiment, imaging of the board surface 300 by the camera 113 is performed without using an imaging instruction as in the first embodiment. FIG. 11 is a flowchart illustrating a captured image acquisition process. This process is executed each time the information processing apparatus 116 acquires a distance image from the distance image sensor 115.

  The distance image processing unit 307 detects a moving object region representing an object region in front of the board surface 300 of the whiteboard 114 from the distance image acquired from the distance image sensor 115 (S1100). The distance image processing unit 307 stores the detected moving body region in the storage area by the moving body region storage unit 308. The distance image processing unit 307 acquires the moving object region detected from the previously acquired distance image from the moving object region storage unit 308 (S1101).

The distance image processing unit 307 compares the latest moving object area detected this time with the previously detected moving object area, and determines the presence or absence of the area where the moving object has disappeared (S1102). The distance image processing unit 307 determines that an area that is included only in the previous moving object area among the latest moving object area and the previous moving object area is an area where the moving object has disappeared.
When there is no area where the object disappears (S1102: N), the control unit 306 determines that the object is not moving in front of the board surface 300, ends the captured image acquisition process, and performs the processes after S403 in FIG. Run. When there is a region where the object has disappeared (S1102: Y), the control unit 306 determines that the object has moved in front of the board surface 300, performs a captured image storage process, and the captured image storage unit 303 stores the captured image. (S1103). Thereafter, the control unit 306 ends the captured image acquisition process, and executes the processes after S403 in FIG.

FIG. 12 is a flowchart showing the captured image storage processing in step S1103.
The control unit 306 acquires the captured image of the board surface 300 captured by the camera 113 via the imaging control unit 302, similarly to S401 of FIG. 4 (S1200).
The control unit 306 cuts out an image of the area where the moving object determined in S1102 disappears from the acquired captured image (S1201). The control unit 306 synthesizes the cut-out area image with the previously acquired captured image stored in the storage area (S1202). The control unit 306 stores the previously acquired captured image obtained by synthesizing the cut-out region images in the captured image storage unit 303 (S1203). Thereby, a part of the captured image acquired last time is updated and stored.

  The control unit 306 determines whether or not the area cut out in S1201 is adjacent to the area cut out in the previous process (S1204). For example, the control unit 306 determines whether or not they are adjacent by comparing the coordinates of the clipped regions.

  When adjacent (S1204: Y), the control unit 306 determines whether or not the time difference between the imaging time of the captured image acquired in S1200 and the imaging time of the previous captured image is less than a predetermined threshold. (S1205). The imaging time is stored in the captured image storage unit 303 together with the captured image. When the time difference is less than the threshold (S1205: Y), the control unit 306 assigns the same ID as the previous captured image to the captured image stored in S1203 (S1206). In this case, the control unit 306 assigns the same ID as a captured image during a series of operations in order to determine that the user is performing a writing or erasing operation through a series of operations.

  When the cut-out area is not adjacent (S1204: N), or when the time difference is equal to or greater than the threshold (S1205: N), the control unit 306 adds an ID different from the previous captured image to the captured image stored in S1203. (S1207). In this case, the control unit 306 assigns different IDs as captured images of different operations in order to determine that the user has finished a series of operations and is performing other operations.

  FIG. 13 is an exemplary view of a captured image stored in the storage area by the captured image storage unit 303 in this way. The ID, the captured image, and the captured time are stored in the storage area of the external memory 210 as one record. In the example of FIG. 13, records 1401 to 1405 are stored in the storage area 1400. Records 1401 to 1403 are determined as a series of user operations ("Aiueo" writing operation), and are assigned the same ID. The record 1404 and the record 1405 are determined as a series of user operations (writing operation of “Kakikakeko”), and are assigned the same ID.

  FIG. 14 is an explanatory diagram of processing for updating a part of the captured image in S1202 of FIG. In FIG. 14, from time t1 to time t3, the user performs a writing operation on the surface 300 of the whiteboard 114 with a writing tool. The board surface 300 of the whiteboard 114 captured by the camera 113 and the distance image sensor 115 is recognized by being divided into a predetermined number. In the present embodiment, the board surface 300 is recognized as being divided into 64 parts, but it may be divided into right and left parts or four parts, or may be divided in units of pixels with all pixels as individual areas.

  The distance image 1300 at time t1 includes a moving object region 1301. The distance image 1305 at time t2 includes a moving object region 1306. The distance image 1310 at time t3 includes a moving object region 1311.

  The distance image processing unit 307 that has acquired the distance image 1305 at time t2 acquires the moving object region 1306 (S1100). The distance image processing unit 307 acquires the moving object region 1301 of the distance image 1300 acquired at time t1 (S1101). The distance image processing unit 307 compares the moving object region 1301 and the moving object region 1306 to determine the presence / absence of the region where the moving object has disappeared (S1102). In this example, a region 1308 in which no moving object is present is extracted. For this purpose, captured image storage processing is performed (S1103).

  The control unit 306 acquires the captured image 1309 (S1200). The control unit 306 cuts out a region 1308 in which no moving object is present from the captured image 1309 (S1201). The control unit 306 combines the image of the clipped area with the captured image 1304 acquired last time (S1202). Here, “up” is cut out and combined with “ai” in the captured image 1304, so that an image similar to the captured image 1309 is obtained.

  The control unit 306 determines whether or not the cut-out area is adjacent to the area cut out in the previous process (S1204). The cut-out area is the same as the area where the moving object disappears. For this reason, the control unit 306 determines whether or not the area 1308 in which the moving object has disappeared and the area 1303 in which the moving object has disappeared are adjacent to each other. In this example, since the area 1308 in which the moving object has disappeared and the area 1303 in which the moving object has disappeared are adjacent (S1204: Y), the control unit 306 determines that the time difference between the captured images 1309 and 1304 is a threshold value. It is determined whether it is less than (S1205). Referring to FIG. 13, the time difference between the captured times of the captured image 1309 and the captured image 1304 is 2 seconds (records 1401 and 1402). When the threshold value is 5 seconds, the control unit 306 determines that the time difference is less than the threshold value (S1205: Y). The control unit 306 assigns the same ID as that of the previous captured image 1304 to the combined captured image and stores it in the storage area (S1206).

  When processing similar to that at time t2 is performed at time t3, an image similar to the captured image 1314 is obtained in S1203 of FIG. The area 1313 where the moving object disappears is adjacent to the area 1308 where the moving object disappeared in the previous process. Since the time difference between the imaging times of the captured image 1314 and the captured image 1309 is less than the threshold, the control unit 306 assigns the same ID as the previous captured image 1309 to the combined captured image, and saves it in the storage area. save.

<Operation cancellation processing for write operations>
FIG. 15 is an explanatory diagram of a specific example of the operation canceling process of the writing operation according to the second embodiment. In FIG. 15, time elapses from time t1 to time t13. The partial update of the captured image is performed as shown in FIG. 14, and the captured image is stored as shown in FIG. A writing operation is performed on the surface 300 of the whiteboard 114 as shown from time t1 to time t10. In addition, the information processing apparatus 111 acquires captured images 1500 to 1504. The captured images 1500 to 1502 are stored with ID “1”, and the captured images 1503 and 1504 are stored with ID “2”.

  When the user inputs an operation cancellation instruction in this state, the camera 113 captures the board surface 300 at time t11, and the captured image 1505 is stored in the storage area (S403 in FIG. 4 and S500 and S501 in FIG. 5). This operation cancellation instruction is an instruction for canceling the operation of writing “Kakikakeko” performed by the user immediately before.

  The difference image processing unit 304 sets the pointer N to “2”. The difference image processing unit 304 generates a difference image from the captured image 1505 acquired in response to the operation cancellation instruction and the captured image 1504 whose ID is “N = 2”, and specifies the type (S502 in FIG. 5). S503). The captured image 1503 and the captured image 1504 have the same ID, but since the captured image 1504 has a newer capturing time, the differential image processing unit 304 uses the captured image 1504 to generate a differential image.

Since the captured image 703 and the captured image 702 are identified as having no difference, the pointer N of the difference image processing unit 304 is decremented by 1 and set to “2” (S504 and S507 in FIG. 5).
The difference image processing unit 304 generates a difference image from the captured image 1505 acquired in response to the operation cancellation instruction and the captured image 1502 whose ID is “N = 1”, and specifies the type (S502 in FIG. 5). S503). Here, a difference image 1506 is generated, and the type is specified as a writing difference. The difference image 1506 becomes “Kakikukeko” written at time t9. The difference image processing unit 304 determines that the difference between the pixel values is greater than or equal to the threshold value due to the characters “Kakikakeko”, and the pixel that is the difference in the captured image 1505 acquired in response to the operation cancellation instruction is not the background color. Is specified as “write difference”. The difference image processing unit 304 generates a difference image using the captured image 1502 having the latest imaging time among the captured images 1500 to 1502 having the same ID.

  The control unit 306 generates a composite image by combining a message prompting to delete the generated difference image 1506 with the projection image, and causes the projector 112 to project the composite image onto the board surface 300 (S506 and S508 in FIG. 5). As a result, at time t12, a frame surrounding the added “Kakikukeko” is displayed as a message prompting deletion. The frame line may be blinking or highlighted. At time t <b> 13, the user erases “Kakiku Kenko” with the eraser in response to the message. Thereby, the operation cancellation process is completed, and the state of the board surface 300 before the additional writing (the state at time t5) can be restored.

In the second embodiment, a captured image is stored each time a moving object moves in front of the board surface 300. Therefore, when the operation canceling process as in the first embodiment is performed, the operation canceling process is executed in units of characters, and the operation canceling becomes complicated. Therefore, in the second embodiment, the same ID is assigned to a captured image that is determined to be a series of user actions, thereby enabling an operation canceling process of a writing operation and an erasing operation by the series of actions. In addition, you may delete the captured image to which the same ID was attached | subjected from a preservation | save area | region, leaving the thing with the newest imaging time.
As described above, in the second embodiment, a captured image is acquired according to a user's operation. Therefore, unlike the first embodiment, it is not necessary for the user to input an imaging instruction, and it is easy for the user to use.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (11)

Display control means for projecting an image by a predetermined projection means on a board surface on which writing using a writing instrument by a user and erasing using an eraser are performed;
Imaging control means for imaging the board surface by predetermined imaging means;
Control means for projecting a difference image representing a difference between two captured images taken at different timings from the projection means onto the board surface by the display control means to clearly indicate the portion where the writing and erasing are performed. And comprising:
Information processing device. An input means for receiving instructions,
When the input unit accepts an instruction to cancel the operation, the control unit includes a captured image acquired according to the instruction to cancel the operation and a captured image acquired before receiving the instruction to cancel the operation. Generating the difference image,
The information processing apparatus according to claim 1. The imaging control means acquires the captured image with an imaging time from the imaging means,
The control means performs the comparison in order starting from the newest captured image before the instruction for canceling the operation until the difference between the captured image and the captured image acquired in response to the instruction for canceling the operation occurs. Features
The information processing apparatus according to claim 2. The control means, when the input means continuously receives an instruction to cancel the operation, the captured image acquired according to the instruction to cancel the operation received last, and the cancellation of the operation received first Generating the difference image with the captured image acquired before the instruction of
The information processing apparatus according to claim 2 or 3. The control means includes
If there is a part added to the new captured image in time from the old captured image, an image that clearly indicates the added part is projected from the projection unit onto the board surface by the display control unit. Let
If the temporally new captured image includes a portion erased from the temporally old captured image, the image of the erased portion is projected from the projection unit onto the board surface by the display control unit. And
The information processing apparatus according to any one of claims 1 to 4. The control means causes the display control means to project an image surrounding the added part with a frame line from the projection means onto the board surface,
The information processing apparatus according to claim 5. A distance image obtained by acquiring a distance image of the board surface from a predetermined distance image sensor at a predetermined time interval, and detecting the presence or absence of an area where the object has moved in front of the board surface from the acquired distance image and the previously acquired distance image. With processing means,
The control means acquires the captured image from the imaging means if there is a region where the object has moved,
The information processing apparatus according to claim 1. The control means generates a difference image by using a captured image obtained by continuously acquiring a region where the object moves adjacently and a time difference in imaging time being less than a predetermined threshold as a captured image during a series of user operations. In addition, using a captured image with the newest imaging time of a series of operations,
The information processing apparatus according to claim 7. This is a method executed by an information processing apparatus connected to a projection means for projecting an image on a board surface to be written by a user and erased using an eraser and an imaging means for imaging the board surface, and having an input means. And
When receiving an instruction to cancel an operation by the input unit, a difference image between the captured image acquired according to the instruction to cancel the operation and the captured image acquired before receiving the instruction to cancel the operation is generated. The difference image is projected from the projection means onto the board surface, and the written and erased portions are clearly indicated.
Operation cancellation method. A computer that includes an input unit connected to a projecting unit that projects an image on a board surface on which writing using a writing instrument by a user and erasure using an eraser are performed, and an imaging unit that images the board surface,
When receiving an instruction to cancel an operation by the input unit, a difference image between the captured image acquired according to the instruction to cancel the operation and the captured image acquired before receiving the instruction to cancel the operation is generated. A computer program for projecting the difference image from the projection means onto the board surface to clearly indicate the portion where the writing and erasing have been performed.   A computer-readable storage medium storing the computer program according to claim 10.

Images