JP5950633B2 - Information processing apparatus, computer program, and information processing method - Google Patents

Description

  The present invention relates to an information processing apparatus that controls display of electronic sticky note data obtained by digitizing a paper sticky note attached on a screen.

  A method called brainstorming is known as a method for creating new knowledge by a plurality of people. In brainstorming, you write ideas one at a time on cards and share them to diverge the ideas. As a method for converging these ideas, a method called KJ method is known. The KJ method is a method of organizing relationships between groups by repeatedly grouping ideas that are similar to each other by collecting and classifying them. And in this work, discovery of the essence of ideas and creation of new ideas are performed.

  In the brainstorming or KJ method, a method of using a sticky sticky note (such as Post-It (trademark)) as the aforementioned card and affixing the sticky sticky note on a whiteboard or the like is often used. Furthermore, a sticky note may be pasted on the whiteboard while projecting an image such as a presentation material on the whiteboard using a projector.

  Patent Document 1 discloses a technique for photographing handwritten information written on a whiteboard on which an image is projected and information on sticky notes pasted on the whiteboard. In Patent Document 1, only the additional information is photographed by linking the projector and the digital camera to hide the projection image from the projector at the timing of photographing the additional information (handwritten information or sticky note information). Obtained captured images. Then, the projected image immediately before photographing and the photographed photographed image are recorded in association with each other so that they can be combined and displayed at the time of reproduction later.

  On the other hand, Patent Document 2 discloses a conference support system that transmits electronic data written on a touch panel of a terminal device to a server, and the server displays the received electronic data at a corresponding position on a display.

JP 2006-184333 A JP 2011-90486 A

  As described in Patent Document 1, electronic data (photographed image) obtained by photographing a sticky note on a paper pasted on a whiteboard is recorded in association with a projected image, so that a recording result can be obtained later. Is reproduced, the projected image and the captured image can be combined and displayed. That is, in Patent Document 1, if the user switches to the reproduction mode and further selects a projection image to be reproduced, the related captured image can be combined with the projection image and displayed. However, in Patent Document 1, when a captured image is to be reproduced, an instruction from the user such as mode switching or image selection is required, which is troublesome for the user and electronic data is being obtained during brainstorming. It is not suitable for uses such as playback.

  Further, Patent Document 2 displays electronic data input from a touch panel immediately on a display, and does not consider displaying a photographed image obtained by photographing a paper sticky note.

  When the “technology for photographing and digitizing a paper sticky note” in Patent Document 1 is combined with the “technology for immediately displaying electronic data on a display” in Patent Document 2, when the sticky note is photographed, the photographed image is immediately displayed. Projecting on a whiteboard is conceivable. However, if a photographed image of a sticky note is projected immediately, the photographed image will be projected with a paper sticky note still attached to the whiteboard. In addition, there may be a slight shift in the reproduction position of the photographed image due to errors during photographing or reproduction. Therefore, when a photographed image is projected with a paper sticky note still attached to the whiteboard, the characters written in the paper sticky note and the characters in the photographed image are slightly shifted and overlapped. Characters are difficult to see.

In order to solve the above-described problem, the information processing apparatus of the present invention extracts a sticky note image from a photographed image obtained by photographing a screen with a paper sticky note attached thereto, and generates electronic sticky note data of the extracted sticky note image. Means, registration means for registering electronic sticky note data generated by the sticky note image extracting means in a storage device, first electronic sticky note data corresponding to the first paper sticky note generated by the sticky note image extracting means, and the Comparison means for comparing electronic sticky note data registered in the storage device, and second electronic sticky note data that matches the first electronic sticky note data as a result of comparison by the comparison means and corresponds to the second paper sticky note data Is determined not to be registered in the storage device, it is determined that a new paper sticky note is attached to the screen, the first electronic sticky note data is registered by the registration means, and the second electronic sticky note is registered. When it is determined that the sticky note data has been registered in the storage device, and the information included in the second electronic sticky note data indicates a pasting state, it is determined that the pasting position of the second paper sticky note has moved on the screen. The display position information in the second electronic sticky note data registered in the storage device is updated to the information indicated by the display position information in the first electronic sticky note data, and the second electronic sticky note data is registered in the storage device. If the information included in the second electronic sticky note data does not indicate the pasting state, it is judged that the second paper sticky note has been pasted again on the screen, and the second registered in the storage device is determined. About the registration control means for updating the information indicating the pasting state in the electronic sticky note data to information indicating that it is pasted, and the electronic sticky note data registered in the storage device, Determining means for determining whether or not a corresponding paper sticky note is affixed on the screen, and the electronic sticky note data determined by the determination means that the corresponding paper sticky note is not affixed on the screen is a corresponding electronic sticky note The electronic sticky note data determined by the determining means that the corresponding paper sticky note is pasted on the screen is displayed on the screen. Display control means for controlling to prevent display.

  According to the present invention, since the display / non-display of the corresponding electronic sticky note is automatically switched according to the sticking situation of the paper sticky note, the image of the electronic sticky note is projected onto the paper sticky note. It can prevent becoming difficult to see.

It is a figure which shows the structure of an information processing system. It is a figure which shows the structure of a server (information processing apparatus). It is a flowchart of the process which converts a paper sticky note into an electronic sticky note. It is a flowchart of the process which image | photographs automatically, when fixed time passes. It is a flowchart of the process which image | photographs automatically in the detection state of a human sensitive sensor. It is a block diagram which shows the detail of a tag image extraction process. It is a figure which shows the data structure of an electronic tag. It is a flowchart of the process which displays an electronic tag. It is a figure which shows the example of the white board with which the paper sticky note was stuck. It is a specific example of the data structure of the paper sticky note 1. It is a figure which shows the example of the white board after peeling the paper sticky note 1. FIG. It is a figure which shows the example of the white board after displaying the electronic sticky note 1. FIG.

  FIG. 1 is a diagram showing a configuration of an information processing system in the present embodiment. The information processing system includes a server 111 as an information processing device, a digital camera 113 as a photographing device, and a display device 115. The display device 115 further includes a projector 112 and a whiteboard 114, and an image projected from the projector 112 is displayed on the whiteboard 114. The information processing system according to the present embodiment can be used in conferences such as brainstorming, the KJ method, or the like, but the usage is not limited thereto.

  In the present embodiment, a sticky sticky tag (Post-it (trademark), etc.) is described using a case where it is pasted on a screen such as a whiteboard. However, the present invention is not limited to a sticky sticky note, The same applies to a case where a document or card is attached using a tape or the like. Further, the material of the sticky note, the document, and the card is not limited to paper, but may be other materials (plastic, cloth, etc.).

  In the following embodiments, a sticky note or document pasted on a whiteboard (screen) is referred to as a paper sticky note (or paper document) regardless of the material. Further, the sticky note data digitized by photographing or scanning the paper sticky note is referred to as an electronic sticky note (or electronic document) to be distinguished.

  FIG. 2 is a diagram illustrating a configuration of the server (information processing apparatus) 111. The server 111 includes a CPU 201, a RAM 202, a ROM 203, a keyboard I / F 205, a display I / F 206, an external memory I / F 207, and an external I / F 208, which are connected by a system bus 204. A keyboard 209 connected via the keyboard I / F 205 receives various operations from the user. A projector 112 is connected to the display I / F 206, and the server 111 projects an image on the whiteboard via the projector 112. A digital camera 113 is connected to the external I / F 208, and the server 111 can capture an image of a paper sticky note photographed by the digital camera 113 and create an electronic sticky note (sticky note data). In the present embodiment, electronic data input by the user using the keyboard 209 is also handled in the same way as an electronic sticky note created from an image of a paper sticky note, and these can be mixed and displayed on the projector 112. To do.

  A CPU (processor) 201 controls the entire server 111, reads programs stored in the ROM 203, the external memory 211, and the like into the RAM 202 and executes various processes. That is, the CPU 201 of the server 111 functions as a processing unit (processing unit) that executes each step of a flowchart described later by executing a computer program stored in a computer-readable storage medium. The ROM 203 stores various data including a program for starting the CPU 201. The RAM 202 is used as a work memory.

  Next, the operation of this system will be described. FIG. 3 is a flowchart of processing for creating and registering or changing an electronic sticky note (sticky note data) using a sticky note image obtained by photographing a paper sticky note in this system. The procedure shown in this flowchart is executed by the CPU 201 with the program stored in one of the ROM 203, RAM 202, and external memory (HDD, etc.) 211 of the server 111.

  In brainstorming, the user's ideas are written on a paper sticky note and pasted on the whiteboard 114 to proceed. Here, a description will be given from the point that three paper sticky notes are pasted and the state of the whiteboard 114 is as shown in FIG. In the present embodiment, as shown in FIG. 9, an example is described in which brainstorming is performed by pasting a paper sticky note in a state where a reference material (such as a presentation image) is not projected, but the present invention is not limited to this. Absent. The present invention can also be applied to a case where a paper sticky note is pasted in a state where a presentation image is projected as in Patent Document 1.

  First, the digital camera 113 that has detected a shooting request from the user captures an image of the whiteboard 114 (step S301). Data captured by the digital camera 113 is stored in the RAM 202 through the external I / F 208 of the server 111. When brainstorming is performed in a state where no reference material is projected as in the present embodiment, the captured image includes a paper sticky note image and a projected electronic sticky note image.

  Note that when the present invention is applied in a form in which a paper sticky note is pasted while the reference material is projected, the image of the reference material is included in the captured image if the image is taken as it is. Therefore, when the present invention is applied in such a form, for example, by obtaining the difference between the captured image and the reference material image, an image obtained by removing the reference material image from the captured image is obtained, and the obtained What is necessary is just to perform the process after step S302 with respect to the obtained image.

  In step S302, a sticky note image is extracted from the photographed image, and electronic sticky note data is generated from the extracted sticky note image. Since three paper sticky notes are affixed to the whiteboard 114 in FIG. 9, three sticky note images are obtained. The data structure of the electronic sticky note including the sticky note image will be described later.

  FIG. 6 is a diagram showing a detailed procedure of the tag image extraction process (step S302). When the server CPU 201 executes the program, the server functions as the processing units 602 to 605 in FIG. 6.

  Contour extraction processing by the contour extraction unit 602 and color determination processing by the color determination unit 603 are first performed on the captured image data 601 captured by the digital camera 113 and stored in the RAM 202 of the server 111. In the contour extraction process, by performing a known edge extraction or contour tracking process, the contour of an object existing in the image (the contour of a paper sticky note, the contour of a character image written on a paper sticky note, etc.) can be extracted. Further, the color of each object (paper tag or character) can be extracted by the color determination process. Next, a sticky note area determination process by the sticky note area determination unit 604 is performed. In the sticky note area determination process, an area of a rectangular outline object having a size satisfying a preliminarily assumed size of the paper sticky note and having a color different from the color of the whiteboard (white background color) is used as the sticky note area. Judge as. At this time, if it is configured to determine whether or not a character-size outline is included in the rectangular outline, the sticky note area can be determined more accurately. Then, a sticky note area cutout unit 605 performs a sticky note area cutout process for cutting out a partial image (sticky note image) of a sticky note area corresponding to each of the paper sticky notes, and the electronic sticky note data is based on the extracted sticky note image. Is generated. At this time, the OCR process is performed on the sticky note image to obtain text data of the character image entered inside the sticky note, or the sticky note image is vectorized to obtain the character image vector data. You may comprise as follows.

  FIG. 7 shows a data structure 701 of the generated electronic sticky note. Each electronic sticky note has a header portion 702 and a data portion 703. In the header portion 702, “ID” for identifying each electronic sticky note, “pasting time” indicating the time when the corresponding paper sticky note is pasted, and the corresponding paper sticky note are still stuck on the whiteboard. “Paste status ON / OFF” indicating whether the label is peeled off, “Paste location information” indicating the location where the paper sticky note is pasted, and “Display terminal information” for identifying the display device 115 that displays the sticky note data , “Paste coordinates in display terminal” indicating the display coordinates of the electronic sticky note on the display device, indicating whether the sticky note is an electronic sticky note created by digitizing a paper sticky note or an electronic sticky note created by inputting through a keyboard or the like. "Initial input status (paper / electronic)" information, "Reference material information for simultaneous display" to identify the reference material (presentation image) that was displayed when a paper sticky note was attached, etc. Consisting of broadcast. The data unit 703 includes image data of a sticky note image cut out from an image photographed by the digital camera 113. The data portion 703 further stores OCR data indicating the text of the OCR processing result when the image data is subjected to OCR processing, and vector data when the image data is subjected to vectorization processing.

  A specific example of the data structure of the electronic sticky note is shown in FIG. Here, it is assumed that the electronic sticky note 1 is generated from the sticky note image of the paper sticky note 1 of FIG. The contents of the header portion 1002 of the electronic sticky note 1 are as follows. Since the ID is 1, the pasted time is 10:00:00, and the corresponding paper sticky note 1 is still pasted, the pasted state is ON, the pasting location is the conference room 1, and the display terminal information is the server 111. The paste coordinates in the display terminal are (X1, Y1), the initial input state is a paper sticky note, and there is no reference material for simultaneous display. Further, the content of the data portion 1003 is that the image data is data1. When the OCR process and the vectorization process are not performed, the OCR data and the vectorization data are null.

  In step S303 of the flowchart of FIG. 3, each of the sticky note images included in the electronic sticky note generated in step S302 is sequentially processed and compared with the electronic sticky note already registered in the storage device (external memory or RAM) of the server 111. It is verified whether or not electronic sticky note data of the same sticky note image has already been registered. If it is determined that the tag image extracted in S302 is different from the electronic tag registered in the storage device of the server 111 based on the collation result in S303, it is determined in step S304 that a new tag has been added and the process proceeds to step 305. . In step S305, the electronic sticky note data created based on the sticky note image is registered in the server 111 as new electronic sticky note data.

  In step S306, it is determined whether all of the sticky note images extracted in S302 have been processed. If it is determined that there are other unprocessed sticky note images, the process returns to step S303 to execute processing of other sticky note images. For example, since the sticky image of the paper sticky note 2 and the sticky note image of the paper sticky note 3 still remain when the processing of the sticky note image of the paper sticky note 1 in FIG. 9 is performed, the process returns to step S303 to return to the next sticky note image. To be processed.

  If it is determined that the sticky note image extracted in S302 matches any of the electronic sticky note data registered in the server 111 based on the collation result in step S303, the sticking position on the white board of the paper sticky note is moved ( Alternatively, it is determined in step S304 that the paper tag once peeled is reapplied), and the process proceeds to step S307. In step S304, the server determines that the sticking position of the paper sticky note on the whiteboard has been moved when the "sticking state" information of the registered electronic sticky note data determined to match is ON. . In step S304, if the “pasted state” information of the registered electronic sticky note data determined to match is OFF, the server further refers to the sticking coordinate information in the display terminal and extracts the sticky note image extracted in S302. Is different from the projected position of the electronic sticky note, it is determined that the paper sticky note once peeled is reapplied. At this time, if the position of the sticky note image extracted in S302 is the same as the projection position of the electronic sticky note, it can be determined that the sticky note image being projected has been shot.

  In step S307, the registered electronic sticky note data determined to match is updated. Here, based on the position on the whiteboard where the paper sticky note is pasted, the information of “the paste coordinates in the display terminal” indicating the coordinates where the electronic sticky note should be displayed on the display device is updated. Also, when it is determined that the sticking position of the paper sticky note has been moved or when it is determined that the paper sticky note has been reattached, the “sticking state” information of the corresponding electronic sticky note is turned ON.

  In step S308, it is determined whether all the sticky note images extracted in S302 have been processed. If it is determined that there is another unprocessed sticky note image, the process returns to step S303 to execute processing of other sticky note images.

  If it is determined in step S306 or S308 that all the sticky note images extracted in S302 have been processed, the process advances to step S309 to determine whether there is a peeled paper sticky note. Here, among the registered electronic sticky note data, the registered electronic sticky note data that has not been determined to match any of the sticky note images in the collation in S303, even though the pasting state is ON, It is determined that the sticky note is registered electronic sticky note data.

  If it is determined in step S309 that there is a peeled paper sticky note, the process proceeds to step S310, the “sticking state” information of the registered electronic sticky note data corresponding to the peeled paper sticky note is changed to OFF, and the process ends. To do.

  Next, processing for displaying an electronic sticky note will be described with reference to the flowchart of FIG. The procedure shown in this flowchart is executed by the CPU 201 with the program stored in one of the ROM 203, RAM 202, and external memory (HDD, etc.) 211 of the server 111.

  In step S802, the server 111 analyzes the header part of the registered electronic sticky note data. When the paper sticky note in FIG. 9 is digitized according to the process of the flowchart in FIG. 3, three electronic sticky notes are registered. First, for the electronic sticky note 1, the header part of the electronic sticky note data is analyzed.

  In step S803, the server 111 determines whether the “sticking state” information of the electronic sticky note data analyzed in step S802 is ON. If it is ON, the server 111 proceeds to step S804, and if OFF, the process proceeds to step S805. . For example, immediately after digitizing the paper sticky note, the paper sticky note is in a state of being pasted, so the “sticking state” information of the header portion 1002 of the electronic sticky note 1 is ON, and the process proceeds to step S804.

  In step S <b> 804, the server 111 hides the electronic sticky note data that has been determined that the “sticking state” information is ON. In the example of FIG. 9, the electronic sticky note 1 whose “sticking state” information is ON is not displayed. This is because if the sticky image of the electronic sticky note 1 is projected while the paper sticky note 1 is pasted on the whiteboard, the character image of the electronic sticky note is projected so as to overlap the character of the paper sticky note. This is because both of them are difficult to see due to the effects of displacement and blur.

  In step S <b> 805, the server 111 displays the electronic sticky note data that has been determined that the “sticking state” information is OFF. That is, when it is determined that the paper sticky note has been peeled off, an electronic sticky note corresponding to the peeled paper sticky note is displayed.

  In step S806, the server 111 determines whether there is unprocessed electronic sticky note data. If it is determined that there is other electronic sticky note data, the server 111 returns to step S802 to analyze the next electronic sticky note data. In the example of FIG. 9, after it is determined that the electronic sticky note 1 is not displayed, the same processing is repeated for the remaining electronic sticky notes 2 and 3.

  In the example of FIG. 9, since the paper sticky notes 1 to 3 are still attached to the whiteboard, the electronic sticky note data does not exist in the display data transferred from the server 111 to the projector 112.

  As described above, when a paper sticky note is still pasted, the contents of the paper sticky note can be easily seen by making it non-display even if it is converted to an electronic sticky note.

  Next, the case where the paper tag attached to the whiteboard 114 changes from the state of FIG. 9 to the state of FIG. 11 will be described with reference to the flowcharts of FIGS. FIG. 11 shows a state where the paper sticky note 1 of FIG. 9 attached to the whiteboard 114 has been removed.

  In step S301, the digital camera 113 that has detected the photographing request captures an image on the whiteboard 114. Data captured by the digital camera 113 is stored in the RAM 202 through the external I / F 208 of the server 111.

  Next, a sticky note image is extracted from the image of the whiteboard 114, and electronic sticky note data is generated from the extracted sticky note image (step S302). Since two paper sticky notes are affixed to the whiteboard 114 in FIG. 11, two pieces of electronic sticky note data are obtained.

  Each of the two electronic sticky note data generated in step S302 is collated with the already registered three electronic sticky note data (step S303). It is determined that the two electronic sticky note data generated in step S302 have already been registered, and the process proceeds to step S307 where the registered electronic sticky note data is updated. In step S309, the server removes the paper sticky note 1 corresponding to the electronic sticky note data (electronic sticky note 1 in the example of FIG. 11) that has not been determined to match in step S303 among the already registered electronic sticky note data. It is determined that Therefore, for the electronic sticky note 1, since the corresponding paper sticky note 1 has been removed, the “sticking state” information of the server registration data of the electronic sticky note 1 is changed from ON to OFF (step S310).

  Since the electronic sticky note data registered in the server 111 has been changed, the display content is also changed accordingly. In the flowchart of FIG. 8, as a result of analyzing the header part of the electronic sticky note 1 in step S802, the “sticking state” information has been changed from ON to OFF, so the process proceeds to step S805, and the electronic sticky note 1 is displayed. . Note that the electronic sticky note 2 and the electronic sticky note 3 remain hidden because the “sticking state” information remains ON. Accordingly, only the data of the electronic sticky note 1 exists in the display data transferred from the display I / F 206 of the server 111 to the projector 112. That is, since the electronic sticky note 1 is projected on the whiteboard 114 in FIG. 11, the electronic sticky note 1 is displayed and the paper sticky note 2 and the paper sticky note 3 are pasted as shown in FIG. Become.

  As described above, immediately after the paper sticky note is removed, the corresponding electronic sticky note is automatically displayed. Note that the user can change the display position of the projected electronic sticky note or delete the display by an operation using the keyboard or mouse (not shown) of the server 111. For example, when a meeting is held by a plurality of people, if a user who is discussing in front of the whiteboard pastes a paper sticky note, the paper sticky note is digitized. When the paper sticky note is peeled off, the corresponding electronic sticky note is automatically displayed, so that the user (for example, chairperson) participating in the discussion near the server (information processing apparatus) 111 can display the displayed electronic sticky note. It becomes possible to operate.

  In the present embodiment, the screen of the whiteboard 114 is photographed with a photographing request from the user as a trigger. However, the photographing may be automatically performed every time a certain time elapses, for example. In this case, as shown in the flowchart of FIG. 4, in step S401, the digital camera 113 automatically captures images after a predetermined time has elapsed. In step S402, the server 111 determines whether the image is the same as the previous image. If the image is the same, the process returns to step S401. If the image is different, the process proceeds to step S302. That is, if the sticking state of the paper sticky note on the whiteboard has changed, the process proceeds to step S302. Since the process after step S302 is the same as that of FIG. 3, description is abbreviate | omitted.

  As another method, as shown in the flowchart of FIG. 5, a human sensor is installed in the vicinity of the whiteboard 114 and the human sensor is operated (step S501). When the human sensor detects a person within a predetermined range (before the whiteboard) and then changes to non-detection (step S502), the digital camera 113 images the whiteboard 114 (step S301). Also good. Since the process after step S302 is the same as that of FIG. 3, description is abbreviate | omitted.

  In this embodiment, the paper sticky note is referred to as electronic sticky note. However, the data directly written on the whiteboard 114 may be digitized.

  As described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention.

(Other examples)
In the above-described embodiment, the CPU (processor) of the server 111 functions as each processing unit by executing a program. However, part or all of the processing is configured by a dedicated electronic circuit (hardware). It doesn't matter if you do.

  The present invention can also be realized by executing the following processing. In this process, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or apparatus executes the program. It is a process to read and execute.

Claims (13)

A sticky note image extracting means for extracting a sticky note image from a photographed image obtained by photographing a screen with a paper sticky note, and generating electronic sticky note data of the extracted sticky note image;
Registration means for registering electronic sticky note data generated by the sticky note image extracting means in a storage device;
Comparing means for comparing the first electronic sticky note data corresponding to the first paper sticky note generated by the sticky note image extracting means and the electronic sticky note data registered in the storage device;
When it is determined that the second electronic sticky note data corresponding to the second paper sticky note is not registered in the storage device as a result of the comparison by the comparison means, the second electronic sticky note data is matched with the first electronic sticky note data. It is determined that a new paper sticky note has been attached, the first electronic sticky note data is registered by the registration means, the second electronic sticky note data is determined to be registered in the storage device, and the second electronic sticky note is registered. When the information included in the data indicates a pasting state, it is determined that the pasting position of the second paper sticky note has moved on the screen, and the display position information in the second electronic sticky note data registered in the storage device Updating to the information indicated by the display position information in the first electronic sticky note data, it is determined that the second electronic sticky note data has been registered in the storage device, and the information contained in the second electronic sticky note data is When the attached state is not indicated, it is determined that the second paper sticky note is attached again on the screen, and information indicating the attached state in the second electronic sticky note data registered in the storage device is attached. Registration control means for updating to information indicating
Determination means for determining whether or not a corresponding paper sticky note is pasted on the screen for the electronic sticky note data registered in the storage device;
The electronic sticky note data determined by the determining means that the corresponding paper sticky note is not pasted on the screen is controlled to display an image of the corresponding electronic sticky note on the screen, while the determining means controls the correspondence The electronic sticky note data determined to have a paper sticky note to be pasted on the screen has display control means for controlling the corresponding electronic sticky note image not to be displayed on the screen. apparatus.   Of the electronic sticky note data registered by the registration means, the third electronic sticky note data to which the corresponding paper sticky note is attached is determined not to match any electronic sticky note image data by the comparison by the comparison means. The control means determines that the paper sticky note corresponding to the third electronic sticky note data has been peeled off on the screen, and indicates that the sticking state in the third electronic sticking data registered in the storage device is not pasted. The information processing apparatus according to claim 1, wherein the information processing apparatus is updated to a state shown.   The display control means can be further controlled to display a reference material on the screen, and the electronic sticky note data registered by the registration means has a corresponding paper sticky note stuck on the screen. The information processing apparatus according to claim 1, further comprising information for identifying the reference material displayed on the screen.   The information processing apparatus according to claim 1, wherein the photographed image is an image photographed in accordance with a photographing instruction of a user.   The information processing apparatus according to claim 1, wherein the photographed image is an image that is automatically photographed at regular intervals and is different from an image photographed one sheet before.   The information processing apparatus according to claim 1, wherein the photographed image is an image photographed in accordance with a change from a state in which the human sensor detects a person to a non-detection state.   The display position changing means for changing the display position of the electronic sticky note image controlled to be displayed on the screen by the display control means based on a user instruction. The information processing apparatus described.   2. The display erasure unit according to claim 1, further comprising: a display erasure unit that erases the display of the electronic sticky note image controlled to be displayed on the screen by the display control unit based on a user instruction. Information processing device.   The information processing apparatus according to claim 1, wherein the paper sticky note includes at least one of a sticky sticky note, a paper document, and a card.   The information processing apparatus according to claim 1, wherein the screen is a whiteboard.   The computer program for functioning a computer as each means of the information processing apparatus of any one of Claims 1 thru | or 10.   A computer-readable storage medium storing the computer program according to claim 11. A sticky note image extracting step of extracting a sticky note image from a photographed image obtained by photographing a screen with a paper sticky note, and generating electronic sticky note data of the extracted sticky note image;
A registration step of registering the electronic sticky note data generated in the sticky note image extraction step in a storage device;
A comparison step of comparing the first electronic sticky note data corresponding to the first paper sticky note generated in the sticky note image extraction step and the electronic sticky note data registered in the storage device;
As a result of the comparison in the comparison step, when it is determined that the second electronic sticky note data corresponding to the second paper sticky note data is not registered in the storage device, the screen matches the first electronic sticky note data. It is determined that a new paper sticky note has been attached, the first electronic sticky note data is registered by the registration step, the second electronic sticky note data is determined to be registered in the storage device, and the second electronic sticky note is registered. When the information included in the data indicates a pasting state, it is determined that the pasting position of the second paper sticky note has moved on the screen, and the display position information in the second electronic sticky note data registered in the storage device Updating to the information indicated by the display position information in the first electronic sticky note data, it is determined that the second electronic sticky note data has been registered in the storage device, and the second electronic sticky note data is present If the information does not indicate the pasting state, it is determined that the second paper sticky note has been pasted again on the screen, and information indicating the pasting state in the second electronic sticky note data registered in the storage device is pasted. A registration control step for updating to information indicating that;
A determination step of determining whether or not a corresponding paper sticky note is pasted on the screen for the electronic sticky note data registered in the storage device;
The electronic sticky note data determined in the determination step that the corresponding paper sticky note is not pasted on the screen is controlled to display an image of the corresponding electronic sticky note on the screen, while the correspondence step is performed in the determination step. The electronic sticky note data determined to have a paper sticky note to be pasted on the screen has a display control step for controlling not to display the image of the corresponding electronic sticky note on the screen. Method.

Images