JP2014238663A - Information processing system, information processing system control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide superimposed display having high visibility in a case of displaying electronic information by superimposing the electronic information on a picked-up image.SOLUTION: An information processing apparatus includes: detection means detecting an object from a picked-up image; managing means managing electronic information input in response to a user's instruction while associating the electronic information with one or a plurality of objects detected by the detection means; selection means selecting at least one object from the objects managed by the managing means; and superimposed display means displaying the electronic information associated with the object selected by the selection means by superimposing the electronic information on the picked-up image without creating an icon representing the electronic information, and displaying electronic information associated with the objects that are not selected by the selection means by creating an icon representing the electronic information and superimposing the icon on each of the unselected objects. The superimposed display means arranges the electronic information associated with the object selected by the selection means on the basis of a position of the selected object in the picked-up image.

Description

  The present invention relates to an information processing system, a control method thereof, and a program.

  There is an electronic bulletin board system (hereinafter simply referred to as an electronic bulletin board) that controls registration and reading of information by a large number of users. Electronic bulletin boards are used to widely communicate various information and communicate between users. Some electronic bulletin boards can handle not only comment text, which is text information, but also images such as photographs (Patent Document 1).

  There is also a technique for improving the visibility of information displayed on the display unit of the information terminal. For example, in a multi-window system in which a plurality of windows are displayed on one display unit, the windows may overlap due to display area limitations of the display unit. When the windows overlap, it becomes difficult for the user to read information displayed in the windows (decrease in visibility). In order to solve this, in Patent Document 2, when windows overlap each other, the lower layer window is displayed as an icon, and the upper layer window is displayed as an icon. When the upper layer window is resized due to resizing or the like, the iconized lower layer window is de-iconified (cancel icon).

  For example, there is a superimposed display system that displays a captured image obtained by capturing with a camera, detects a feature pattern from the captured image, and superimposes and displays electronic information corresponding to the feature pattern. In such a superimposed display system, electronic information corresponding to each of a plurality of detected feature patterns may overlap each other. When electronic information overlaps, it becomes difficult for users to read each other's electronic information (decrease in visibility). In order to solve this problem, there is an apparatus in which electronic information is calculated and superimposed so that electronic information does not overlap (Non-Patent Document 1).

JP 2001-101114 A JP-A-6-124184

Takaaki Makita and two others, “View management of annotations considering the position and shape of moving objects for wearable augmented reality”, [online], September 2007, Proceedings of the 12th Annual Conference of the Virtual Reality Society of Japan, [ Search on December 21, 2012], Internet <URL: http: // yokoya. naist. jp / paper / datas / 1008 / vrsj2007j_makita. pdf>

  In Japanese Patent Application Laid-Open No. 2004-151867, when information handwritten on a paper material by a user is registered on an electronic bulletin board, the user images the paper material including the handwritten information and registers the captured image on the electronic bulletin board. Here, when the user tries to add a comment to the information (handwritten comment) of the handwritten part reflected in the captured image registered in the electronic bulletin board, the information (electronic comment) is additionally input via the information terminal The electronic comment is added to the above-described captured image.

  However, there is a possibility that a plurality of handwritten comments are included in the captured image. In that case, when referring to the electronic comment and the captured image later, it may be difficult to determine which handwritten comment in the captured image corresponds to the additionally input electronic comment. That is, since the information of the part handwritten on the paper material by the user is not handled as one object, the input electronic comment cannot be managed in association with the handwritten comment instead of the captured image. As a result, there is a problem that it is difficult to understand the relationship between the handwritten comment and the electronic comment in the captured image when referring to the comment. In addition, it is not possible to confirm the status of addition of electronic comments from handwritten paper materials.

  On the other hand, when the present applicant takes an image of a paper material using an information terminal such as a mobile phone, the electronic comment is superimposed on the captured image obtained by the imaging to display the addition status of the electronic comment. I'm thinking of making it possible to confirm. On the other hand, in a case where an electronic comment is superimposed and displayed on a captured image obtained by imaging a paper material in this way, the user can add the content of the paper material, the handwritten comment on the paper material, and the superimposed electronic Compare this with the comment. In that case, it is necessary to ensure the visibility of each information. However, if a plurality of feature patterns are detected from the captured image and electronic comments corresponding to the respective feature patterns are displayed in a superimposed manner, there may be cases where the electronic comments are displayed overlapping each other. In addition, even if the electronic comments do not overlap with each other, there are cases in which it is difficult to see the contents of the paper material or the handwritten comments with a plurality of electronic comments. In order to deal with these problems, in the method of Patent Document 1, electronic comments that do not overlap each other are not converted into icons. Therefore, there is a problem that the contents of paper materials and handwritten comments handwritten on the paper materials cannot be visually recognized due to electronic comments that are not converted into icons. Remains. Further, in the method of Non-Patent Document 1, it is difficult to ensure visibility if the number of electronic comments to be superimposed increases due to the limitation of the display area of the display of the mobile terminal.

  In order to solve the above problems, the present invention has the following configuration. That is, in an information processing system, detection means for detecting one or more objects from a captured image, and electronic information input based on a user instruction for one or more objects detected by the detection means A management unit that manages the association, a selection unit that selects at least one object from the objects managed by the management unit, and the electronic information associated with the object selected by the selection unit without iconifying the electronic information. Superimposing display means that superimposes and displays on a captured image, and displays electronic information related to an object not selected by the selection means as an icon and superimposes on the captured image. , Based on the position of the object selected by the selection means in the captured image, the selected object Placing the electronic information associated with the-object.

  When the related electronic information is superimposed and displayed on the object in the captured image, the visibility of the electronic information superimposed on the content of the captured image can be ensured.

The figure which shows the example of the whole structure of an information processing system. The figure which shows the structural example of the component of a system. The figure for demonstrating paper material. The flowchart of the transmission process of the electronic pen locus | trajectory information in an electronic pen. The flowchart of the calculation process of the coordinate which shows the movement locus | trajectory of the electronic pen in a communication terminal. The flowchart of the acquisition process of the handwritten comment locus | trajectory information in a communication terminal. The figure for demonstrating paper material. The figure for demonstrating an electronic bulletin board. The flowchart of the acquisition process of the handwritten comment information in a server. The figure for demonstrating an electronic bulletin board. The flowchart of the feature pattern detection process in a server. The figure for demonstrating an electronic bulletin board. The flowchart of the input process of the electronic input form in an information terminal. The figure for demonstrating the example of a display of an electronic bulletin board. The figure which shows the example of a search screen. The figure which shows the example of a search result. The figure which shows the example of a reference file path input screen. The figure which shows the structural example of thread information. The flowchart of the thread management process in a server. The figure which shows the structural example of feature pattern information. The flowchart of the superimposition display process in an information terminal. The flowchart of the type determination process of the image object in an information terminal. The flowchart of the image object production | generation process in an information terminal. The flowchart of the fixed mode information determination process in an information terminal. The flowchart of the superimposition display process in a server. The figure for demonstrating thread display and icon display. The figure for demonstrating thread display and icon display. The figure for demonstrating the example of a thread | sled display in a different feature pattern position.

<First Embodiment>
Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

[System configuration]
FIG. 1 is a diagram showing an example of a device configuration of an information processing system (electronic bulletin board system) showing a first embodiment of the present invention. In FIG. 1, original data (original image) is printed on paper material 10, and for example, it may be distributed to a plurality of users or posted on a bulletin board. The material of the paper material 10 is not limited to paper, and any material can be used as long as the original data can be printed. The electronic pen 100 is connected to the communication terminal 200 on a one-to-one basis by wireless communication. Communication terminal 200, server 400, and information terminal 300 are communicably connected via a network 50 such as a LAN. The network 50 may have either a wired / wireless configuration. The electronic pen 100 is a device for manually writing characters and figures on the paper material 10, and the trajectory information (electronic pen trajectory information) of the electronic pen 100 operated on the paper material 10 is transmitted to the communication terminal 200. Send.

  The communication terminal 200 extracts trajectory information (handwritten comment trajectory information) of the electronic pen 100 when handwriting is additionally written from the received electronic pen trajectory information, and transmits the extracted information to the server 400.

  The server 400 generates handwritten comment information based on the received handwritten comment trajectory information. Further, the server 400 receives electronic comment information and reference file information from the information terminal 300. In the present embodiment, the original data, handwritten comment information, electronic comment information, and reference file information are collectively referred to as electronic bulletin board information. The electronic bulletin board information is stored and managed by the server 400. Details of the electronic bulletin board information will be described later.

  The information terminal 300 (mobile terminal such as a mobile phone or an electronic tablet) includes a display unit, and displays electronic bulletin board information managed by the server 400 on the display unit. In addition, the information terminal 300 transmits electronic comment information and reference file information input by the user to the electronic bulletin board information displayed on the display unit to the server 400. In addition, the information terminal 300 includes an imaging unit such as a camera, images the paper material 10 by the imaging unit, and displays an image of the paper material 10 on the display unit. At this time, the information terminal 300 can also superimpose and display the electronic bulletin board information on the captured image displayed on the display unit. When the information terminal 300 does not include an imaging unit, an imaging unit such as a camera externally connected by a USB (Universal Serial Bus) cable or the like may be used.

(Electronic pen)
FIG. 2A shows a configuration example of the electronic pen 100 according to the present embodiment. The writing unit 105 is realized by a pen and writes characters and figures on the paper material 10 in a visible state using ink in the ink cartridge. The writing pressure detection unit 104 is realized by a writing pressure sensor, and detects the writing pressure applied to the pen when writing on the paper material 10. The electronic pen data transmission unit 101 is realized by an infrared transmission device that transmits infrared light and an ultrasonic transmission device that transmits ultrasonic waves, and transmits electronic pen trajectory information to the communication terminal 200. The storage unit 103 is realized by a memory such as a RAM or a ROM, and stores temporary data required by the control unit 102 in the memory.

  The control unit 102 performs overall control of the electronic pen 100, and is realized by the CPU reading and executing the program code stored in the storage unit 103. When the writing pressure detection unit 104 detects the start of pressurization, the control unit 102 controls the electronic pen data transmission unit 101 to transmit a pressurization start signal indicating the start of pressurization by infrared rays. In addition, when the writing pressure detection unit 104 detects the end of pressurization, the control unit 102 controls the electronic pen data transmission unit 101 to transmit a pressurization end signal indicating the end of pressurization to infrared rays. In addition, the control unit 102 controls the electronic pen data transmission unit 101 so as to transmit ultrasonic waves at a constant period within a time period from the start of pressurization to the end of pressurization. The ultrasonic signal transmitted by the electronic pen data transmission unit 101 is a signal representing the position information of the electronic pen as will be described later.

(Communication terminal)
FIG. 2B shows a configuration example of the communication terminal 200 according to the present embodiment. The electronic pen data reception unit 204 is realized by a receiver having one infrared reception sensor and two ultrasonic reception sensors. The electronic pen data receiving unit 204 receives electronic pen trajectory information composed of an infrared signal and an ultrasonic signal transmitted from the electronic pen data transmitting unit 101 of the electronic pen 100. The network communication unit 201 is realized by a LAN I / F, and transmits handwritten comment trajectory information to the server 400 via the network 50. The storage unit 203 is realized by a memory such as a ROM, a RAM, and an HDD, and stores temporary data used by the control unit 202, electronic pen locus information, handwritten comment locus information, and the like.

  The control unit 202 performs overall control of each unit of the communication terminal 200, and is realized by the CPU reading and executing the program code stored in the storage unit 203. The control unit 202 performs control so that the electronic pen locus information received by the electronic pen data reception unit 204 is stored in the storage unit 203. The control unit 202 controls the network communication unit 201 to extract handwritten comment trajectory information from the electronic pen trajectory information read from the storage unit 203 and transmit it to the server 400. Furthermore, the control unit 202 receives trajectory information from the electronic pen in accordance with the handwriting action of the user's electronic pen and calculates a paper surface coordinate (FIG. 5) or a flow for extracting a series of handwriting as handwritten comment trajectory information. (FIG. 6) is executed.

(server)
FIG. 2C shows a configuration example of the server 400 according to the present embodiment. The network communication unit 401 is realized by a LAN I / F, and transmits and receives various data between the communication terminal 200 and the information terminal 300 via the network 50. The storage unit 403 is realized by a memory such as a RAM, a ROM, or an HDD, and stores the electronic bulletin board information in the present embodiment. Further, the network communication unit 401 receives the comment information and stores it in the storage unit 403. The electronic bulletin board information includes original data having the same content as that of the image printed on the paper, thread information, temporarily stored data, and the like.

  The control unit 402 performs overall control of each unit of the server 400, and is realized by the CPU reading and executing the program stored in the storage unit 403. Further, the control unit 402 includes a comment type determination unit 411, a handwritten comment input unit 412, a thread management unit 413, a print layout unit 414, a feature extraction unit 415, and a feature pattern management unit 416.

  The comment type determination unit 411 determines which type of comment information received from the network communication unit 401 is handwritten comment information, electronic comment information, or reference file information. In the present embodiment, the server 400, the information terminal 300, and the communication terminal 200 communicate using the HTTP (Hyper Text Transfer Protocol) protocol. In addition, in the transmission / reception of the comment information, the comment type identifier (comment identifier) is also transmitted / received. By referring to the identifier of this comment type, the comment type of the received comment information is determined.

  When the received comment information belongs to the comment type of the handwritten comment information, the received comment information is handwritten comment trajectory information. Therefore, the handwritten comment input unit 412 generates handwritten comment information by converting the handwritten comment trajectory information into a bitmap, and stores it in the storage unit 403. The generated handwritten comment information is managed by the thread management unit 413.

  When the received comment information belongs to the comment type of electronic comment information or reference file information, the received comment information is electronic comment information or reference file information. The electronic comment information or reference file information is managed by the thread management unit 413.

  The thread management unit 413 generates thread information in which handwritten comment information, electronic comment information, and reference file information are associated, and stores the thread information in the storage unit 403. As shown in FIG. 18, thread information is electronic information including handwritten comment information, electronic comment information, and reference file information, and is associated with a feature pattern (object). Further, the parent-child relationship and the positional relationship when each information is input are shown. Each information can be added, and the thread management unit 413 updates the thread information according to the addition of handwritten comment information, electronic comment information, or reference file information.

  The print layout unit 414 generates a print layout of the electronic bulletin board information when a print request for the electronic bulletin board information is received from the information terminal 300. In the present embodiment, the print layout unit 414 prints an electronic bulletin board display result in which thread information stored in the storage unit 403 is superimposed on original data 700 described later. Printing is realized by issuing a print instruction to a printer (not shown) connected to the information terminal 300 using a printer driver installed in the information terminal 300. That is, the information terminal 300 includes a print instruction unit (not shown) that instructs printing of the electronic bulletin board information.

  The feature extraction unit 415 extracts feature patterns such as text patterns and graphic patterns from the original data 700. In the present embodiment, a known feature point detection technique and a feature quantity extraction technique are used for feature pattern extraction. For example, the feature extraction unit 415 detects feature points in the image around the designated comment, and when the number of feature points is larger than a predetermined threshold, the feature extraction unit 415 extracts a rectangular area including these feature points as a feature pattern. At the same time, the feature amount is calculated. The position of this feature pattern is defined as the coordinates of the upper left corner of the rectangular area including the feature point. The feature pattern detection method is not limited to the above method, and other methods may be used. The feature pattern may be detected as an object instead of a rectangular area.

  The feature pattern management unit 416 generates a feature pattern information table and stores it in the storage unit 403. As shown in FIG. 20, the feature pattern information table is a data table that holds a list of feature pattern information indicating information for identifying the positions of feature patterns and feature points themselves. The feature pattern management unit 416 updates the feature pattern information table every time a new thread is added.

(Information terminal)
FIG. 2D shows a configuration example of the information terminal 300. The network communication unit 301 is realized by a LAN I / F, and transmits / receives various data to / from the server 400 via the network 50. The storage unit 303 is realized by a memory such as a RAM, a ROM, or an HDD, and stores temporarily stored data used by the control unit 302 in the present embodiment.

  The display unit 304 is realized by a display device typified by a display (including a touch panel) and displays electronic bulletin board information, an image captured by the imaging unit 306, and the like. The input unit 305 is realized by an input device represented by a touch panel, a mouse, a keyboard, and the like, and receives an input operation from a user. The imaging unit 306 is realized by an imaging unit represented by a camera or the like, and acquires a captured image of the paper material 10.

  The control unit 302 is realized by comprehensively controlling each unit of the information terminal 300, and the CPU reading and executing the program stored in the storage unit 303. Further, the control unit 302 includes an electronic bulletin board information display unit 311, an electronic comment input unit 312, a reference file input unit 313, a search unit 314, and a superimposition processing unit 315. The electronic bulletin board information display unit 311 displays the electronic bulletin board information acquired from the server 400 on the display unit 304. The electronic bulletin board information display unit 311 displays an electronic input form for accepting an input of an electronic comment (character string) from the user. Details of the electronic input form will be described later.

  The electronic comment input unit 312 acquires an electronic comment input by the user using the input unit 305 to the electronic input form, and transmits the electronic comment to the server 400 as electronic comment information. The reference file input unit 313 acquires a path (reference file path) to a reference file input by the user using the input unit 305 with respect to the electronic input form, and transmits it to the server 400 as reference file information. The search unit 314 performs a search process based on a search key input by the user using the input unit 305 to the electronic input form. The superimposition processing unit 315 displays the image of the paper material 10 captured by the imaging unit 306 on the display unit 304 and displays the electronic bulletin board information superimposed on the image of the paper material.

(Handwritten comment input)
FIG. 3 is a view for explaining the paper surface of the paper material 10 in the present embodiment. An object 110 is printed on the paper material 10 as the contents of the original data.

In the present embodiment, the coordinate system of the paper surface is expressed by (X _paper , Y _paper ), and the upper center of the paper material 10 is the paper surface origin 260 (0, 0). This coordinate system is recognized by a receiver that implements the electronic pen data receiving unit 204. This ensures consistency with the electronic bulletin board with the upper center of the original data 700 described later as the coordinate origin. The paper material 10 has positions (−X _p , 0), (X _p , 0), (−X _p , Y _p ), and (X _p , Y _p ) in the coordinate system of the paper surface.

  The user checks the handwritten comment start area 251 when starting writing a handwritten comment on the paper material 10 with the electronic pen 100. Next, the user writes “ABC” which is a handwritten comment. In FIG. 3, 211 to 216 indicated by “◯” correspond to pressurization start points for the comment “ABC” detected by the writing pressure detection unit 104. Further, 221 to 226 indicated by “□” correspond to the pressurization end point for the comment “ABC” detected by the writing pressure detection unit 104. It is assumed that the user checks the handwritten comment end area 252 after the handwritten comment has been written.

[Processing flow]
(Electronic pen locus information transmission processing in the electronic pen)
FIG. 4 shows a flowchart in which the electronic pen 100 transmits a series of electronic pen locus information to the communication terminal 200. This flowchart is realized by the CPU of the electronic pen 100 reading and executing a program stored in a ROM or the like.

  In step S <b> 1001, the control unit 102 of the electronic pen 100 determines whether the pressure detection unit 104 has detected the start of pressurization. When the pressurization start is detected (YES in S1001), the process proceeds to S1002, and the control unit 102 controls the electronic pen data transmission unit 101 to transmit a pressurization start signal with infrared rays. In S1003, the control unit 102 transmits an ultrasonic signal according to the pressurization state. The control unit 102 continues to transmit ultrasonic signals at a constant period until the writing pressure detection unit 104 detects the end of pressurization (NO in S1004). When the end of pressurization is detected (YES in S1004), in S1005, the control unit 102 controls the electronic pen data transmission unit 101 to transmit a pressurization end signal with infrared rays. An ultrasonic signal from when the pressurization start signal is transmitted until when the pressurization end signal is transmitted is a series of electronic pen trajectory information.

(Paper coordinate calculation process in communication terminal)
FIG. 5 is a flowchart in which the communication terminal 200 receives a series of electronic pen locus information from the electronic pen 100 and calculates coordinates (paper surface coordinates) indicating the movement locus of the electronic pen. This flowchart is realized by the CPU of the communication terminal 200 reading and executing a program stored in a ROM or the like.

  In step S2001, the control unit 202 of the communication terminal 200 determines whether the electronic pen data reception unit 204 has received an infrared press start signal. If a pressurization start signal is received (YES in S2001), the process proceeds to S2002. In step S2002, the control unit 202 determines whether the electronic pen data reception unit 204 has received an ultrasonic signal. If an ultrasonic signal has not been received (NO in S2002), the process proceeds to S2004.

  When the ultrasonic signal is received (YES in S2002), in S2003, the control unit 202 determines from the difference in reception time of the ultrasonic signals received by the two ultrasonic reception sensors and the distance between the two ultrasonic sensors. The paper surface coordinates of the electronic pen are calculated. The paper plane coordinates will be described later with reference to FIG.

  In step S2004, the control unit 202 determines whether the electronic pen data reception unit 204 has received an infrared pressurization end signal. If the pressurization end signal has not been received (NO in S2004), the process returns to S2002. When a pressurization end signal is received (YES in S2004), control unit 202 ends the acquisition of a series of pen trajectory information from the start of pressurization to the end of pressurization. That is, the control unit 202 sequentially calculates the paper surface coordinate information indicating the movement locus of the electronic pen from the electronic pen locus information continuously received by executing the processing flow of FIG. In the following description, the information on the paper plane coordinates indicating the movement locus of the electronic pen calculated in S2003 is also referred to as electronic pen locus information.

(Handwritten comment trajectory information extraction process in communication terminal)
FIG. 6 is a flowchart for extracting the handwritten comment “ABC” of FIG. 3 as one handwritten comment trajectory information in the communication terminal 200. This flowchart is realized by the CPU of the communication terminal 200 reading and executing a program stored in a ROM or the like. That is, handwritten comment trajectory information is extracted from a series of electronic pen trajectory information by the control unit 202 executing the processing flow of FIG.

  In step S2011, the control unit 202 acquires electronic pen trajectory information including information on the paper surface coordinates of the electronic pen 100 calculated in the processing flow of FIG. In S2012, control unit 202 determines whether the handwritten comment start flag is ON. The handwritten comment start flag is stored in the storage unit 203, and the initial value is OFF.

  When the handwritten comment start flag is OFF (NO in S2012), in S2013, the control unit 202 determines whether or not the paper surface coordinates of the electronic pen 100 indicated by the electronic pen locus information are within the handwritten comment start area 251. If the paper surface coordinates of the electronic pen 100 are not within the handwritten comment start area 251 (NO in S2013), the process returns to S2011. When the paper surface coordinates of the electronic pen 100 are within the handwritten comment start area 251 (YES in S2013), the control unit 202 sets the handwritten comment start flag to ON in S2014, and returns to S2011.

  If the handwritten comment start flag is ON (YES in S2012), in S2015, the control unit 202 determines whether or not the paper surface coordinates of the electronic pen 100 indicated by the electronic pen trajectory information are within the handwritten comment end area 252. If the paper surface coordinates of the electronic pen 100 are not within the handwritten comment end area 252 (NO in S2015), the control unit 202 stores the electronic pen trajectory information in the storage unit 203 as part of the handwritten comment trajectory information in S2016. , Return to S2011. If the paper surface coordinates of the electronic pen are within the handwritten comment end region 252 (YES in S2015), the control unit 202 sets the handwritten comment start flag to OFF in S2017.

  In S2018, the control unit 202 transmits a series of electronic pen trajectory information groups from the start to the end of the handwritten comment stored in the storage unit 203 in S2016 to the server 400 as one handwritten comment trajectory information (for example, “ABC”). . At the time of this transmission, the control unit 202 transmits the handwritten comment trajectory information including a handwritten comment identifier indicating that the type of information belongs to the handwritten comment information.

  FIG. 7 is a diagram for explaining handwritten comment trajectory information that is a result of the paper surface coordinate calculation described in FIG. 3. In FIG. 7, coordinates indicated by “●” such as coordinates 231 and 232 are examples of coordinates obtained as a result of the paper surface coordinate calculation. Then, as a result of sequentially performing the paper surface coordinate calculation on the pressurization end points 221 to 226 corresponding to the pressurization start points 211 to 216 shown in FIG. 3, a series of comment trajectory information as handwritten comment trajectory information “ABC”. 240 groups are obtained.

  FIG. 8 is a diagram illustrating the original data 700 corresponding to the paper material 10 in the present embodiment. The storage unit 403 stores original data 700. In the present embodiment, the original data 700 is original data of an image printed on the paper material 10, and the paper material 10 is printed by performing a printing process on the storage medium using the original data 700. The

In the present embodiment, the coordinate system of the electronic bulletin board is represented by (X _bbs , Y _bbs ), and the upper center of the original data 700 is the electronic bulletin board origin 701 (0, 0). The electronic bulletin board origin 701 in the original data 700 corresponds to the paper origin 260 in the paper material 10 of FIG. The original data 700 is a rectangular area composed of four points (−X _o , 0), (X _o , 0), (−X _o , Y _o ), and (X _o , Y _o ) in the coordinate system of the electronic bulletin board. It is expressed by

(Handwritten comment information acquisition processing in the server)
FIG. 9 is a flowchart for acquiring handwritten comment information by converting the handwritten comment locus information into a bitmap by the handwritten comment input unit 412 of the server 400. This flowchart is realized by the CPU of the server 400 reading and executing a program stored in a ROM or the like. Here, “ABC” which is the handwritten comment locus information 240 of FIG. 3 will be described as an example. The storage unit 403 stores original data 700.

In S4001, the handwritten comment input unit 412 receives the handwritten comment trajectory information 240 and the handwritten comment identifier transmitted by the communication terminal 200 in S2018 of FIG. The handwritten comment trajectory information 240 includes coordinate information of a handwritten comment expressed in a _paper plane coordinate system (X _paper , Y _paper ). In S4002, the handwritten comment input unit 412 converts the coordinate information of the handwritten comment expressed in the paper surface coordinate system included in the received handwritten comment locus information 240 into the coordinate system ( _Xbbs , _Ybbs ) of the electronic bulletin board. The conversion process is realized by determining conversion coefficients in advance based on the size of the paper material 10 and the size of the original data 700. FIG. 10 is an example showing a result of converting the handwritten comment trajectory information 240 into the coordinate system (X _bbs , Y _bbs ) of the electronic bulletin board, and coordinates represented by coordinates 711 and 712 are obtained.

In S4003, the handwritten comment input unit 412 calculates the minimum rectangular size 720 of the rectangle that includes the coordinate group of the handwritten comment trajectory information subjected to coordinate conversion. Then, the handwritten comment input unit 412 obtains the upper left coordinates (−X _w1 , Y _w1 ) of the minimum rectangle and the lower right coordinates (X _w2 , Y _w2 ) of the minimum rectangle.

In S4004, the handwritten comment input unit 412 gives a color to the image data at the coordinates of the handwritten comment trajectory information subjected to the coordinate conversion, and renders it as a bitmap. The handwritten comment input unit 412 generates a comment 721 having a rectangular size 720 obtained by converting the handwritten comment into a bitmap, and stores the comment 721 in the storage unit 403. Then, the handwritten comment input unit 412 performs thread management using the file path to the comment 721 stored in the storage unit 403, the upper left coordinates (−X _w1 , Y _w1 ) of the bitmap, and the handwritten comment identifier received in S4001 as handwritten comment information. To the unit 413.

(Feature pattern detection processing in the server)
FIG. 11 is a flowchart for detecting and registering a feature pattern existing in the vicinity region of a comment. This flowchart is realized by the CPU of the server 400 reading and executing a program stored in a ROM or the like.

  For example, when this flowchart is executed, when the comment and the feature pattern 710 are in the positional relationship as shown in FIG. 14B, the object in the lower right region when viewed from the head comment “ABC” is detected as the feature pattern 710. . This process is performed by the feature pattern management unit 416 of the server 400, and is performed when a new thread is added in the thread management process of FIG. 19 to be described later (S4021). Further, thread information of a newly added thread is given as an input at this time.

In step S4061, the feature pattern management unit 416 uses the center position of the head comment (corresponding to “ABC” in this case) of the designated thread as the origin as shown in FIG. Divide into crosses. Subsequently, the feature pattern management unit 416 defines a rectangular area 717 having a search distance (W _s , H _s ) with the origin 716 as one vertex for each divided area as a search area. Here, the initial value of the search distance (W _s , H _s ) is (0, 0).

In S4062, the feature pattern management unit 416 increases the search distance (W _s , H _s ) by a predetermined size for each divided region, and increases the rectangular region 717. In S4063, the feature pattern management unit 416 passes the image of each search area to the feature extraction unit 415, and determines whether or not a feature pattern is included in each divided area (search area).

  If the feature pattern is not detected (NO in S4063), the feature pattern management unit 416 determines in S4064 whether or not the document edge has been examined. If the document edge has been examined (YES in S4064), this process ends. On the other hand, if the document edge has not been checked (NO in S4064), the process proceeds to S4062.

  When the feature pattern is detected (YES in S4063), in S4065, the feature pattern management unit 416 adds information on the detected feature pattern to the feature pattern information table (FIG. 20).

FIG. 20 shows a configuration example of the feature pattern information table. The relative position in the x direction of the feature pattern information table 2601 is the relative position (X _m ) from the comment coordinate (X _com ) in the X coordinate of the first comment to the x coordinate of the feature pattern. The relative position in the y direction of the feature pattern information table 2601 is a relative body (Y _m ) from the comment coordinate (Y _com ) in the Y coordinate of the first comment to the y coordinate of the feature pattern. The value of the related thread ID is the thread ID of the new thread included in the thread information given as input. The comment coordinates ( _Xcom , _Ycom ) of the head comment may be the center coordinates in the rectangular area of the head comment, or may be any of the four ends. Here, the coordinates of the upper right corner of the first comment are used.

(Electronic input form)
FIG. 12 is a diagram for explaining an electronic input form. FIG. 12 shows an example in which the electronic bulletin board information received from the server 400 is displayed on the electronic bulletin board information display unit 311 when the information terminal 300 receives the electronic bulletin board information stored in the storage unit 403 of the server 400.

Here, when the user instructs to select the coordinates in the area where the electronic bulletin board information displayed on the display unit 304 is displayed via the input unit 305, the electronic bulletin board information display unit 311 displays the electronic input form 730. FIG. 12 shows that a coordinate in the area of the handwritten comment 721 has been selected. In the present embodiment, the electronic bulletin board information display unit 311 displays the electronic input form 730 at an offset position (−X _w1 + Δx1, Y _w1 + Δy1) shifted by (Δ _x1 , Δy1) from the coordinates of the handwritten comment 721. When the user instructs to select coordinates outside the area of the handwritten comment 721 via the input unit 305, the electronic bulletin board information display unit 311 displays the electronic input form 730 at a position corresponding to the selected coordinates.

  The electronic input form 730 includes a character string input area 731, an electronic comment addition button 732, a keyword search button 733, an image search button 734, a moving image search button 735, and a reference file attachment button 736.

(Input processing in information terminals)
FIG. 13 is a flowchart of input processing using an electronic input form. This flowchart is realized by the CPU of the information terminal 300 reading and executing a program stored in a ROM or the like.

  In step S <b> 3011, the electronic bulletin board information display unit 311 detects whether or not the user has instructed selection of coordinates in the area displayed on the display unit 304 via the input unit 305. When a coordinate selection instruction is detected (YES in S3011), electronic bulletin board information display unit 311 determines whether or not the selected coordinate is within the area of the electronic bulletin board information. When the coordinate in the area of the electronic bulletin board information is selected, the electronic bulletin board information display unit 311 displays the electronic input form 730 at the offset position in step S3012. When a coordinate outside the area of the electronic bulletin board information is selected, the electronic bulletin board information display unit 311 displays the electronic input form 730 at a position corresponding to the selected coordinate. At this time, the electronic comment input unit 312 stores the coordinates of the position where the electronic input form 730 is displayed in the storage unit 303.

  In step S <b> 3013, the electronic comment input unit 312 determines whether the user has entered characters into the character string input area 731 via the input unit 305. When a character is input (YES in S3013), electronic comment input unit 312 stores the input character string in storage unit 303. In step S3014, the electronic comment input unit 312 determines whether the electronic comment addition button 732 has been pressed.

  When the add electronic comment button 732 is pressed (YES in S3014), the electronic comment input unit 312 is input to the coordinates and character string input area 731 of the electronic input form 730 stored in the storage unit 303 in S3015. A character string is transmitted to the server 400 as electronic comment information. When transmitting electronic comment information, the electronic comment input unit 312 transmits the electronic comment information including an electronic comment identifier indicating that the comment information to be transmitted is information of a type belonging to the electronic comment information. Then, it progresses to S3021.

  If the electronic comment addition button 732 has not been pressed (NO in S3014), in S3016, the electronic comment input unit 312 presses any of the keyword search button 733, the image search button 734, or the video search button 735 by the user. It is determined whether or not it has been done. If any one of the search buttons is pressed (YES in S3016), in S3017, the search unit 314 uses the Web search engine assigned to the search button to search for the character string input in the character string input area 731. Search for. Since this well-known search technique is used for the search of this character string, the detailed description here is omitted.

  After the search process, the electronic bulletin board information display unit 311 displays the search result of the input character string using the search engine selected by the user by button selection on the display unit 304. In step S3018, the search unit 314 stores the search result of the search process in step S3017 in the storage unit 303. The electronic bulletin board information display unit 311 can hide the search engine screen by pressing a non-display button on the search engine screen (FIG. 15). Thereafter, the process returns to S3013.

  If no character is input in character string input area 731 (NO in S3013), in S3019, electronic bulletin board information display unit 311 determines whether or not reference file attachment button 736 is pressed by a user instruction. When reference file attachment button 736 is pressed (YES in S3019), bulletin board information display unit 311 displays a reference file path input screen (FIG. 17) in S3020. Then, the reference file input unit 313 stores the reference file path input on the reference file path input screen in the storage unit 303. Reference files include image files, moving image files, document files, audio files, and electronic files represented by search results stored in the search unit 314.

  Further, the reference file input unit 313 calculates the coordinates of the position where the reference file access button for referring to the reference file is to be displayed based on the coordinates of the position where the electronic input form is displayed. In the present embodiment, the specifically calculated coordinates are coordinates of a position obtained by shifting the coordinates of the electronic input form by (Δx2, 0).

  In step S <b> 3020, the reference file input unit 313 transmits the reference file path and reference file access button coordinate information to the server 400 as reference file information. At the time of this transmission, the reference file input unit 313 transmits the reference file information including a reference file identifier indicating that the file information to be transmitted is information of a type belonging to the reference file information. The processing up to this point is the processing of S3020. The reference file access button is an image object selected by a user instruction to refer to the reference file. FIG. 14B shows a reference file access button 742 as an example.

  If reference file attachment button 736 has not been pressed (NO in S3019), the process returns to S3013.

In step S3021, the electronic bulletin board information display unit 311 hides the electronic input form 730. In S3022, the electronic bulletin board information display unit 311 displays the input result at the coordinates where the electronic input form 730 is displayed. Here, when the handwritten comment 721 is selected by a user instruction and the electronic input form 730 is displayed, electronic comment information is displayed if the input content to the electronic input form is an electronic comment. For example, when the handwritten comment 721 in FIG. 14A is selected, the electronic comment 741 (character string “DEF”) is set to (−X _w1 + Δx1, Y _w1 + Δy1) at the same coordinates as the coordinates of the electronic input form 730. Is displayed.

When the handwritten comment 721 is selected by a user instruction and the electronic input form 730 is displayed, if the input content to the electronic input form 730 is an attached file, a reference file access button 742 is displayed. For example, when the handwritten comment 721 in FIG. 14A is further selected, the reference file access button 742 is a coordinate that is shifted by a predetermined position from the coordinate at which the electronic input form 730 is displayed as shown in FIG. (−X _w1 + Δx1 + Δx2, Y _w1 + Δy1).

  FIG. 15 is a diagram for explaining the search engine screen 751. The search engine screen 751 displays the character string input in the character string input area 731 of the electronic input form 730 shown in FIG. The search result is displayed in the search result column 753. The next button 754 is used when displaying the search result of the next page. The non-display button 755 is used when closing the search engine screen.

  FIG. 16 is a diagram for explaining the details of the search result column 753 shown in FIG. A search result 754 is a display example of a keyword search result. A search result 755 is a display example of an image search result. A search result 756 is a display example of a moving image search result. The user can select a file to be attached to the electronic bulletin board from the files obtained as a search result.

  FIG. 17 is a diagram for explaining an example of the reference file path input screen 761 displayed in S3020 of FIG. The reference file path input field 762 is an area for inputting a reference file to be attached. The decision button 763 is used to decide to attach the reference file input in the reference file path input field 762. A cancel button 764 is used to close the reference file path input screen.

  In the present embodiment, the reference file path is a path to a file stored in the storage unit 303 of the information terminal 300. However, the file may be uploaded (transmitted) from the information terminal 300 to the server 400 along with the transmission of the reference file path. In this case, when the reference file path is transmitted to the server 400, the reference file input unit 313 uploads the reference file path to the storage unit 403 of the server 400, converts the reference file path into the upload destination path of the reference file, and transmits the reference file path. . As a result, the reference file is uploaded to the server 400 and the path to the file is stored, so that the reference file can be referenced from any information terminal 300.

(Thread information)
FIG. 18 shows a configuration example of thread information managed by the thread management unit 413 in the server 400. The thread management unit 413 manages a plurality of thread information. One thread information is expressed by a tree structure including one thread header and one or more comment information or zero or more reference file information.

The thread header 801 includes the following information.
Thread ID: Information for uniquely identifying a thread First comment ID: Comment ID of comment information at the head of the thread
Thread coordinates (X _thread ): X _thread coordinates of the thread information displayed in the electronic bulletin board coordinate system (X _bbs , Y _bbs ) Thread coordinates (Y _thread ): The thread information of the thread information displayed in the electronic bulletin board coordinate system Y _thread coordinates thread size _{(X SIZEthread): X bbs} axis direction size thread size of the thread to be displayed by the electronic bulletin board coordinate system _{(Y SIZEthread):} the thread _{Y bbs} shaft which is displayed on the electronic bulletin board coordinate system Direction size

The comment information 802 includes the following information.
Comment ID: Information for uniquely identifying a comment Parent comment ID: Comment ID of comment information that is a parent node of the comment. If the comment is the top comment, there is no parent node.
Child comment ID: Comment ID of comment information that is a child node of the comment. The parent comment information can have a plurality of child comment information.
Reference file ID: A reference file ID of a reference file related to the comment information. The comment information can have a plurality of reference file information.
Type: Comment identifier indicating whether the comment information type is a bitmap that is a handwritten comment or a character string that is an electronic comment. Data: a bitmap file name for handwritten comment information, and a character string data for electronic comment information.
Comment coordinates (X _com ): Coordinates in the X _bbs axis direction of the comment information displayed in the electronic bulletin board coordinate system Comment coordinates (Y _com ): Y _bbs axis directions in the comment information displayed in the electronic bulletin board coordinate system Coordinate comment size (X _SIZEcom ): size in the X _bbs axis direction of the comment displayed in the electronic bulletin board coordinate system Comment size (Y _SIZEcom ): Y _bbs axis direction of the comment displayed in the electronic bulletin board coordinate system Size. The comment size (X _SIZEcom ) and the comment size (Y _SIZEcom ) are used when the comment information type is a bitmap. When the type is a character string, the character string size is calculated from the character size defined on the server 400 side.
Superimposition target: Information indicating whether or not the comment information is to be superimposed. The server 400 may be set in advance as a superimposed display target in advance, or may be set to be a superimposed display target from the first comment ID up to N. The maximum number N may be set by a numerical value input based on a user instruction. In the comment information 806 in FIG. 18, N = 3.

The reference file information 803 includes the following information.
Reference file ID: Information for uniquely identifying reference file information Parent comment ID: Comment ID of comment information that is a parent node of the reference file information
Type: Information indicating the type of the reference file information. Examples of types include moving images, audio, documents, drawings, URLs, shortcuts, and the like. It can be played back with an application corresponding to the type.
Data: Reference file path of reference file information Access button coordinates (X _acc ): Coordinate in the _Xbbs axis direction of the access button of the reference file information displayed in the electronic bulletin board coordinate system Access button coordinates (Y _acc ): Electronic bulletin board coordinate access button size in Y _bbs axis direction of the access button of the reference file information displayed in the coordinate system (X _SIZEacc): X _bbs axis direction of the access button of the reference file information displayed in the electronic bulletin board coordinate system size access button size _{(Y SIZEacc):} the size of the _{Y bbs} axis direction of the access button of the reference file information displayed in the electronic bulletin board coordinate system

  A thread 804 in FIG. 18 is an example representing a tree structure of thread information. In this example, the thread 804 is configured based on one thread header, five comment information, and three reference file information. The comment information of comment ID = 3 has two child comment information of comment ID = 4 and comment ID = 5. The comment information with comment ID = 4 has two pieces of reference file information with reference file ID = 2 and reference file ID = 3.

  The thread header 805 is an example representing the thread header of the thread 804. The comment information 806 is an example representing a list of comment information of the thread 804. Reference file information 807 is an example representing a list of reference file information of the thread 804. In these examples, parent comment ID = 0 means head comment information that does not have parent comment information. Child comment ID = 0 means that it is the comment information at the end that does not have child comment information. Reference file ID = 0 means no reference file information. Superimposition target = 1 means to be a superimposition target. Superimposition target = 0 means that it is not a superposition target.

(Thread management processing in the server)
FIG. 19 is a flowchart of thread management processing in the server 400. This flowchart is realized by the CPU of the server 400 reading and executing a program stored in a ROM or the like.

  In step S4011, the thread management unit 413 determines whether a comment has been added. Specifically, when the thread management unit 413 receives comment information (handwritten comment information and electronic comment information) or reference file information, it is determined that a comment has been added.

  If it is determined that a comment has been added (YES in S4011), in S4012, the thread management unit 413 determines whether it is necessary to generate a new thread. Specifically, when the coordinate information included in the comment information is not information indicating the display area of the comment information 802 in the thread information stored in the storage unit 303 or the reference file access button area of the reference file information 803. It is determined that the generation of a new thread is necessary. When the coordinate information included in the comment information is information indicating the display area of each comment information in the thread information stored in the storage unit 303 or the area of the reference file access button, generation of a new thread is unnecessary. Determined.

  If it is determined that a new thread needs to be generated (YES in S4012), the thread management unit 413 generates a new thread by generating a new thread header 801 in S4013. Then, the process proceeds to S4014. In this case, the subsequent processing executed by the thread management unit 413 is processing related to information on the generated new thread. If it is not determined that a new thread needs to be generated (NO in S4012), the process proceeds to S4014. In this case, subsequent processing executed by the thread management unit 413 is processing related to information of the corresponding existing thread.

  In S4014, the thread management unit 413 determines whether the input comment information is handwritten comment information. Specifically, the comment type determination unit 411 determines whether the comment information is handwritten comment information by referring to the comment identifier included in the input comment information, and transmits the determination result to the thread management unit 413. To do. Then, the thread management unit 413 determines whether the input comment information is handwritten comment information based on the received determination result. When the input comment information is handwritten comment information (YES in S4014), in S4015, the thread management unit 413 converts the input comment information into a thread information format so that the input comment information can be displayed as a thread as handwritten comment information. . Then, the process proceeds to S4019. Details of the thread display will be described later.

  If it is not handwritten comment information (NO in S4014), in S4016, the thread management unit 413 determines whether the input comment information is electronic comment information. Specifically, the comment type determination unit 411 determines whether the comment information is electronic comment information by referring to the comment identifier included in the input comment information, and transmits the determination result to the thread management unit 413. To do. Then, the thread management unit 413 determines whether the input comment information is electronic comment information based on the received determination result. If the input comment information is electronic comment information (YES in S4016), in S4017, the thread management unit 413 converts the input comment information into a thread information format so that it can be displayed as a thread as electronic comment information. . Then, the process proceeds to S4019.

  If it is not electronic comment information (NO in S4016), since the input information is reference file information, in S4018, the thread management unit 413 converts the reference file information into a thread information format so that the thread can be displayed. Then, the process proceeds to S4019.

  In S4019, the thread management unit 413 adds comment information or reference file information converted into the thread information format (see 802 and 803 in FIG. 18) to the thread information. Further, the thread management unit 413 updates the thread header and stores the thread information in the storage unit 403.

  In step S4020, the thread management unit 413 determines whether the currently processed thread is a new thread. If it is a new thread (YES in S4020), in S4021, the thread management unit 413 detects a feature pattern existing in the vicinity of the new thread. Specifically, the feature pattern detection flow of FIG. 11 is performed with the thread information of the new thread as input.

  Note that the thread information managed by the thread management unit 413 may be read from the storage unit 403 of the server 400 by the information terminal 300 in some cases. The original data 700 stored in the storage unit 403 may also be read by the information terminal 300 and displayed on the display unit 304 with thread information added. In this case, the server 400 transmits the thread information stored in the storage unit 403 and the original data 700 to the information terminal 300. Then, the control unit 302 of the information terminal 300 displays the original data 700 in the electronic bulletin board coordinate system, and displays the thread information superimposed on the original data 700.

  Specifically, the control unit 302 of the information terminal 300 acquires comment information and reference file information used for displaying one thread by the thread header of the thread information. Then, the control unit 302 generates an image object of a bitmap, a character string, or a reference file access button based on the acquired comment information and reference file data and type. Further, the control unit 302 acquires coordinate information (comment coordinates, access button coordinates) and size information (comment size, access button size) of the comment information and the reference file information.

  Then, based on the acquired coordinate information and size information, the control unit 302 generates an image obtained by combining the image object on the original data 700 so that the generated image object is superimposed on the original data 700. Then, the control unit 302 displays the synthesized image on the display unit 304. The image data obtained by combining is an image in which the positional relationship of the input or display of each information when the information of thread information (comment information and reference file information) is input is maintained as one thread. Displayed. Such a display is called a thread display. A method of displaying thread information as an icon for thread display is called icon display. Each display format is collectively described as a display type.

  When the thread information (thread header 805, comment information 806, reference file information 807) shown in FIG. 18 is displayed as a thread in the original data 700, the displayed image is an image as shown in FIG. That is, when the information terminal 300 reads the thread information together with the original data 700 and displays the thread, the information terminal 300 superimposes all the thread information on the original data 700 because there is no limit on the size of the displayed screen. Display thread.

  In such a case, the superimposition processing unit 315 thread-displays the “superimposition target” information of the comment information as “superimposition target = 1” for all the thread information. A configuration may be adopted in which whether to display all of the thread information or a part of the thread information (for example, N from the first comment) is switched based on a user instruction.

(Feature pattern information)
FIG. 20 is a diagram for describing feature pattern information managed by the feature pattern management unit 416 in the server 400. The feature pattern management unit 416 manages a plurality of feature pattern information. Specifically, individual feature pattern information is managed as a feature pattern information table configured as a record.

One feature pattern information includes the following information.
Feature pattern ID: Information for uniquely identifying a feature pattern Feature point information path: Path to a file storing feature point information including feature quantity information of a feature point group of feature patterns X-direction relative position (X _m ): Relative position information in the x-axis direction of the feature pattern from the thread associated with the feature point y-direction relative position (Y _m ): Relative position information in the y-axis direction of the feature pattern from the thread associated with the feature point Related thread ID : Thread ID of the thread associated with the feature pattern

  A feature pattern information table 2602 is an example of a feature pattern information table having, as records, feature pattern information associated with the thread 804 shown in FIG.

(Superimposed display)
In the electronic bulletin board system according to the present embodiment, the handwritten comment input to the paper material 10 is converted into a bitmap. The bitmapped handwritten comment is displayed on the display unit 304 of the information terminal 300 together with the original data 700. Then, the user who added the handwritten comment to the handwritten comment or another user can add the electronic comment via the information terminal 300.

  In such an electronic bulletin board system, a user who has manually added information to the paper material 10 may want to see comments from other users regarding the handwritten comment. Alternatively, the user who has distributed the paper material 10 before the handwritten comment is added may want to immediately see the handwritten comment or the electronic comment already added by another user. In order to meet such a use case, when the paper material 10 is imaged by the imaging unit 306 of the information terminal 300, an image in which a handwritten comment or an electronic comment is superimposed on the captured image is displayed on the display unit 304 in real time. The process of updating to will be described with reference to FIGS.

  FIG. 21 is a flowchart of processing of the information terminal 300 regarding superimposed display. This flowchart is realized by the CPU of the information terminal 300 reading and executing a program stored in a ROM or the like. With this processing, the information terminal 300 uses the thread information stored in the storage unit 303 for the image of the paper material 10 captured by the imaging unit 306 and displayed on the display unit 304, and the feature pattern on the image of the paper material 10 Based on the position, it can be displayed superimposed.

  In S3050, the superimposition processing unit 315 downloads the feature pattern information table 2601 (a) from the server 400 and stores it in the storage unit 303.

  In step S <b> 3051, the imaging unit 306 captures the paper material 10 and transmits the captured image (captured image 3001 in FIG. 27) to the superimposition processing unit 315 of the control unit 302. Then, the superimposition processing unit 315 of the control unit 302 determines whether a feature pattern is detected from the captured image 3001. The feature pattern is a pattern (object) that enables the correspondence between the paper surface coordinate system and the electronic bulletin board coordinate system when the pattern is detected. Thereby, a bitmap of comment information managed in the electronic bulletin board coordinate system and an image object such as a character string or a reference file access button can be displayed superimposed on an appropriate position in the captured image 3001.

  In order to detect a specific feature pattern from the captured image 3001, a known feature point detection technique and feature quantity extraction technique are used in the present embodiment (for example, SURF (Speed-Up Robust Features)). The superimposition processing unit 315 extracts feature amount information from the captured image using these known techniques, and compares it with the feature amount information of each feature pattern in the feature pattern information table stored in the storage unit 303. The feature pattern is detected. In the present embodiment, the feature value comparison method is not limited, and an example is the use of RANSAC (RANdom Sample Consensus).

  Along with the detection of the feature pattern, the enlargement ratio of the feature pattern on the captured image with respect to the feature pattern on the original data is calculated. The enlargement ratio is the size on the screen relative to the size of the feature pattern in the electronic bulletin board coordinate system. In S3051, a plurality of patterns with high similarity may be detected. In that case, it is good also as a structure which displays a thing with the highest similarity, and it is good also as a structure which presents each thread | sled information to a user and selects which to superimpose and display.

  If a feature pattern is detected (YES in step S3051), the superimposition processing unit 315 transmits a thread information acquisition request including feature pattern information to the server 400 in step S3052. In addition, the superimposition processing unit 315 stores the position of the detected feature pattern on the screen coordinates in the storage unit 303.

In step S <b> 3053, the superimposition processing unit 315 receives information (thread information list) from the server 400 as a response to the transmission of the feature pattern information ID in step S <b> 3052. In step S3054, the superimposition processing unit 315 calculates a thread position where the thread is superimposed and displayed on the captured image. In other words, the superimposition processing unit 315 multiplies the relative position (X _m , Y _m ) of the feature pattern information (2601) from the position on the screen of the feature pattern stored in S3052 by the enlargement rate obtained in S3051. Subtract the relative position that matches the coordinate unit on the screen.

  In step S3055, the superimposition processing unit 315 reads from the storage unit 303 fixed mode information indicating whether or not the superimposition display type for the thread is the fixed mode. If the fixed mode information is not the fixed mode (NO in S3055), superimposition processing unit 315 determines to update the display type of the thread to be superimposed and proceeds to S3056. If the fixed mode information is the fixed mode (YES in S3055), the superimposition processing unit 315 determines not to update the display type of the thread to be superimposed and proceeds to S3059. Details of the determination of the fixed mode information will be described with reference to FIG.

  In step S <b> 3056, the superimposition processing unit 315 determines a display type for displaying an image object to be superimposed and displayed as a thread display or an icon display. This process will be described with reference to FIG. The superimposition processing unit 315 stores the thread information list in the storage unit 303 in association with the display type determined for each thread. Thereafter, the process proceeds to S3057.

  In step S3059, the superimposition processing unit 315 reads the previous list of thread information associated with the determined display type from the storage unit 303. In step S3060, the superimposition processing unit 315 determines whether there is a difference between the thread information list received in step S3053 and the previous thread information list read out in step S3059. If there is a difference (YES in S3060), the superimposition processing unit 315 determines the display type of the difference thread in S3061. In this embodiment, the display type is determined to display an icon for a thread that exists in the thread information list received in S3053 but does not exist in the previous thread information list read in S3059. Then, the superimposition processing unit 315 stores the thread information list in the storage unit 303 in association with the display type determined for each thread. Thereafter, the process proceeds to S3057. If there is no difference (NO in S3060), the process proceeds to S3057.

  In step S3057, the superimposition processing unit 315 generates an image object for superimposed display. This process will be described later with reference to FIG. In step S <b> 3058, the superimposition processing unit 315 superimposes and displays the generated image object on a position based on the detection position of the feature pattern on the image captured by the imaging unit 306. The superimposition processing unit 315 enlarges / reduces the image object generated in step S <b> 3057 to match the size on the screen based on the enlargement ratio, and then draws the image object at the thread position.

(Display type determination process)
FIG. 22 is a flowchart of processing for determining a display type for displaying an image object to be superimposed or displayed as an icon by the superimposition processing unit 315 in S3056 of FIG. In the present embodiment, the flowchart shown in FIG.

  In step S3071, the superimposition processing unit 315 acquires the number of threads included in the information received from the server 400 in step S3053. In S3072, the superimposition processing unit 315 reads the position on the screen coordinates of the feature pattern detected from the imaging screen stored in the storage unit 303 in S3052, and calculates the distance to the center position of the screen.

  In S3073, the superimposition processing unit 315 selects a feature pattern corresponding to the smallest distance among the distances calculated in S3072. In step S3074, the superimposition processing unit 315 determines to display the thread corresponding to the feature pattern selected in step S3073 as a thread, and determines to display the thread corresponding to the feature pattern not selected in step S3073 as an icon. Then, the superimposition processing unit 315 stores the display type of the thread corresponding to the feature pattern selected in S3073 in the storage unit 303.

(Image object generation processing)
FIG. 23 is a flowchart for generating an image object in S3057 of FIG. This flow is performed by the superimposition processing unit 315 of the information terminal 300.

  In step S <b> 3081, the superimposition processing unit 315 reads and determines whether the display type of the target thread information is thread display or icon display from the storage unit 303. When the display type is thread display (YES in S3081), in S3082, the superimposition processing unit 315 determines the position of the feature pattern from the sign of the x-direction relative position and the y-direction relative position in the feature pattern information table 2601. Investigate in which direction the upper right, lower right, upper left, lower left, and so on from the thread position. For example, when the relative position in the x direction is positive and the relative position in the y direction is positive, it is determined that the position of the feature pattern is at the lower right relative to the position of the thread. When the sign is reversed, the result is the result of inverting left, right, up and down.

  In step S <b> 3083, the superimposition processing unit 315 recalculates the position of the comment so that the arrangement of each comment included in the thread information has a positional relationship that extends in the direction of the feature pattern. As shown in FIG. 3, the coordinates of the comment are defined so that the right is positive in the x direction and the lower is positive in the y direction. The electronic comment is moved by (Δx1, Δy1) each time it is added. Here, a case where Δx1 and Δy1 are positive will be described. When Δx1 and Δy1 are positive, the arrangement of each comment included in the thread looks like it extends toward the lower right. If the feature pattern is in a direction other than the lower right, the x-coordinate and / or y-coordinate of each comment information 802 is recalculated according to the direction. At this time, the coordinates are recalculated so that the thread information is arranged in the direction of the feature pattern in order from the first comment.

  In step S <b> 3084, the superimposition processing unit 315 generates, for each piece of comment information 802 having the recalculated coordinates, comment information image objects to be superimposed and displayed based on the “superimposition target” information included in the comment information 802. To do. When not subject to superimposition display, the head comment of the processing target thread is rendered as an object with 100% transparency. In addition, when generating the image object of the comment information, the superimposition processing unit 315 generates an image object of the reference file access button for the reference file information having the comment ID of the comment information to be superimposed and displayed as the parent comment ID. .

  In addition, the superimposition processing unit 315 does not generate an image object of the reference file access button for the reference file information having the comment ID of the comment information that is not the target of superimposition display as the parent comment ID, and does not generate the superimposition target. The image object generated as described above is generated based on each coordinate information in the thread information so as to be displayed at a position where the positional relationship between the comment information and the file information is maintained. That is, as shown in FIG. 26B, a handwritten comment 721, electronic comments 741 and 743, and a reference file access button 742 are displayed in a thread form.

  If the display type is icon display (NO in S3081), superimposition processing unit 315 acquires the number of comments included in the thread from the thread information in S3085. In step S <b> 3086, the superimposition processing unit 315 generates an icon including the acquired number of threads.

  For example, in FIGS. 27A and 27B, the captured image 3001 includes the object 110 and the object 120 on the paper surface. The object 110 is associated with a first thread including a handwritten comment 721, electronic comments 741, 743, and a reference file access button 742 as shown in FIG. A second thread composed of a handwritten comment 771 and an electronic comment 772 as shown in FIG.

  In the captured image 3001 in FIG. 27A, the magnitude relationship between the distance R1 from the object 110 to the captured image center 900 and the distance R2 from the object 120 to the captured image center 900 is R1 <R2. Therefore, in FIG. 27A, the first thread related to the object 110 is displayed as a thread, and the second thread related to the object 120 is displayed as an icon 902 indicating that the number of comments is two. .

  In the captured image 3001 in FIG. 27B, the magnitude relationship between the distance R1 from the object 110 to the captured image center 900 and the distance R2 from the object 120 to the captured image center 900 is R1> R2. Therefore, in FIG. 27A, the first thread related to the object 110 is an icon display 901 indicating that the number of comments is three, and the second thread related to the object 120 is a thread display. .

  In addition, in the captured image 3001 shown in FIG. 27, although the example in which two objects are included was shown, it is not limited to this. One or more objects may be included. When three or more objects are included, thread information associated with two or more objects may be displayed as a thread.

(Fixed mode information decision processing)
FIG. 24 is a flowchart for determining the fixed mode information used for determining the display type of whether the image object to be superimposed and displayed by the superimposition processing unit 315 is displayed as a thread or an icon in S3055 of FIG. . In the first embodiment, FIG. 24A is applied.

  In step S3091, the control unit 302 determines whether the display unit 304 has detected a tap operation on the display unit by the user. When the tap operation is detected (YES in S3091), in S3092, the control unit 302 reads out the fixed mode information stored in the storage unit 303, and then inverts and stores the value. The fixed mode information means the fixed mode or the non-fixed mode, and the initial value is the fixed mode.

(Superimposed display processing on the server)
FIG. 25 is a flowchart in the server 400 regarding superimposed display. This flowchart is realized by the CPU of the server 400 reading and executing a program stored in a ROM or the like. This is processing performed between S3052 and S3053 in FIG.

  In S4051, the control unit 402 of the server 400 determines whether or not the thread information acquisition request in S3052 has been received from the information terminal 300. The thread information acquisition request includes information on the feature pattern detected by the superimposition processing unit 315.

  When a thread information acquisition request is received (YES in S4051), in S4052, the feature pattern management unit 416 of the control unit 402 searches for thread information corresponding to the received feature pattern information using the feature pattern information table. Next, in S4053, the control unit 402 transmits to the information terminal 300 a list of thread information obtained by searching in S4052 as a response to the thread information acquisition request. Then, this processing flow ends.

  21 to 25, the handwritten comment, the electronic comment, and the reference file access button can be threadedly displayed on the captured image 3001 captured by the imaging unit 306 as illustrated in FIG. Each information can also be displayed as an icon.

  As a result, the user can easily acquire an electronic comment added to the handwritten comment simply by imaging the paper material 10, and since the thread is displayed, the user can easily understand the relationship between the comments. . Also, as shown in FIG. 28, when each feature pattern is at the lower right (110), upper right (111), upper left (112), lower left (113) with respect to the center point of the original data, the thread display is shown in FIG. a) to FIG. 28 (d).

  In FIG. 28, handwritten characters 210 are character strings actually recorded on paper with pen ink. As shown in FIG. 28, by changing the extension direction of the thread arrangement according to the position of the feature pattern, the user can easily point the camera to the area where the feature pattern exists. As a result, it is possible to avoid the invalidation of the superimposed display due to the feature pattern deviating from the captured image without the user's special awareness.

  For example, as shown in FIG. 28C, when the feature pattern is in the upper left (112), if the thread is simply extended to the lower right, the user tries to read the tip of the thread and turns the camera to the lower right. Things are expected. As a result, the feature pattern 710 is not included in the captured image, and superimposed display becomes impossible. If the thread information is expanded to the upper left as shown in FIG. 28C of the present embodiment, even if the user tries to read the tip of the thread and points the camera to the upper left, there is a feature pattern 710 in that direction. A situation in which the feature pattern 710 is not included therein is less likely to occur.

  The processing flow in FIGS. 21 and 25 is a processing flow in the case where the information terminal 300 displays thread information superimposed on the image captured by the image capturing unit 306, and an image as illustrated in FIG. 26B is displayed. Displayed on the part 304. Here, the information terminal 300 displays the display mode in which the thread information is superimposed on the original data 700 as shown in FIG. 26A and the captured image 3001 as shown in FIG. You may have the mode selection means (not shown) which selects the display mode on which thread information is superimposed and displayed.

  Note that this embodiment can also be applied to cases where a large number of original data and printed materials are used. This can be done by preparing a plurality of electronic pens and having a configuration in which a printed matter of each original data and the electronic pen can be associated one to one.

  Moreover, although the method of inputting handwritten comment information is based on coordinate calculation using the electronic pen 100 that emits ultrasonic waves, it may be based on another method. For example, writing is performed with an electronic pen on a special sheet on which a predetermined dot pattern is printed, and the dot pattern corresponding to the locus is read by an imaging unit attached to the electronic pen. The server may store a special paper dot pattern information and generate a handwritten comment bitmap in the electronic bulletin board coordinate system based on the trajectory information transmitted from the electronic pen.

  In addition, by setting the display format to thread display or icon display, the user can grasp the information of the paper material and the superimposed information while ensuring the visibility. In addition, the switching means for determining whether the thread display mode or the icon display mode is fixed can automatically switch the display according to a change in the imaging screen, and can prevent display switching unintended by the user. Can also be obtained.

<Second Embodiment>
The second embodiment is different in that FIG. 22B is applied in S3056 of FIG. 21 shown in the first embodiment.

  In step S3171, the superimposition processing unit 315 acquires the number of threads included in the thread information based on the thread information received from the server 400 in step S3053. Next, in S3172, the superimposition processing unit 315 calculates the distance from the position of the thread to be superimposed and displayed on the captured image calculated in S3054 to the center of the captured image.

  Next, in S3173, the superimposition processing unit 315 selects a feature pattern corresponding to the smallest distance among the distances calculated in S3172. In step S3174, the superimposition processing unit 315 determines to display the thread corresponding to the feature pattern selected in step S3173, and determines to display the icon corresponding to the thread corresponding to the feature pattern not selected in step S3173. Then, the superimposition processing unit 315 stores the display type of the thread corresponding to the feature pattern selected in S3173 in the storage unit 303. Thereafter, this processing flow ends.

  According to this embodiment, the same effect as that of the first embodiment can be obtained.

<Third Embodiment>
The third embodiment is different in that FIG. 22C is applied in S3056 of FIG. 21 shown in the first embodiment.

  In step S3271, the superimposition processing unit 315 acquires the number of threads included in the thread information based on the thread information received from the server 400 in step S3053. Next, in S3272, the superimposition processing unit 315 reads the position on the screen coordinates of the feature pattern detected from the imaging screen stored in the storage unit 303 in S3052, and calculates the distance to the center position of the screen.

  Next, in S3273, the superimposition processing unit 315 selects a feature pattern corresponding to a distance equal to or less than a predetermined threshold (or less than the threshold) among the distances calculated in S3272. In step S3274, the superimposition processing unit 315 determines to display the thread corresponding to the feature pattern selected in step S2733, and determines to display the thread corresponding to the feature pattern not selected in step S2733 as an icon. Then, the superimposition processing unit 315 stores the display type of the thread corresponding to the feature pattern selected in S3273 in the storage unit 303. Thereafter, this processing flow ends.

  According to this embodiment, the same effect as that of the first embodiment can be obtained.

<Fourth Embodiment>
The fourth embodiment is different in that FIG. 22D is applied in S3056 of FIG. 21 shown in the first embodiment.

  In step S3371, the superimposition processing unit 315 acquires the number of threads included in the thread information based on the thread information received from the server 400 in step S3053. Next, in S3372, the superimposition processing unit 315 calculates the distance from the position of the thread to be superimposed and displayed on the captured image calculated in S3054 to the center of the captured image.

  Next, in S3373, the superimposition processing unit 315 selects a feature pattern corresponding to a distance equal to or less than a predetermined threshold (or less than the threshold) among the distances calculated in S3372. In step S3374, the superimposition processing unit 315 determines to display a thread corresponding to the feature pattern selected in step S3373, and determines to display the icon corresponding to the thread corresponding to the feature pattern not selected in step S3373. The superimposition processing unit 315 stores the display type of the thread corresponding to the feature pattern selected in S3373 in the storage unit 303. Thereafter, this processing flow ends.

  According to this embodiment, the same effect as that of the first embodiment can be obtained.

<Fifth Embodiment>
The fifth embodiment is different in that FIG. 24B is applied in S3055 of FIG. 21 shown in the first embodiment.

  In step S3093, the control unit 302 determines whether the change direction of the imaging screen (that is, the moving direction of the imaging unit 306) is the same as the direction in which the comment included in the thread information recalculated in step S3082 is arranged. That is, it is determined whether or not the image has been updated so that the object moves in a direction away from the center of the captured image. When it is determined that the directions are the same (YES in S3093), in S3094, the control unit 302 reads out the fixed mode information stored in the storage unit 303, and then inverts and stores the value. The fixed mode information means the fixed mode or the non-fixed mode, and the initial value is the fixed mode. If it is determined that the directions are different (NO in S3093), control unit 302 does not update the fixed mode information. Thereafter, this processing flow ends.

  According to this embodiment, the same effect as that of the first embodiment can be obtained.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (13)

Detection means for detecting one or more objects from the captured image;
Management means for associating and managing electronic information input based on a user instruction to one or more objects detected by the detection means;
Selection means for selecting at least one object from the objects managed by the management means;
The electronic information associated with the object selected by the selection means is displayed as being superimposed on the captured image without being iconified, and the electronic information related to the object not selected by the selection means is iconized to the captured image. Superimposing display means for superimposing and displaying on,
Have
The information processing system, wherein the superimposed display unit arranges electronic information associated with the selected object based on a position of the object selected by the selection unit in the captured image.   The information processing system according to claim 1, wherein the superimposed display unit updates the arrangement of the electronic information when the captured image is updated. An object has multiple electronic information associated with it,
The information processing system according to claim 1, wherein the superimposed display unit arranges the plurality of pieces of electronic information sequentially toward the object.   The said selection means selects an object based on the distance of each position of the object managed by the said management means, and the predetermined position of the said captured image, It is any one of Claim 1 thru | or 3 characterized by the above-mentioned. Information processing system described in 1.   The selection unit is configured to select an object based on a distance between a position when the electronic information associated with each object managed by the management unit is displayed superimposed on the captured image and a predetermined position of the captured image. The information processing system according to claim 1, wherein the information processing system is selected.   6. The information processing system according to claim 4, wherein the selection unit selects an object having the smallest distance.   The information processing system according to claim 4, wherein the selection unit selects an object whose distance is equal to or less than a predetermined threshold.   The information processing system according to claim 4, wherein the predetermined position is a center position of the captured image.   9. The superimposed display unit updates the content of the superimposed display when the captured image is updated or when electronic information associated with the object is updated. Information processing system described in 1.   The information processing system according to claim 1, wherein the selection unit changes an object to be selected based on a user instruction.   The selection unit is configured to change an object to be selected when an object selected by the selection unit in the captured image is away from a center of the captured image when the captured image is updated. The information processing system according to claim 1. A detection step of detecting one or more objects from the captured image;
A management process in which the management unit associates electronic information input based on a user instruction with respect to one or a plurality of objects detected in the detection process;
A selection step of selecting at least one object from the objects managed by the management unit;
The electronic information associated with the object selected in the selection step is displayed as being superimposed on the captured image without being converted into an icon, and the electronic information associated with the object not selected in the selection step is converted into an icon. A superimposing display process for superimposing and displaying on
Have
In the superimposition display step, electronic information associated with the selected object is arranged based on the position of the object selected in the selection step in the captured image. Computer
Detection means for detecting one or more objects from the captured image;
Management means for associating and managing electronic information input based on a user instruction to one or more objects detected by the detection means;
Selection means for selecting at least one object from the objects managed by the management means;
The electronic information associated with the object selected by the selection means is displayed as being superimposed on the captured image without being iconified, and the electronic information related to the object not selected by the selection means is iconized to the captured image. Superimposing display means for superimposing and displaying on,
Function as
The superimposition display unit arranges electronic information associated with the selected object based on the position of the object selected by the selection unit in the captured image.

Images