JP6384091B2 - Server apparatus, program, and communication method - Google Patents

Description

  The present invention relates to a server device that provides information about a medium based on image data of a medium of a printed matter such as a book, which is photographed by a communication terminal device, a program thereof, and a communication method.

  In recent years, with the development of networks such as the Internet and communication terminal devices such as personal computers, digitized printed materials such as books have become common, and it has become common to browse printed materials electronically via a network or recording medium. Yes. Recently, content is often browsed through portable portable communication terminal devices such as portable game devices having a communication function, tablet portable terminal devices, and portable telephones. In addition, information related to an electronic book is also provided to a user via a communication terminal device.

  On the other hand, a system that provides related book information related to a printed matter such as a book to a user via a communication terminal device is known (for example, Patent Document 1). In Patent Document 1, a barcode that is recorded on a barcode attached to a book by printing or pasting is associated with a book code that identifies the book and related book information in advance, so that the barcode is read when the barcode is read. A technique for providing related book information related to the book code recorded in the user to the user has been proposed.

JP 2014-10730 A

  However, in the technique proposed in Patent Document 1, it is necessary to attach a barcode or the like to a printed matter such as a book, and there is a problem that the design of the medium of the printed matter is impaired. In addition, a technique for embedding information in a medium of a printed material with a digital watermark or an encoded image is generally known, but since it is necessary to embed information when printing a book or the like, a medium of a printed material that has already been printed There is a problem that information cannot be attached to.

  SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and an object of the present invention is to read a printed material by reading an image of a printed material without changing the printed material medium. Another object of the present invention is to provide a server device or the like that can provide a user with information related to the medium.

  (1) In order to solve the above-described problem, the server device of the present invention is configured to detect an image of the medium detected based on image data generated by a user photographing the medium with a camera included in the communication terminal device. Receiving means for receiving data including local invariant feature values of a plurality of feature points calculated from a plurality of feature points as photographed image feature value data from the communication terminal device, and image features of each medium recorded in the database Determining means for determining whether image feature quantity data of a medium having a part or all of the photographed image feature quantity data received by the receiving means is present in the database based on quantity data; and the photographed image When it is determined that image feature amount data of a medium having a part or all of the feature amount data exists in the database, the captured image feature amount data And a part or the media-related information about the medium that has been recorded in the database in association with the image feature amount data of the medium having a total have a configuration and a transmission means for transmitting to the communication terminal device.

  With this configuration, the server device according to the present invention receives the captured image feature amount data for the image captured by the user with the camera provided in the communication terminal device, and thereby the image of the medium having part or all of the captured image feature amount data. The medium related information recorded in the database in association with the feature data can be acquired as information embedded in the medium, and the medium related information can be provided to the user.

  Therefore, the server device of the present invention is a camera provided in the communication terminal device when the user wants to check information related to the printed medium with respect to the printed material such as a book, newspaper, brochure, magazine, book, ticket, etc. By simply reading an image, information regarding the medium of the printed material can be provided to the user as information embedded in the medium via the communication terminal device. As a result, for users who are interested in information such as URLs, texts, images, videos, and sounds related to the printed media without changing the printed media and also for the printed materials. Information can be provided and the user can view the information.

  (2) In the server device according to the present invention, when the reception unit receives, from the communication terminal device, medium-related information related to a medium associated with the captured image feature amount data together with the captured image feature amount data. The apparatus further includes recording means for recording the captured image feature amount data as image feature amount data of the medium in association with the medium related information.

  With this configuration, the server device according to the present invention is a medium related to a medium as information (information to be associated) that the user wants to embed in the medium, together with the captured image feature amount data for the image of the medium captured by the user with the camera included in the communication terminal device. By receiving the related information, the photographed image feature data can be recorded in the database as image feature data of the medium in association with the medium related information. Therefore, when another user takes an image of the same medium with a camera included in the communication terminal device, the medium related information recorded in the database in association with the image feature amount data of the medium is provided to the other user. And other users can view the information.

  (3) Moreover, the server apparatus of this invention is based on the receiving means which receives from the said communication terminal device the image data produced | generated when the user image | photographed the medium with the camera with which a communication terminal device is equipped, and the said image data , Based on the image feature value data of each medium recorded in the database, calculating means for detecting a plurality of feature points in the image of the medium and calculating local invariant feature values of the detected feature points, Determining means for determining whether image feature data of a medium having a part or all of the calculated local invariant feature quantities of the plurality of feature points exists in the database; and local invariant features of the plurality of feature points When it is determined that image feature data of a medium having a part or all of the quantity exists in the database, some or all of the local invariant feature quantities of the plurality of feature points are present. That has a medium-related information about the medium that has been recorded in the database in association with the image feature amount data of the medium have the configuration and transmission means for transmitting to the communication terminal device.

  With this configuration, the server device of the present invention receives the image data of the medium generated by the user shooting with the camera included in the communication terminal device, and thereby the local invariance of the plurality of feature points based on the image data. Calculates feature values and acquires media-related information recorded in the database in association with image feature data of media that have some or all of the local invariant feature values of multiple feature points as information embedded in the media In addition, the medium-related information can be provided to the user.

  Therefore, the server device of the present invention is a camera provided in the communication terminal device when the user wants to check information related to the printed medium with respect to the printed material such as a book, newspaper, brochure, magazine, book, ticket, etc. By simply reading an image, information regarding the medium of the printed material can be provided to the user as information embedded in the medium via the communication terminal device. As a result, for users who are interested in information such as URLs, texts, images, videos, and sounds related to the printed media without changing the printed media and also for the printed materials. Information can be provided and the user can view the information.

  (4) In the server device of the present invention, when the receiving unit receives from the communication terminal device medium-related information related to a medium associated with the image data, the calculating unit includes the image data. Based on the data, a plurality of feature points in the image of the medium are detected, and local invariant feature amounts of the detected plurality of feature points are calculated, and the server device calculates the local invariant features of the calculated feature points. The image processing apparatus further includes recording means for recording data including the amount as image feature amount data of the medium in association with the medium related information.

  With this configuration, the server apparatus of the present invention is a medium related to a medium as information (information to be associated) that the user wants to embed in the medium, together with image data of the medium generated by the user shooting with a camera included in the communication terminal apparatus. By receiving the related information, the local invariant feature amount of a plurality of feature points with respect to the image of the medium is calculated based on the image data of the medium, and the data including the local invariant feature amount of the plurality of feature points is calculated as the image feature amount of the medium Data can be recorded in a database in association with medium-related information. Therefore, when another user takes an image of the same medium with a camera included in the communication terminal device, the medium related information recorded in the database in association with the image feature amount data of the medium is provided to the other user. And other users can view the information.

  (5) In the server device of the present invention, a local invariant feature amount of the plurality of feature points is detected based on image data generated by photographing a part of the medium. It has a configuration including local invariant feature values calculated from a plurality of feature points in the image.

  With this configuration, the server device of the present invention enables the captured image feature amount data calculated by the communication terminal device based on the image captured by the user with the camera included in the communication terminal device to be a part of the medium of the printed matter. All of the information related to the one or more media associated with the image feature data of one or more media having a part or all of the media can be provided to the user via the communication terminal device. Therefore, the user is not required to perform complicated operations such as adjusting the position of the camera so that the entire image of the medium can be captured by the camera, or shooting a moving image of the entire medium while tracing the entire medium with the camera. Can be improved. In addition, it is possible to promptly provide the user with information related to the medium of the printed material taken.

  (6) Moreover, in order to solve the above-described problem, the program of the present invention is an image generated by a user shooting a medium with a camera provided in the communication terminal device, which provides a server device that provides information to the communication terminal device. Receiving means for receiving, from the communication terminal device, data including local invariant feature amounts of a plurality of feature points calculated from a plurality of feature points in the detected image of the medium based on the data as photographed image feature amount data Whether image feature data of a medium having a part or all of the photographed image feature data received by the receiving unit is present in the database based on image feature data of each medium recorded in the database Determining means for determining whether or not image feature data of a medium having a part or all of the photographed image feature data is present in the database; Transmitting means for transmitting to the communication terminal device medium related information relating to the medium recorded in the database in association with the image feature quantity data of the medium having part or all of the captured image feature quantity data, It has the structure which functions as.

  With this configuration, the program of the present invention is a medium that receives, by the server device, captured image feature amount data for an image captured by a user with a camera included in the communication terminal device, and includes a part or all of the captured image feature amount data. The medium related information recorded in the database in association with the image feature amount data can be acquired as information embedded in the medium, and the medium related information can be provided to the user.

  Therefore, the program according to the present invention is printed on the camera provided in the communication terminal device when the user wants to check information related to the medium of the printed material with respect to the printed material such as a book, a newspaper, a brochure, a magazine, a book, or a ticket. By simply reading an image on the medium, information regarding the medium of the printed matter can be provided to the user as information embedded in the medium via the communication terminal device. As a result, for users who are interested in information such as URLs, texts, images, videos, and sounds related to the printed media without changing the printed media and also for the printed materials. Information can be provided and the user can view the information.

  (7) Further, in order to solve the above-described problem, the communication method of the present invention includes a method for detecting the medium detected based on image data generated by a user shooting a medium with a camera included in the communication terminal device. A reception step of receiving data including local invariant feature amounts of a plurality of feature points calculated from a plurality of feature points in an image as photographed image feature amount data from the communication terminal device, and each medium recorded in the database A determination step of determining whether image feature data of a medium having a part or all of the captured image feature data received by the receiving unit is present in the database based on the image feature data; When it is determined that image feature amount data of a medium having a part or all of the photographed image feature amount data exists in the database, the photographed image feature amount data It has a configuration comprising a transmission step of transmitting data of a part or the media-related information about the medium that has been recorded in the database in association with the image feature amount data of the medium having a total to the communication terminal apparatus.

  With this configuration, in the communication method of the present invention, the server device 30 receives the captured image feature amount data for the image captured by the user with the camera included in the communication terminal device 10, so that a part of the captured image feature amount data or The medium related information recorded in the database 330 in association with the image feature amount data of all the media can be acquired as information embedded in the medium, and the medium related information can be provided to the user.

  Therefore, the communication method of the present invention enables the user of the printed terminal such as a book, a newspaper, a pamphlet, a magazine, a book, a ticket, etc. to check the information about the printed medium with a camera included in the communication terminal device. By simply reading an image, information regarding the medium of the printed material can be provided to the user as information embedded in the medium via the communication terminal device. As a result, for users who are interested in information such as URLs, texts, images, videos, and sounds related to the printed media without changing the printed media and also for the printed materials. Information can be provided and the user can view the information.

  The server device, the program, and the communication method according to the present invention read information related to a print medium by reading an image of the print medium without changing the print medium. Can be provided.

1 is a system configuration diagram showing a configuration of a communication system in an embodiment of the present invention. It is a block diagram which shows the structure of the communication terminal device in one Embodiment. It is a block diagram which shows the structure of the server apparatus in one Embodiment. It is a figure which shows an example of the information which linked | related the image feature-value data of each paper surface recorded on feature-value data DB in the server apparatus in one Embodiment, and the paper related information regarding this paper surface. It is a figure which shows an example of the image feature-value data of each paper surface recorded on feature-value data DB in the server apparatus in one Embodiment. It is a flowchart (the 1) which shows the paper related information display process performed in the communication system of one Embodiment. It is a flowchart (the 2) which shows the paper related information display process performed in the communication system of one Embodiment. It is a flowchart which shows the process performed in the communication system of one Embodiment in association with the image | photographed image feature-value data of a paper surface, and the information regarding the paper surface in association. It is a figure which shows an example of the printed matter related information recorded on the database in the server apparatus in one modification. It is a figure which shows an example of the hierarchical information of the image feature-value data recorded on the database in the server apparatus in one modification.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following embodiments, the server device, the program, and the communication method according to the present invention are applied to a communication system having a communication terminal device and a server device connected to the communication terminal device via a network. This is an embodiment. Further, in the following embodiments, a case where the medium is a paper medium will be described. However, the medium is written with characters or figures such as plastic, cloth, nonwoven fabric, metal, wood, glass, ceramics, stone, etc. The medium is not particularly limited.

[1] Configuration and Outline of Communication System First, the configuration and outline of the communication system 1 in the present embodiment will be described with reference to FIG. FIG. 1 is a system configuration diagram showing the configuration of the communication system 1 in the present embodiment. Further, in order to prevent the figure from becoming complicated, FIG. 1 shows only some users and communication terminal devices owned by the users. That is, in the actual communication system 1, there are more users and communication terminal devices 10 than displayed.

  In the communication system 1 of the present embodiment, a user who wants to view paper-related information related to the paper surface of a printed material such as a book, a book, a pamphlet, a ticket, etc., photographs the paper surface with a camera provided in the communication terminal device 10. This is a system that can provide paper-related information to the user.

  In particular, the communication system 1 is a system that can provide a user with page-related information by reading an image of a printed sheet of a printed material without embedding information in the printed sheet. is there.

  In addition, the communication system 1 of the present embodiment is a system that provides paper-related information to a user who is a member so as to be browsed by an SNS or other network service. In the following description, a system in which the communication system 1 provides a user who is a member to be able to view paper-related information will be described. However, the communication system 1 is a non-member user who views paper-related information. You may make it make it.

  Specifically, as shown in FIG. 1, the communication system 1 of the present embodiment is connected to a plurality of communication terminal devices 10, a network 20, a server device 30, and a network 20 that are possessed by each user. Base station BS.

  The network 20 is configured, for example, by connecting a public telephone network including a mobile telephone network and an IP (Internet Protocol) network. However, the configuration of the network 20 is not limited to this.

  The communication terminal device 10 is a device having a communication function and a photographing function, for example, a personal computer (PC), a smartphone, a mobile phone, a tablet information terminal device, a portable game machine, or the like.

  Further, the communication terminal apparatus 10 receives resource data for the WWW system described in a markup language such as XML (extensible Markup Language) and a unique URL (Uniform Resource Locator) indicating a network address of the resource data. It has a browsing function for performing data communication with the server device 30 while being used.

  The communication terminal device 10 uses unique identification information (hereinafter referred to as “specific identification information”) for each communication terminal device 10 in order to use various functions of the SNS or to communicate with the server device 30. , Simply referred to as “terminal ID”). In addition, the communication terminal device 10 displays various information based on the XML data received from the server device 30 using the browsing function, and is connected to the server device 30 via the base station BS and the network 20, The server apparatus 30 is provided with various services such as SNS. The communication connection with the server device 30 has a predetermined condition, for example, a login condition, and is permitted to the logged-in communication terminal device 10.

  On the other hand, the communication terminal device 10 includes an imaging unit (camera) having an optical system including a lens and an imaging element, and is configured to be able to perform imaging. Specifically, the communication terminal apparatus 10 captures an image by allowing the user to shoot the printed material using an application (hereinafter referred to as “photographing application”) for capturing a paper surface of the printed material with a still image or a moving image. On the basis of the output of this unit, still image data or each frame constituting a moving image is generated as individual image data and recorded in a storage area such as a memory.

  The communication terminal device 10 detects a plurality of feature points that can determine the position of an object in the image in still image data or image data of each frame constituting a moving image, and detects the plurality of feature points. On the other hand, an invariant image feature amount (local invariant feature amount) of a local region centered on a feature point is calculated.

  Further, when the communication terminal device 10 calculates the local invariant feature amounts of the plurality of feature points in the image of the printed paper, the server device 30 uses the data including the local invariant feature amounts of the plurality of feature points as the captured image feature amount data. Execute the process to send to. Then, the communication terminal device 10 provides the user with information related to the paper surface of the printed matter received from the server device 30 (hereinafter also referred to as “paper surface related information”) in response to the transmission of the captured image feature amount data to the server device 30. A process of displaying in a viewable manner (hereinafter referred to as “paper related information display process”) is executed. The page-related information is, for example, information such as URL, text, image, moving image, and sound.

  In addition, the communication terminal device 10 executes processing for transmitting, to the server device 30, paper-related information that the user wants to associate with the image of the paper, together with the captured image feature amount data for the image of the paper photographed by the user. The page-related information is, for example, information such as URL, text, image, moving image, and sound.

  The page-related information varies depending on the printed matter and the contents of the printed matter.For example, the printed matter is a book or a book, and the page is the title of the front cover, back cover, back cover, book or book. In the case of an explicit door page, imprint, etc., information on the contents of the book or book, information on the latest version of the book or book, different book information on the book or book author, etc. In the case of a door page that clearly indicates a part / chapter / story, information on the contents of each part / each chapter / each story, information on the contents of each page included in each part / each chapter / each story, and the like can be given. When the printed material is a pamphlet, information on the contents described in the pamphlet, other pamphlet information issued by the pamphlet issuer, and the like can be given. If the printed material is a ticket, information about the event, map information to the event venue, accommodation information and parking lot information around the event venue, detailed information on products that can be purchased at the event venue, other events of the performers who appear in the event Information etc. can be mentioned.

  While providing various network services such as SNS, the server device 30 executes various processes in cooperation with the communication terminal device 10 based on a user instruction via the communication terminal device 10.

  In particular, when the server device 30 receives the captured image feature amount data from the communication terminal device 10, the server surface 30 has the captured image feature amount data based on the image feature amount data of each sheet of each printed matter recorded in the database 330. The process for determining whether or not the image feature amount data of the image exists in the database 330 is executed.

  In addition, when the server device 30 determines that the image feature data of the page having the captured image feature data exists in the database 330, the server device 30 associates the image feature data of the paper with the captured image feature data with the database 330. A process for transmitting the paper-related information recorded in the communication terminal device 10 is executed.

  On the other hand, when the server device 30 receives the photographed image feature value data and the paper-related information associated therewith from the communication terminal device 10, the server device 30 associates the photographed image feature-value data with the paper-related information as image feature data on the paper. The process of recording in 330 is executed.

  In this embodiment, a database 330 is provided in the server device 30 for recording image feature amount data of each page of each printed matter and page related information associated therewith. However, the server 330 is independent of the server device 30 on the network. You may comprise.

  Further, when the server device 30 records the image feature quantity data of the page and the page related information in the database 330 in association with each other, the server device 30 transmits a notification to that effect to the communication terminal device 10.

  By having such a configuration, the communication system 1 according to the present embodiment is generated by a user who desires to view the paper-related information related to the paper surface of the printed material by photographing with the camera included in the communication terminal device 10. The server device 30 receives the data including the local invariant feature amounts of the plurality of feature points calculated by the communication terminal device 10 based on the image data on the paper surface as the captured image feature amount data, thereby having the captured image feature amount data. Information relating to the paper surface recorded in the database 330 in association with the image feature data of the paper surface can be acquired.

  That is, in the communication system 1 according to the present embodiment, the server device 30 receives photographed image feature value data for an image photographed by the user with a camera included in the communication terminal device 10, thereby allowing the paper surface having the photographed image feature value data. The paper-related information recorded in the database 330 in association with the image feature data can be acquired as information embedded in the paper, and the paper-related information can be provided to the user.

  Therefore, the communication system 1 according to the present embodiment uses the camera included in the communication terminal device to print the printed material when the user wants to check information on the printed surface of the printed material such as a book, a newspaper, a brochure, a magazine, a book, or a ticket. By simply reading an image on a paper surface, information relating to the paper surface of the printed material can be provided to the user as information embedded in the paper surface via the communication terminal device. As a result, for users who are interested in information such as URLs, texts, images, videos, and voices related to the printed page without changing the printed page and also for the printed page. Information can be provided and the user can view the information.

  Further, in the communication system 1, the server apparatus 30 includes, as photographed image feature amount data for the image of the sheet image captured by the user with the camera included in the communication terminal apparatus 10, information that the user wants to embed on the sheet (information to be associated). By receiving the paper-related information related to the image, the captured image feature data can be recorded as image feature data on the paper in the database in association with the paper-related information. Accordingly, the communication system 1 uses the page-related information recorded in the database 330 in association with the image feature amount data of the page when another user takes an image of the same page with the camera included in the communication terminal device 10. Since the information can be provided to other users, the information can be browsed by other users.

[2] Communication Terminal Device The configuration of each communication terminal device 10 of the present embodiment will be described using FIG. FIG. 2 is a configuration diagram illustrating the configuration of the communication terminal device 10 according to the present embodiment.

  As illustrated in FIG. 2, the communication terminal device 10 according to the present embodiment includes a recording unit 100, a network communication unit 110, an imaging unit 130, a terminal data processing unit 140, a display unit 150, and a display control unit 160. And an operation unit 170 and a terminal management control unit 180. The above-described units are connected to each other by a bus B, and data transfer is performed between the components.

  The recording unit 100 includes, for example, a hard disk drive (hereinafter abbreviated as “HDD”) or a nonvolatile flash memory such as a NAND type or a NOR type. The recording unit 100 also includes a random access memory (RAM) and a read only memory (ROM) as work areas of the network communication unit 110 and the terminal management control unit 180.

  The recording unit 100 includes at least an application recording unit 101, an image data recording unit 102, and a paper-related information recording unit 103.

  The application recording unit 101 stores various application programs (hereinafter referred to as “applications”) executed by the terminal data processing unit 140 and the terminal management control unit 180 such as a photographing application, and the above browsing. A browser program for realizing the function is also recorded. The application recording unit 101 includes a user name, a user ID for enjoying various network services (for example, SNS) provided by the server device 30, a terminal ID for uniquely identifying the communication terminal device 10, User registration information related to the user is recorded.

  The image data recording unit 102 records image data of a printed material photographed during the activation of the photographing application. The image data recording unit 102 may be an access area accessible by the user or an inaccessible area that the user cannot access.

  The paper-related information recording unit 103 stores the paper-related information related to the paper surface of the printed material to be associated with the photographed image feature amount data for a part or the whole of the paper image of the printed material photographed during the start of the photographing application, and from the server device 30. The page-related information recorded in the database 330 is recorded in association with the received image feature quantity data of the page having the photographed image feature quantity data.

  The network communication unit 110 establishes a wireless communication channel with the base station BS according to a communication standard compliant with the base station BS, and exchanges various data with the server device 30 via the network 20. In particular, the network communication unit 110 captures image feature amount data for a part or the whole of the image of the printed material photographed while the image capturing application is activated, and information related to the paper surface of the printed material associated with the photographed image feature data. And the like are transmitted to the server device 30.

  The network communication unit 110 uses, for example, still image data on a paper surface of a printed material, or image data of each frame constituting a moving image obtained by photographing the paper surface of the printed material as the captured image feature amount data. For a plurality of feature points that can be determined, data including an invariant image feature amount (local invariant feature amount) of a local region centered on the feature point is transmitted to the server device 30.

  Further, the network communication unit 110 transmits the captured image feature data to the server device 30 and then records it in the database 330 in association with the image feature data on the page having the captured image feature data from the server device 30. Receive paper-related information. The network communication unit 110 supplies the received paper-related information to the paper-related information recording unit 103 so that the display unit 150 displays the received paper-related information.

  Note that the communication standard to which the base station BS conforms varies depending on the configuration of the network 20. For example, when the network 20 includes a mobile phone network, it conforms to a communication standard such as 3GPP (3rd Generation Partnership Project), 3GPP2, or LTE (Long Term Evolution).

  The imaging unit 130 includes an optical system such as a lens for imaging a subject and an imaging element. The output signal of the image sensor is transmitted to the terminal data processing unit 140.

The terminal data processing unit 140 executes various data processing according to the application recorded in the application recording unit 101. In particular, the terminal data processing unit 140 captures a paper surface of a printed material while the shooting application is running.
(1) A generation process for generating image data of a still image on the paper surface of the printed material or image data of each frame constituting a moving image obtained by photographing the paper surface of the printed material,
(2) a recording process for recording the image data generated in (1) in the image data recording unit 102;
(3) Based on the image data, a plurality of feature points that can determine the position of the object in the image are detected, and a local region that is centered on the feature point is invariable with respect to the detected feature points. A calculation process for calculating an image feature amount (local invariant feature amount);
(4) A transmission process for transmitting data including local invariant feature amounts of a plurality of feature points to the image of the printed surface of the printed matter calculated in (3) to the server device 30 as photographed image feature amount data,
(5) Reception processing for receiving, from the server device 30, paper-related information recorded in the database 330 in association with the image feature data of the paper having photographed image feature data.
(6) instruction processing for supplying the received paper-related information to the display control unit 160 and instructing generation of display data indicating the paper-related information;
Execute.

  In addition, when the user inputs paper-related information about the paper before and after the user captures the paper of the printed material while the shooting application is running, the terminal data processing unit 140 performs the generation process (1) and (2 ) And the calculation process of (3) above, or before or after the process of recording the sheet related information is executed, and the sheet related information associated with the captured image feature amount data of (4) is as follows: A process of transmitting to the server device 30 together with the captured image feature data is executed. In addition, the terminal data processing unit 140 receives from the server device 30 a notification indicating that the captured image feature value data has been recorded as image feature value data on the page in association with the page related information, and displays the notification. An instruction process for instructing generation of display data indicating the notification is supplied to the control unit 160.

  The terminal data processing unit 140 includes at least an image data generation unit 141, a calculation unit 142, and a data processing unit 143.

  For example, the image data generation unit 141 converts the output signal of the imaging element received from the imaging unit 130 to AD to generate image data. In particular, the image data generation unit 141 generates image data of each frame constituting the captured still image or the captured moving image when the imaging unit 130 captures the paper surface of the printed material as a still image or a moving image. The image data generated by the image data generation unit 141 is supplied to the image data recording unit 102 and recorded.

  When the image data generation unit 141 generates image data, the calculation unit 142 detects a plurality of feature points that can determine the position of the target in the image from the image of the printed surface indicated by the image data, For a plurality of detected feature points, an invariant image feature amount (local invariant feature amount) of a local region centered on the feature point is calculated. Specifically, the calculation unit 142 detects, for example, a plurality of feature points in an image on a printed sheet by using an algorithm such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed Up Robust Features). The local invariant feature amount is calculated. In addition, the calculation unit 142 detects a plurality of feature points in the image of the printed sheet by using algorithms such as Harris corner detection, LoG (Laplacian of Gaussian), and DoG (Difference of Gaussian), and performs LBP (Local Binary Pattern), The local invariant feature may be calculated by an algorithm such as (Histograms of Oriented Gradients) or GLOH (Gradient Location and Orientation Histogram).

  The data processing unit 143 uses the data including the local invariant feature amounts of a plurality of feature points with respect to the image of the printed sheet as calculated by the calculation unit 142 as captured image feature amount data to the server device 30 via the network communication control unit 110. The transmission process is executed so that it is transmitted. When the data processing unit 143 receives, from the server device 30, paper-related information recorded in the database 330 in association with the image feature data of the paper having the captured image feature data, the data processing unit 143 stores the paper-related information in the paper related information recording unit 103. In addition to performing the recording process, the received paper-related information is supplied to the display control unit 160, and processing for instructing generation of display data indicating the paper-related information is executed. In addition, if the data processing unit 143 detects that the user has input the paper related information regarding the paper before and after photographing the paper of the printed material while the shooting application is running, the data processing unit 143 records the paper related information. A process of recording in the unit 103 is performed, and a process of transmitting to the server device 30 together with the captured image feature amount data as paper-related information associated with the captured image feature amount data is performed.

  The display unit 150 is configured by, for example, a panel of a liquid crystal element or an organic EL (Electro Luminescence) element, and displays a predetermined image based on display data generated by the display control unit 160.

  The display control unit 160 is configured to generate display data necessary for display on the display unit 150, and outputs the generated display data to the display unit 150. In particular, the display control unit 160 selects whether it is desired to acquire the paper-related information related to the paper surface of the printed material, or to embed (associate) the paper-related information related to the paper surface with respect to the paper surface of the printed material and share it with others. Screen data, display data indicating paper-related information received from the server device 30 via the network communication unit 110, and photographed image feature amount data for a part or the whole of a paper image of a printed matter photographed while the photographing application is activated by the user Screen data or the like for setting paper related information relating to the paper to be embedded (to be associated) is generated.

  The operation unit 170 includes various confirmation buttons, operation buttons for inputting each operation command, a number of keys such as a numeric keypad, and a touch panel, and is used when performing each operation.

  The terminal management control unit 180 is mainly composed of a central processing unit (CPU) and includes various input / output ports such as a key input port and a display control port, and executes various applications recorded in the recording unit 100. By doing so, the overall functions of the communication terminal apparatus 10 are controlled in a comprehensive manner.

  In particular, the terminal management control unit 180 monitors overall control and execution of each process based on various applications stored in the application recording unit 101. For example, when the terminal management control unit 180 recognizes a user input operation on the operation unit 170, the terminal management control unit 180 reads the user ID from the application recording unit 101 and logs in to the SNS.

[3] Server Device Next, the configuration of the server device 30 according to the present embodiment will be described with reference to FIGS. FIG. 3 is a block diagram illustrating the configuration of the server device 30 according to the present embodiment. FIG. 4 illustrates image feature amount data for each page recorded in the feature amount data DB 332 in the server device 30 according to the present embodiment. FIG. 5 is a diagram illustrating an example of information that associates the page-related information with respect to the page, and FIG. 5 illustrates an example of image feature amount data of each page recorded in the feature amount data DB 332 in the server device 30 of the present embodiment. FIG.

  As shown in FIG. 3, the server device 30 according to the present embodiment includes a communication control unit 310, a ROM / RAM 320, a recording device 330, a server data processing unit 340, and a server management control unit 350. The above-described units are connected to each other by a bus B, and data transfer is performed between the components.

  For example, the communication control unit 310 configures a receiving unit and a transmission (providing) unit of the server device 30 according to the present invention, and the recording device 330 configures a database of the server device 30 according to the present invention. It constitutes a recording means of the server device 30 according to the present invention. Further, for example, the server data processing unit 340 constitutes a determination unit of the server device 30 according to the present invention.

  The communication control unit 310 is a predetermined network interface, establishes a communication channel with the communication terminal device 10, and exchanges various data. In particular, the communication control unit 310 receives, from the communication terminal device 10, captured image feature value data, paper-related feature value data, and the like together with the captured image feature value data. Further, the communication control unit 310 transmits, to the communication terminal device 10, the page related information recorded in the database 330 in association with the image feature quantity data of the page having the photographed image feature quantity data.

  Various programs necessary for driving the server device 30 are recorded in the ROM / RAM 320. The ROM / RAM 320 stores various applications executed by the server data processing unit 340 and the server management control unit 350. The ROM / RAM 320 is used as a work area when various processes are executed.

  The server management control unit 350 is mainly configured by a central processing unit (CPU), and integrally controls each unit of the server device 30 by executing a program. For example, the server management control unit 350 performs login processing to various network services (for example, SNS) of each user based on a login request from the communication terminal device 10 by a user operation in conjunction with the recording device 330.

  The recording device 330 is configured by an HDD or an SSD (Solid State Drive), and has at least a member DB 331, a feature amount data DB 332, and a paper related information DB 333 in the recording area.

  The member DB 331 is a database that stores various types of information for managing information of each SNS user. For example, in the member DB 331, a user ID, a terminal ID, user registration information, and the like are registered (recorded) in association with each other.

  As shown in FIG. 4, the feature amount data DB 332 is an example of information that associates image feature amount data for some or all images of each page of the printed material with page related information about each page. Data and paper-related information are registered (recorded) in association with each other.

Further, the feature data DB is extracted as an example of image feature data for a part or all of the image on each paper surface of the printed material, as shown in FIG. 5, based on a part or all of the image on the paper surface of the printed material. In addition, for multiple feature points that can determine the position of the object in the image,
(1) x- and y-coordinates of feature points that can determine the position of the object in the image (2) magnification (scale) of the local area centering on the feature points
(3) Rotation angle of local region (4) 128-dimensional 128-dimensional obtained by dividing the local region into 16 blocks of 4 blocks per side and calculating a histogram of gradient direction in 8 directions (45 degrees each) for each block Local invariant feature quantities such as SIFT feature quantities are registered (recorded) in association with each other.

  The image feature amount data of each page is extracted based on a part or all of the image of the page of the printed material, and a plurality of feature points that can determine the position of the object shown in the image are extracted. Data of local invariant feature amounts of a plurality of feature points obtained by calculating an invariant image feature amount (local invariant feature amount) of a local region centered on the feature point with respect to the feature point. The local invariant feature quantities of a plurality of feature points for a part or all of the image of the printed paper can be obtained by an algorithm such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed Up Robust Features). In addition, a plurality of feature points in a paper image of a printed matter are detected by an algorithm such as Harris corner detection, LoG (Laplacian of Gaussian), DoG (Difference of Gaussian), etc., and LBP (Local Binary Pattern), Hog ) And GLOH (Gradient Location and Orientation Histogram).

  The paper surface related information DB 333 is a database in which paper surface related information associated with image feature amount data for some or all images of each paper surface of a printed matter is stored. For example, in the page related information DB 333, the URL of the site where the information related to the page is disclosed, the text data describing the information related to the page, the image data obtained by imaging the information related to the page, and the moving image data in which the information related to the page is recorded. In addition, audio data or the like in which information relating to the page is recorded is registered (recorded).

  The items of the above-mentioned information stored in each DB of the recording device 330 are an example, and an item other than the above-mentioned items is added to the DB according to the situation of the network service or the like, Some items may be deleted.

  The server data processing unit 340 executes various types of data processing in accordance with applications recorded in the ROM / RAM 320. In particular, the server data processing unit 340 includes a determination unit 341 and a recording unit 342. For example, the determination unit 341 and the recording unit 342 of the present embodiment constitute a recording unit and a determination unit of the present invention, respectively.

  For example, when the determination unit 341 determines that the data received from the communication terminal device 10 via the communication control unit 310 does not include paper-related information associated with the captured image feature data other than the captured image feature data. The image feature data on the paper having the captured image feature data received based on the image feature data for part or all of the image of each printed paper recorded in the feature data DB 332 is the feature data. It is determined whether or not the data DB 332 exists. Specifically, the local invariant feature amounts of a plurality of feature points included in the captured image feature amount data are acquired, and the local invariant feature amounts of the plurality of feature points included in the image feature amount data of each paper surface are obtained as feature amount data. The local invariant features of the plurality of feature points included in the photographed image feature value data are read out from the DB 332 and compared with the local invariant feature amounts of the feature points included in the photographed image feature value data and the image feature value data of each page. It is determined whether or not the image feature amount data of the paper having all the amounts exists.

  If the captured image feature data is for an image of the entire printed sheet, there is paper image feature data that matches the local invariant feature values of multiple feature points included in the captured image feature data. It may be determined whether or not to do so.

  Further, when the determination unit 341 determines that there is image feature data on the page having all the local invariant feature values of the plurality of feature points included in the captured image feature value data, Referring to the information associated with each page related information, the page related information related to the image feature quantity data of the page having all the local invariant feature quantities of the plurality of feature points included in the captured image feature quantity data is related to the page. The information is read from the information DB 333 and supplied to the communication control unit 310. The communication control unit 310 that has received the paper-related information transmits the paper-related information as information embedded in the paper to the communication terminal apparatus 10 that has transmitted the captured image feature data.

  In addition, the recording unit 342 includes paper-related information associated with the captured image feature data other than the captured image feature data by the data received from the communication terminal device 10 by the determination unit 341 via the communication control unit 310, for example. When it is determined that the image data is included, the photographed image feature data is recorded in the database 330 as image feature data on the paper in association with the paper related information. Specifically, the photographed image feature value data is recorded in the feature value data DB 332 as image feature value data on the page, and information relating the image feature value data on the page and the page related information is recorded in the feature value data DB 332. The page related information is recorded in the page related information DB 333. As described above, the captured image feature amount data with respect to the paper image photographed by the user and the paper surface related information that the user wants to embed (relate) to the paper image are recorded in the database 330 in association with each other. When a user takes an image of the same page with a camera included in the communication terminal device 10, the page related information recorded in the database by the user can be provided to other users.

  The recording unit 342 may record the same information separately in the database 330 even if the same paper related information is already recorded in the database 330 for the captured image feature data on the same paper. The combination of the photographed image feature data and the paper related information received from the communication terminal device 10 is referred to by referring to the information associating the paper image feature data and the paper related information previously recorded in the feature data DB 332. It is determined whether or not matching information already exists. If it is determined that the information exists, the recording process may be terminated assuming that the information is recorded, or the information is updated to the same information. Processing may be performed.

  In addition, the recording unit 342 refers to the information relating the image feature data on the paper and the paper related information recorded in advance in the feature data DB 332, and the captured image feature data received from the communication terminal device 10. It is determined whether there is information that matches the paper-related information, and it is determined that there is information that matches the captured image feature data but does not match the paper-related information associated with the captured image feature data. In this case, the process of adding the paper-related information received from the communication terminal device 10 to the paper-related information recorded in the feature data DB 332 in association with the image feature data of the paper that matches the captured image feature data. May be performed.

[4] Operation of Communication System Next, specific processing executed in the communication system 1 of the present embodiment will be described with reference to FIGS. 6 and 7 are flowcharts showing the paper related information display process executed in the communication system 1 of the present embodiment. FIG. 8 is a flowchart showing processing for associating and recording the captured image feature value data on the page and information on the page, which is executed in the communication system 1 of the present embodiment.

  Prior to these processes, it is assumed that each user is in a state where SNS use registration has been completed, that is, various types of information have been registered in the member DB 331. The method for registering use in SNS is not different from that of a conventional SNS, and a description thereof will be omitted. Further, prior to these processes, it is assumed that the user's communication terminal device 10 is logged in to the SNS.

  First, the display process of the paper related information executed in the communication system 1 of the present embodiment will be described. Prior to this processing, the user wants to acquire the paper-related information related to the paper surface of the printed material displayed on the display unit 150, or embeds (sets) the paper-related information related to the paper surface in the paper surface of the printed material, In the screen for selecting whether or not to share with others, it is assumed that the user selects an option for acquiring the paper related information regarding the paper surface of the printed material by operating the operation unit 170, and the server device 30 stores the captured image feature data. Wait for reception.

  In this operation, it is assumed that the terminal data processing unit 140 of the communication terminal device 10 is waiting for an input of a shooting instruction from the user to the operation unit 170.

  First, in an input standby state of a shooting instruction by a user, when the image data generation unit 141 detects a shooting instruction based on a user operation on the operation unit 170 (step S100), the image data generation unit 141 controls the imaging unit 130 to control the page of the printing unit. A part or all of the image is taken, and image data of the taken paper is generated (step S101).

  Next, the image data generation unit 141 records the generated image data in the image data recording unit 102 (step S102).

  Next, the calculation unit 142 detects a plurality of feature points that can determine the position of the object shown in the image based on the image data, and localizes the feature points with respect to the detected plurality of feature points. A region invariant image feature amount (local invariant feature amount) is calculated (step S103).

  As described above, the local invariant feature quantities of a plurality of feature points include the x and y coordinates of the feature points that can determine the position of the object in the image, and the enlargement of the local region centered on the feature points. / Reduction magnification, local area rotation angle, local area is divided into 16 blocks of 4 blocks per side, and a 128-dimensional gradient obtained by calculating a histogram in 8 directions (45 degrees each) for each block SIFT feature value is included.

  Next, the data processing unit 143 controls the network communication unit 110 to transmit data including local invariant feature amounts of a plurality of feature points with respect to the captured image of the paper to the server device 30 as captured image feature amount data ( Step S104). At this time, the terminal data processing unit 140 shifts to a state of waiting for data reception from the server device 30.

  Next, when the communication control unit 310 receives the captured image feature amount data in the server device 30 (step S200), the determination unit 341 includes a part or all of the images of the printed sheets recorded in the feature amount data DB 332. It is determined whether or not the image feature quantity data of the page having the received photographed image feature quantity data exists in the feature quantity data DB 332 by referring to the image feature quantity data for (step S201).

  When the determination unit 341 determines that there is image feature data on the page having all the local invariant feature values of the plurality of feature points included in the captured image feature data (step S201; Yes), the feature data DB 332 A paper image having all the local invariant feature values of a plurality of feature points included in the captured image feature value data with reference to information relating the image feature value data of each page and each page related information recorded in The paper related information associated with the feature amount data is read from the paper related information DB 333 (step S202). Then, the communication control unit 310 transmits the page-related information read by the determination unit 341 to the communication terminal device 10 as information embedded in the page (step S203), and the server management control unit 350 ends this operation.

  On the other hand, when the determination unit 341 determines that there is no image feature data on the page having all the local invariant feature values of the plurality of feature points included in the captured image feature data (step S201; No), communication The control unit 310 notifies the communication terminal apparatus 10 that there is no paper-related information (step S204), and the server management control unit 350 ends this operation.

  Next, in the communication terminal device 10, when the terminal data processing unit 140 receives from the server device 30 via the network communication unit 110 a notification that there is no page related information or page related information (step S105), the display is performed. The control unit 160 creates display data for the page related information or display data indicating that the page related information does not exist, and causes the display unit 150 to display the display data (step S106). Then, the terminal management control unit 180 ends this operation.

  Next, a process executed in the communication system 1 according to the present embodiment to record the captured image feature amount data on the paper in association with the information related to the paper will be described. Prior to this processing, the user wants to acquire the paper-related information related to the paper surface of the printed material displayed on the display unit 150, or embeds (sets) the paper-related information related to the paper surface in the paper surface of the printed material, In the screen for selecting whether to share with others, by operating the operation unit 170, the page-related information related to the page is embedded (set) on the page of the printed material, and the option to share with others is selected Then, the server device 30 stands by in a state of receiving the captured image feature amount data and the paper related information desired to be associated therewith.

  Also in this operation, it is assumed that the terminal data processing unit 140 of the communication terminal device 10 is on standby for a user to input a shooting instruction to the operation unit 170.

  First, in an input standby state of a shooting instruction by a user, when the image data generation unit 141 detects a shooting instruction based on a user operation with respect to the operation unit 170 (step S110), the image data generation unit 141 controls the imaging unit 130 to control the page of the printing unit. A part or all of the images are captured, and image data of the photographed paper is generated (step S111).

  Next, the image data generation unit 141 records the generated image data in the image data recording unit 102 (step S112).

  Next, the calculation unit 142 detects a plurality of feature points that can determine the position of the object shown in the image based on the image data, and localizes the feature points with respect to the detected plurality of feature points. An invariant image feature amount (local invariant feature amount) of the region is calculated (step S113).

  Next, the data processing unit 143 controls the network communication unit 110 so that the user can perform the captured image feature amount data including the local invariant feature amounts of a plurality of feature points with respect to the captured image of the paper surface and the paper surface of the printed material. The page related information regarding the page set as desired to be embedded (registered in association) is transmitted to the server device 30 (step S114). At this time, the terminal data processing unit 140 shifts to a state of waiting for reception of notification from the server device 30 that the captured image feature value data and the paper-related information are recorded in association with each other.

  Next, in the server device 30, when the communication control unit 310 receives the captured image feature amount data and the paper surface related information (step S 210), the determination unit 341 uses the captured image feature amount data as the image feature amount data of the paper surface. In addition to recording in the data DB 332, information associating the image feature quantity data of the page with the page related information is recorded in the feature quantity data DB 332, and the page related information is recorded in the page related information DB 333 (step S211).

  Next, the communication control unit 310 notifies the communication terminal device 10 that the captured image feature value data and the paper-related information have been recorded in association (recording completion) (step S212), and the server management control unit 350 performs this operation. End.

  Next, in the communication terminal device 10, when the terminal data processing unit 140 receives a notification that the captured image feature value data and the paper-related information are recorded in association with each other from the server device 30 via the network communication unit 110, display control is performed. Unit 160 creates display data indicating the notification and causes display unit 150 to display the display data. Then, the terminal management control unit 180 ends this operation.

  As described above, in the present embodiment, the communication system 1 and the server device 30 receive the captured image feature amount data with respect to the image captured by the user with the camera included in the communication terminal device 10, so that the page having the captured image feature amount data is received. The paper-related information recorded in the database 330 in association with the image feature data can be acquired as information embedded in the paper, and the paper-related information can be provided to the user.

  Therefore, the communication system 1 and the server device 30 can use the camera provided in the communication terminal device to print the printed material when the user wants to check information on the printed surface of the printed material such as a book, a newspaper, a brochure, a magazine, a book, or a ticket. By simply reading an image on a paper surface, information relating to the paper surface of the printed material can be provided to the user as information embedded in the paper surface via the communication terminal device. As a result, for users who are interested in information such as URLs, texts, images, videos, and voices related to the printed page without changing the printed page and also for the printed page. Information can be provided and the user can view the information.

  In addition, the communication system 1 and the server device 30 relate to the paper surface as information (information to be associated) that the user wants to embed in the paper, together with the captured image feature amount data for the paper image captured by the user with the camera included in the communication terminal device 10. By receiving the paper surface related information, the captured image feature amount data can be recorded in the database as image feature amount data of the paper surface in association with the paper surface related information. Accordingly, the communication system 1 uses the page-related information recorded in the database 330 in association with the image feature amount data of the page when another user takes an image of the same page with the camera included in the communication terminal device 10. Since the information can be provided to other users, the information can be browsed by other users.

[5] Modification [5.1] Modification 1
In the above embodiment, each DB is provided and managed and controlled in the server device 30. However, each computer system that manages and controls each DB is operated by a separate computer system. Also good. In addition, a computer system serving as a management and control entity for each DB may be installed outside via the network 20.

[5.2] Modification 2
In the above embodiment, when the photographed image feature quantity data and the paper-related information are recorded in association with each other, a notification to that effect is transmitted from the server device 30 to the communication terminal device 10. May be provided to the user.

[5.3] Modification 3
In the above-described embodiment, the printed material of different plates and the printed materials of different works of the same author have different paper surfaces (writing shadows), so the image feature amount data of the printed materials of each plate or of the printed materials of different works by the same author It is not possible to provide the user with the paper-related information associated with each. Therefore, as shown in FIG. 9, the image feature amount data of the printed surface of each plate or the printed surface of each different work by the same author (for example, front cover, back cover, back cover, imprint, etc.) is regarded as the same. In this case, the related printed matter related information is recorded in the database 330, and the paper related information related to the image feature amount data of the printed matter of each plate or the printed matter of each different work by the same author is provided to the user. You may be able to do it. Specifically, when the server device 30 receives, from the communication terminal device 10, captured image feature amount data on a sheet of a certain plate or an author's printed matter, the image feature amount data on the sheet having captured image feature amount data. The page related information associated with the image feature quantity data regarded as the same as the photographed image feature quantity data is read out from the database 330 with reference to the page related information associated with the image and the printed matter related information. May be transmitted to the communication terminal device 10.

[5.4] Modification 4
In the above embodiment, the image feature amount data for each paper surface of the printed matter, for example, the cover, each part, each chapter, each story, each paper surface of each page is managed separately. May be managed in a hierarchy. Specifically, as shown in FIG. 10, the image feature data of the cover of a printed material such as a book or a book, the image feature data of each story of the printed material, and the image feature data of each page of the printed material , And the hierarchization information of the image feature amount data in which the respective image feature amount data is associated may be recorded in the database 330 and managed. By recording and managing such hierarchization information of image feature data in the database 330, the server device 30 captures image data when the captured image feature data on the cover of the printed matter is received from the communication terminal device 10. In addition to the paper surface related information recorded in the database 330 in association with the image feature data of the cover of the printed material having the image feature data, the image feature data of all lower layers of the image feature data of the cover of the printed material, That is, the page-related information recorded in association with the database 330 in the image feature data of each story of the printed material and the image feature data of each page of the printed material associated with the image feature data of the cover of the printed material Is read out, and the paper-related information can be transmitted to the communication terminal device 10. Therefore, the server device 30 can provide not only the cover of the printed matter but also the page-related information of each story and each page to the user. In addition, when the server apparatus 30 receives the captured image feature value data of the door page of each story of the printed material from the communication terminal device 10, the image feature value data of each story of the printed material having the captured image feature value data. In addition to the page-related information recorded in the database 330 in association with each other, the image feature data of each page of the printed material associated with the image feature data of each story of the printed material is recorded in association with the database 330. Paper related information can also be provided to the user.

[5.5] Modification 5
In the above-described embodiment, the calculation unit 142 of the communication terminal device 10 detects a plurality of feature points based on image data generated by a user photographing a paper surface with a camera included in the communication terminal device 10. However, the server data processing unit 340 of the server device 30 may be provided with a calculation unit. As a result, the terminal data processing unit 140 of the communication terminal apparatus 10 can transmit the image data generated by the user photographing the paper surface to the server apparatus 30 via the network communication unit 110 as it is. Then, when the communication control unit 310 of the server device 30 receives the image data, the calculation unit of the server data processing unit 340 detects a plurality of feature points in the image on the paper based on the image data, and detects the detected plurality of the plurality of feature points. A local invariant feature amount of the feature point can be calculated. Further, the terminal data processing unit 140 of the communication terminal device 10 sends the paper-related information related to the paper as information that the user wants to embed in the paper (information to be associated) together with the image data of the paper via the network communication unit 110. Can be sent to. Then, when the communication control unit 310 of the server device 30 receives the image data, the calculation unit of the server data processing unit 340 detects a plurality of feature points in the image on the paper based on the image data, and detects the detected plurality of the plurality of feature points. The local invariant feature amount of the feature point is calculated, and the recording unit 342 of the server data processing unit 340 associates the calculated data including the local invariant feature amount of the plurality of feature points with the paper surface related information as the image feature amount data of the paper surface. Can be recorded.

[5.6] Modification 6
In the above embodiment, the server device 30 associates with the database 330 the image feature amount data of the paper having all the local invariant feature amounts of the plurality of feature points included in the captured image feature amount data received from the communication terminal device 10. The recorded page-related information is transmitted to the communication terminal device 10, but the server device 30 includes the local invariant feature values of a plurality of feature points included in the captured image feature value data received from the communication terminal device 10. The paper surface related information recorded in the database 330 in association with the image feature amount data of the paper surface having a part may be transmitted to the communication terminal device 10.

DESCRIPTION OF SYMBOLS 1 ... Communication system 10 ... Communication terminal device 20 ... Network 30 ... Server apparatus 100 ... Recording part 101 ... Application recording part 102 ... Image data recording part 110 ... Network communication part 130 ... Imaging part 140 ... Terminal data processing part 141 ... Image data Generation unit 142 ... calculation unit 143 ... data processing unit 150 ... display unit 160 ... display control unit 170 ... operation unit 180 ... terminal management control unit 310 ... communication control unit 320 ... ROM / RAM
330 ... Recording device 331 ... Member DB
332 ... feature data DB
333 ... Paper-related information DB
340 ... Server data processing unit 341 ... Determination unit 342 ... Recording unit 350 ... Server management control unit

Claims (7)

Local invariant feature quantities of a plurality of feature points calculated from a plurality of feature points in the detected image of the medium based on image data generated by a user photographing the medium with a camera provided in the communication terminal device Receiving means for receiving from the communication terminal device as captured image feature amount data,
Whether image feature data of a medium having a part or all of the photographed image feature data received by the receiving unit is present in the database based on image feature data of each medium recorded in the database Determination means for determining whether or not
When it is determined that the image feature data of a medium having part or all of the captured image feature data is present in the database, the image feature data of the medium having part or all of the captured image feature data Transmitting means for transmitting medium-related information related to the medium recorded in the database to the communication terminal device;
With
In the database, the medium related information is hierarchized,
The transmission means is
When it is determined that the image feature data of a medium having part or all of the captured image feature data is present in the database, the image feature data of the medium having part or all of the captured image feature data Along with the first medium-related information recorded in association, other medium-related information recorded in a lower layer of the first medium-related information on the database is transmitted to the communication terminal device. Server device. The server device according to claim 1,
When the reception unit receives, from the communication terminal device, medium-related information regarding the medium associated with the photographed image feature data together with the photographed image feature data, the photographed image feature data is stored as the image feature data of the medium. A server device further comprising recording means for recording data in association with medium-related information. Receiving means for receiving, from the communication terminal device, image data generated by a user photographing a medium with a camera provided in the communication terminal device;
Calculating means for detecting a plurality of feature points in the image of the medium based on the image data and calculating local invariant feature amounts of the detected feature points;
Whether image feature data of a medium having a part or all of the locally invariant feature values of the plurality of feature points calculated based on the image feature data of each medium recorded in the database exists in the database Determination means for determining whether or not
When it is determined that image feature data of a medium having a part or all of the local invariant feature quantities of the plurality of feature points exists in the database, a part or all of the local invariant feature quantities of the plurality of feature points are Transmitting means for transmitting to the communication terminal device medium related information relating to the medium recorded in the database in association with image feature data of the medium having;
With
In the database, medium-related information is layered,
The transmission means is
When it is determined that image feature data of a medium having a part or all of the local invariant feature quantities of the plurality of feature points exists in the database, a part or all of the local invariant feature quantities of the plurality of feature points are In addition to the first medium-related information recorded in association with the image feature amount data of the medium having the other medium-related information recorded in the lower layer of the first medium-related information on the database, the communication A server device that transmits to a terminal device. The server device according to claim 3,
When the receiving means receives from the communication terminal device medium-related information related to the medium associated with the image data together with the image data, the calculating means is configured to output a plurality of images in the image of the medium based on the image data. Detecting feature points, calculating local invariant feature quantities of the detected feature points,
The server apparatus further includes a recording unit that records data including the calculated local invariant feature quantities of the plurality of feature points as image feature quantity data of a medium in association with the medium-related information. In the server apparatus according to any one of claims 1 to 4,
The local invariant feature quantities of the plurality of feature points are calculated from a plurality of feature points in a part of the detected image of the medium based on image data generated by photographing a part of the medium. A server device including feature quantities. A server device that provides information to the communication terminal device,
Local invariant feature quantities of a plurality of feature points calculated from a plurality of feature points in the detected image of the medium based on image data generated by a user photographing the medium with a camera provided in the communication terminal device Receiving means for receiving from the communication terminal device data including captured image feature data;
Whether image feature data of a medium having a part or all of the photographed image feature data received by the receiving unit is present in the database based on image feature data of each medium recorded in the database Determining means for determining whether
When it is determined that the image feature data of a medium having part or all of the captured image feature data is present in the database, the image feature data of the medium having part or all of the captured image feature data Transmitting means for transmitting medium-related information relating to the medium recorded in the database in association to the communication terminal device;
Function as
In the database, the medium related information is hierarchized,
The transmission means is
When it is determined that the image feature data of a medium having part or all of the captured image feature data is present in the database, the image feature data of the medium having part or all of the captured image feature data Along with the first medium-related information recorded in association, other medium-related information recorded in a lower layer of the first medium-related information on the database is transmitted to the communication terminal device. Program. A communication method executed by a server device,
Local invariant feature quantities of a plurality of feature points calculated from a plurality of feature points in the detected image of the medium based on image data generated by a user photographing the medium with a camera provided in the communication terminal device A reception step of receiving from the communication terminal device as data including captured image feature data;
Based on the image feature data of each medium recorded in the database, it is determined whether or not the image feature data of the medium having a part or all of the received captured image feature data exists in the database. A determination step to
When it is determined that the image feature data of a medium having part or all of the captured image feature data is present in the database, the image feature data of the medium having part or all of the captured image feature data A transmission step of transmitting medium-related information relating to the medium recorded in the database in association to the communication terminal device;
Including
In the database, the medium related information is hierarchized,
When it is determined that the image feature data of a medium having part or all of the captured image feature data is present in the database, the image feature data of the medium having part or all of the captured image feature data Along with the first medium-related information recorded in association, other medium-related information recorded in a lower layer of the first medium-related information on the database is transmitted to the communication terminal device. Communication method.

Images