JP2013077328A - Information processing system, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily acquire merchandise information of merchandise to be purchased.SOLUTION: A portable terminal 200 images image data of a leaflet caught in a field angle and detects merchandise designation action by a user to the leaflet on the basis of the image data of the leaflet. Next, the portable terminal 200 acquires image data about merchandise being an object of the designation action, and transmits the image data to a purchase management server 100. The purchase management server 100 stores image data about each merchandise carried on the leaflet. Then, the purchase management server 100 receives the image data about the merchandise from the portable terminal 200, specifies the merchandise designated by the user by matching the image data with the image data about each merchandise, and performs association with the user. Next, the purchase management server 100 acquires merchandise information about the merchandise associated with the user, and transmits the merchandise information to the portable terminal 200. The portable terminal 200 performs superimposed display so as to match computer graphic image data showing the merchandise information to a real space.

Description

  The present invention relates to an information processing system, an information processing method, and a program that use augmented reality technology that synthesizes and superimposes real-world video and computer graphics image data.

Navigation technology is used in various fields such as route guidance to a destination and presentation of work procedures during work. Since navigation technology is a technology that assists in facilitating certain actions, a mechanism that can be understood as intuitively as possible has been demanded. As one of such mechanisms, navigation using a technology called augmented reality that superimposes data managed by computers such as characters and images on the real environment and gives supplementary information to the real environment. Technology exists.

  In the theme parks and shopping malls, a technology for three-dimensional navigation using augmented reality (hereinafter referred to as AR) is disclosed for the route to the desired attraction and the display information of the product recommended by the store side (for example, AR) (for example, , See Patent Document 1). Specifically, the position / orientation of the portable information terminal at the time of shooting is calculated, and a marker on the image shot by the portable information terminal is used, and an image obtained by shooting predetermined information previously associated with the marker is displayed. This is a technique for superimposing and displaying on the screen. Note that AR is a technique for providing supplementary information by superimposing and displaying computer-managed data (hereinafter referred to as computer graphics image data) such as characters, graphics, still images, and moving images in an actual environment.

  In addition, a technique is known in which merchandise to be purchased is registered in a list of mobile terminals in advance, and product information of the merchandise is displayed on the mobile terminal when walking near the merchandise registered in the list (for example, , See Patent Document 2).

JP 2004-48674 A JP-A-10-198731

  However, in the technique disclosed in Patent Document 1, the navigation information may include information that is recommended for the store side but is unnecessary for the user. On the other hand, the technology disclosed in Patent Document 2 can display the product information of the products scheduled for shopping registered in the list in advance on the mobile terminal. However, there is a possibility that the product to be purchased is out of stock or the price of the product to be purchased fluctuates when going to shopping.

  Furthermore, in order to be able to understand intuitively in an environment where the situation is constantly changing, such as the price and availability of goods, the timing and type of information that can be obtained is almost as realistic as possible, and immediately It was required to present it in a state where it could be perceived by. SUMMARY OF THE INVENTION An object of the present invention is to provide support for easily acquiring product information and guide information of shopping-scheduled products at an arbitrary timing.

An information processing system according to the present invention connects an augmented reality presentation device having a transmissive or non-transmissive display and a management device that manages product information about each product posted on the flyer via a network. In the information processing system, the augmented reality presentation device includes an imaging unit that captures the image data of the flyer captured at an angle of view when a user of the augmented reality presentation device specifies a product to be purchased. , Based on the image data of the flyer imaged by the imaging unit, a detection unit that detects an operation of designating the product by the user with respect to the flyer, and the product that is the target of the designation operation detected by the detection unit First acquisition means for acquiring image data according to the image data, and image data related to the product acquired by the first acquisition means for the management device First management means for transmitting, and the management device stores flyer information storage means for storing image data relating to each product posted in the flyer, and the first of the augmented reality presentation device. The image data related to the product is received from the transmission unit, and the product specified by the user is identified by matching the image data related to the product with the image data related to each product stored in the flyer information storage unit Acquired by the specifying unit for associating the user with the specified product, the second acquiring unit for acquiring product information regarding the product associated with the user, and the second acquiring unit. Second transmission means for transmitting product information to the augmented reality presentation device, and the augmented reality presentation device displays the second information of the management device on the display. Computer graphic image data indicating the product information transmitted from the signal means, and further comprising a presentation means for superimposing display to match the real space.
An information processing method according to the present invention connects an augmented reality presentation device having a transmissive or non-transmissive display to a management device that manages product information regarding each product posted on the flyer via a network. An information processing method executed by an information processing system, wherein the augmented reality presentation device captures the flyer image captured at an angle of view when a user of the augmented reality presentation device specifies a product to be purchased. An imaging step for imaging data, a detection step for detecting a product designation operation by the user for the flyer based on the image data of the flyer imaged in the imaging step, and a designation operation detected by the detection step Acquired by the first acquisition step of acquiring the image data relating to the product that is the target of the acquisition, and the first acquisition step A first transmission step of transmitting image data relating to the product to the management device, wherein the management device receives the image data relating to the product from the augmented reality presentation device, and The product specified by the user is identified by matching the image data related to the product with the image data related to each product posted in the leaflet stored in the flyer information storage means, and the user and the specified product A step of associating with the user, a second acquisition step of acquiring product information regarding the product associated with the user, and the product information acquired by the second acquisition step as the augmented reality presentation device And a second transmission step for transmitting the data.
The program according to the present invention is an information processing system that connects an augmented reality presentation device having a transmissive or non-transmissive display and a management device that manages product information about each product posted on the flyer via a network. A program for causing a computer to execute an information processing method executed by the system, wherein when the user of the augmented reality presentation device designates a product to be purchased, the computer of the augmented reality presentation device An imaging step of capturing the image data of the flyer captured at a corner, a detection step of detecting a product designating operation by the user for the flyer based on the image data of the flyer captured by the imaging step; Acquire image data related to the product that is the target of the specified action detected in the detection step. 1 acquisition step and a first transmission step of transmitting image data relating to the product acquired in the first acquisition step to the management device, and causing the computer of the management device to execute the extension The image data related to the product is received from the reality presentation device, and the image data related to the product is matched with the image data related to each product listed in the leaflet stored in the leaflet information storage unit. A specifying step of specifying a product designated by the user and associating the user with the specified product; a second acquiring step of acquiring product information relating to the product associated with the user; A second transmission step of transmitting the product information acquired in the acquisition step to the augmented reality presentation device. And wherein the door.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to assist so that the merchandise information and guidance information of shopping planned goods can be acquired easily at arbitrary timings.

It is a figure showing roughly the composition of the shopping navigation system concerning the embodiment of the present invention. It is a figure which shows the hardware constitutions of the shopping management server which concerns on embodiment of this invention. It is a figure which shows the hardware constitutions of the portable terminal which concerns on embodiment of this invention. It is a flowchart which shows the flow of registration processing of the goods planned for purchase in the shopping navigation system concerning the embodiment of the present invention. It is a flowchart which shows the presentation process of the merchandise information of the goods planned for purchase, and the navigation process from the present location to the store in the shopping navigation system according to the embodiment of the present invention. It is a figure which shows the state which the user pointed to the purchase plan goods posted on the flyer, and the state was caught in the angle of view of the imaging part of the portable terminal. It is a figure which shows the example of a display which synthesize | combined the computer graphics image data of merchandise information with the image | video of real space. It is a figure which shows the example of AR display when the product information of the apple which is a user's purchase plan goods can be acquired for 3 stores. It is a figure which shows typically the example of the database structure of flyer information DB. It is a figure which shows typically the example of the database structure of user information DB. It is a figure which shows typically the example of the database structure of merchandise information DB. It is a figure which shows schematically the structure of the shopping navigation system which concerns on other embodiment of this invention. It is a flowchart which shows the flow of the registration process of the goods planned for purchase in the shopping navigation system concerning other embodiments of the present invention. It is a flowchart which shows the presentation process of the merchandise information of a goods planned for purchase, and the navigation process from a present location to a store in the shopping navigation system concerning other embodiments of the present invention. It is a figure which shows the example of a shopping list.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram schematically showing a configuration of a shopping navigation system according to an embodiment of the present invention. As shown in FIG. 1, the shopping navigation system according to the present embodiment includes a shopping management server 100 and a mobile terminal 200, which are connected to each other via a wireless communication line.

  The user browses the flyer 300 and selects a product to be purchased from the products posted on the flyer 300. Next, the user points to the selected purchase planned product, and captures the state in the angle of view of the imaging unit of the mobile terminal 200, whereby image data in a state in which the purchase planned product is pointed is generated by the imaging unit. The portable terminal 200 detects image data of a product scheduled for purchase from the image data generated by the imaging unit. As a method for detecting image data of a product to be purchased, for example, several finger shape patterns in the state of pointing are prepared, and image data matching or similar to those finger shape patterns is detected in the image data. In such a case, a method of recognizing the image data as a finger and detecting an object that exists within a certain distance from the finger in the direction of the finger may be considered.

  When the mobile terminal 200 detects the image data of the purchase planned product, the mobile terminal 200 transmits the image data of the purchase planned product to the shopping management server 100. When the shopping management server 100 receives the image data of the product to be purchased from the mobile terminal 200, the product matching unit 106 stores the image data of each product listed in the flyer 300 stored in the flyer information DB 102, and the mobile terminal. The consistency with the image data of the product scheduled for purchase received from 200 is determined, and it is determined whether or not the product planned for purchase is a product listed on the flyer 300. If the purchase planned product is a product listed on the flyer 300, the registration / inquiry unit 105 registers the purchase planned product and the user in association with each other in the user information DB 101. As a result, the user's planned purchase product is registered in the shopping management server 100.

  After the product to be purchased is registered in the shopping management server 100, when the user wants to know the transition of price information or stock information of the product to be purchased, the mobile terminal 200 is again held over the flyer 300, and the mobile terminal 200 The product to be purchased is captured at the angle of view of the imaging unit. Thereby, the image data of the product to be purchased is transmitted from the mobile terminal 200 to the shopping management server 100. The product matching unit 106 of the shopping management server 100 analyzes the image data received from the mobile terminal 200 to detect that the image data is image data of a product posted on the flyer 300. Then, the registration / inquiry unit 105 determines that the product is a user's planned purchase product. Then, the product information search unit 107 acquires product information such as the latest (current) sales price and inventory information of the product to be purchased and transmits it to the mobile terminal 200. The mobile terminal 200 synthesizes computer graphics image data indicating the latest product information of a product scheduled for purchase with a video in the real space and displays it on the display. At this time, the computer graphics image data indicating the product information is displayed in the vicinity of the real space image so that the product information is related to the product to be purchased. Is synthesized.

  The map information search unit 108 acquires the current location information of the mobile terminal 200 and the address information of the store selling the product to be purchased, and includes a location indicated by the current location information and a location indicated by the address information. Information is acquired from the map information DB 104. Then, the map information search unit 108 generates guide information for guiding the route from the current location of the mobile terminal 200 to the store selling the product to be purchased using the acquired map information, and stores the information in the mobile terminal 200. Send to. Using the guidance information, the mobile terminal 200 displays computer graphics image data that guides the route from the current location to the store that sells the product to be purchased with the video in the real space. Accordingly, by capturing the street at the angle of view of the imaging unit of the mobile terminal 200, the user confirms on the display of the mobile terminal 200 an image in which the route guidance based on the computer graphics image data is combined with the image of the real space on the street. While moving to the store.

  In the present embodiment, an expanded sense of reality is presented by displaying computer graphics image data at a position that matches the image in the real space on the display of the mobile terminal 200. Hereinafter, such display of computer graphics image data that gives an expanded sense of reality may be referred to as AR (Augmented Reality) display. There are various types of HMD such as non-transmission type, video transmission type, and optical transmission type. The composite display (AR display) is to superimpose and display computer graphics image data at a matching position on the real space image displayed on the lens portion of the HMD by any method. In the case of the type method, the image is superimposed and displayed on the image of the real space seen through, and in the case of the video transmission type, the image is displayed on the image of the real space captured by the video camera.

  In addition, the mobile terminal 200 includes a mobile terminal type AR display device that displays a real-space image captured by the imaging unit on a display and displays computer graphics image data in combination with the displayed real-space image. Assuming, the AR display device applicable to the present embodiment is not limited to this. For example, a glasses-type AR display device worn on the user's head, which displays computer graphics image data at a position that matches optically transmitted and projected images in a real space. An AR display device (HMD) may be used, or a head that is installed in the direction of the line of sight of the driver's seat of a moving vehicle and displays computer graphics image data in combination with a real space image that is optically transmitted and projected. An AR display device of an up display type may be used.

  In addition, when computer graphics image data is displayed in alignment with an image in the real space, coordinates between the object in the real space image and the computer graphics image data are performed. As a method of coordinate alignment, based on the position (latitude, longitude) and posture (direction, elevation angle) of the mobile terminal 200, it is estimated where the user holding the mobile terminal 200 is looking in the real space through the imaging unit. Then, the computer graphics image data may be synthesized so as to match the estimated position, or a marker is attached to the object in the real space, the marker is photographed by the imaging unit, and the image data is captured from the photographed image data. You may detect the position of a marker and synthesize | combine computer graphics image data so that it may correspond with the detected position. Furthermore, the position of the object in the real space may be detected by analyzing the image data in the real space captured by the imaging unit, and the computer graphics image data may be synthesized so as to match the detected position.

  FIG. 2A is a diagram illustrating a hardware configuration of the shopping management server 100 according to the present embodiment. The CPU 1001 comprehensively controls each device and controller connected to the system bus. The ROM 1003 or the HD (hard disk) 1009 stores a basic input / output system (BIOS) that is a control program for the CPU 1001 and an operating system program, for example, the shopping management server 100 in the processes shown in FIGS. A program of processing to be executed is stored.

  In the example of FIG. 2A, the HD 1009 is configured to be arranged inside the shopping management server 100. However, as another embodiment, a configuration corresponding to the HD 1009 is arranged outside the shopping management server 100. Also good. Further, for example, the program for performing the processing shown in FIG. 3A and FIG. 3B according to the present embodiment is recorded on a computer-readable recording medium such as a flexible disk (FD) or a CD-ROM. It is good also as a structure supplied from a recording medium, and good also as a structure supplied via communication media, such as the internet.

  A RAM 1002 functions as a main memory, work area, and the like for the CPU 1001. The CPU 1001 implements various operations by loading a program necessary for execution of processing into the RAM 1002 and executing the program.

  The HD 1009 and the FD 1008 function as an external memory. The CPU 1001 implements various operations by loading a program necessary for execution of processing into the RAM 1002 and executing the program.

  The disk controller 1007 controls access to an external memory such as the HD 1007 and the FD 1008. The communication I / F controller 1006 is connected to the Internet or a LAN, and controls communication with the outside by, for example, TCP / IP.

  A display controller 1010 controls image display on the display 1011.

  The KB controller 1004 receives an operation input from the KB (keyboard) 1005 and transmits it to the CPU 1001. Although not shown, in addition to the KB 1005, a pointing device such as a mouse can also be applied to the shopping management server 100 according to the present embodiment as a user operation means.

  The registration / inquiry unit 105, the product matching unit 106, the product information search unit 107, and the map information search unit 108 illustrated in FIG. 1 are stored in, for example, the HD 1009 and loaded into the RAM 1002 as necessary, and execute the program. This is a configuration realized by the CPU 1001. The user information DB 101, flyer information DB 102, product information DB 103, and map information DB 104 have a configuration corresponding to a partial storage area in the HD 1009.

  FIG. 2-2 is a diagram illustrating a hardware configuration of the mobile terminal 200 according to the present embodiment. The CPU 2001 comprehensively controls each device and controller connected to the system bus. The ROM 2003 stores, for example, a program for processing executed by the mobile terminal 200 among the processing illustrated in FIGS. 3A and 3B. Note that the program for performing the processes shown in FIGS. 3A and 3B may be supplied via a communication medium such as the Internet.

  A RAM 2002 functions as a main memory, work memory, and the like for the CPU 2001. The CPU 2001 implements various operations by loading a program necessary for execution of processing into the RAM 2002 and executing the program. A communication I / F controller 2005 is connected to the Internet or a LAN, and controls communication with the outside by TCP / IP, for example.

  The imaging unit 2004 converts a subject image incident through an optical lens and formed on the imaging element into an electrical signal, and outputs moving image data or still image data. A display controller 2006 controls image display on the display 2007. In the present embodiment, it is assumed that the display 2007 employs a liquid crystal display.

  FIG. 3A is a flowchart illustrating a flow of registration processing of a product scheduled for purchase in the shopping navigation system according to the present embodiment. First, the user points to a purchase planned product posted on the flyer 300 and captures the state in the angle of view of the imaging unit 2007 of the mobile terminal 200. As a result, in step S101, the imaging unit 2007 generates image data captured at the angle of view, and the mobile terminal 200 analyzes the image data to detect image data of a product to be purchased designated by pointing. . In step S <b> 102, the mobile terminal 200 transmits the image data of the product scheduled for purchase detected in step S <b> 101 to the shopping management server 100.

  FIG. 4 is a diagram illustrating a state in which the user points to a purchase planned product posted on the flyer 300 and the state is captured by the angle of view of the imaging unit 2007 of the mobile terminal 200. The user selects and points to the apple to be purchased among the products posted on the flyer 300. In the example of FIG. 4, the photographing unit of the mobile terminal 200 keeps this state in the angle of view, and the flyer 300 The display 2007 shows a video in which the user points to the product to be purchased (apple) above. Therefore, in the example of FIG. 4, the image data of the apple is detected as the image data of the product scheduled for purchase. In addition to the method described above, for example, the user may specify a product to be purchased by moving his / her finger so as to surround the apple on the flyer and capturing it in the angle of view of the imaging unit 2007. In this case, the mobile terminal 200 detects the trajectory of the user's finger and determines a product that falls within the trajectory as a purchase-scheduled product. By adopting the method for designating a purchase planned product as described above, it is possible to easily designate a product planned for purchase.

  In step S <b> 201, the shopping management server 100 receives image data of a product scheduled for purchase from the mobile terminal 200. In step S <b> 202, the product matching unit 106 acquires the image data of each product posted in the flyer 300 from the flyer information DB 102 and compares it with the image data of the planned purchase product received from the mobile terminal 200. In step S <b> 203, the product matching unit 106 determines whether the image data of the product scheduled for purchase and the image data of the product listed on the flyer 300 match, whereby the product scheduled for purchase is the product listed on the flyer 300. It is determined whether or not. This matching is determined by performing pattern matching between the image data of the product to be purchased and the image data of the product listed in the leaflet 300, and when it is determined that both are the same or similar, It is determined that the product is on the flyer 300. When it is determined that the product to be purchased is a product posted on the flyer 300, the process proceeds to step S204. On the other hand, if it is determined that the product to be purchased is not a product listed in the flyer 300, the process returns to step S201 and waits for reception of image data of a new product to be purchased.

  FIG. 7 is a diagram schematically illustrating an example of the database structure of the flyer information DB 102. In FIG. 7, the flyer information DB 102 manages the ID (product ID) of each product listed in the flyer 300 and image data in association with each other. In step S201, when the shopping management server 100 receives, for example, apple image data from the portable terminal 200, in step S203, the product matching unit 106 displays the image data of the product to be purchased and each product listed in the flyer 300. As a result of the comparison with the image data, it is determined that the image data of the planned purchase product is identical or similar to the image data of the apple 701, and the purchase planned product is determined to be an apple.

  In step S <b> 204, the registration / inquiry unit 105 acquires the product ID associated with the image data of the product determined to be a purchase planned product from the flyer information DB 102 via the product matching unit 106. In addition, the registration / inquiry unit 105 acquires the transmission source user ID from the header information of the image data of the product to be purchased received in step S201. Then, the registration / inquiry unit 105 registers the user ID and the product ID of the product scheduled for purchase in the user information DB 101 in association with each other.

  FIG. 8 is a diagram schematically illustrating an example of the database structure of the user information DB 101. In FIG. 8, the user information DB 101 manages a user ID and a purchase planned product ID in association with each other. For example, when the product ID of the product to be purchased is “1011” and the user ID “aaa” can be acquired from the header information of the image data of the product to be purchased, as shown in the record 801, the user ID “ “aaa” and purchase planned product ID “1011” are managed in association with each other.

  FIG. 3-2 is a flowchart illustrating a process for presenting product information of a product to be purchased and a navigation process from the current location to the store in the shopping navigation system according to the present embodiment. For example, when the user wants to grasp the latest price information, inventory information, etc. of the product scheduled to be purchased that is already registered in the shopping management server 100, the mobile terminal 200 is held over the product scheduled to be posted on the flyer 300 again. The product to be purchased is captured at the angle of view of the imaging unit 2007. As a result, in step S103, the imaging unit 2007 of the mobile terminal 200 generates image data of a product to be purchased. In step S <b> 104, the mobile terminal 200 transmits the image data of the product scheduled for purchase to the shopping management server 100.

  In step S <b> 205, the shopping management server 100 receives image data of a product scheduled for purchase from the mobile terminal 200. In step S <b> 206, the product matching unit 106 acquires the image data of each product listed in the flyer 300 from the flyer information DB 102 and compares it with the image data of the planned purchase product received from the mobile terminal 200. In step S <b> 207, the product matching unit 106 determines a match between the image data of the product planned for purchase and the image data of the product listed in the flyer 300, so that the product planned for purchase is the product listed in the flyer 300. It is determined whether or not. When it is determined that the product to be purchased is a product posted on the flyer 300, the process proceeds to step S213. On the other hand, when it is not determined that the product to be purchased is a product listed in the flyer 300, the process returns to step S205.

  In step S 213, the registration / inquiry unit 105 acquires a product ID associated with the image data of the product determined to be a purchase planned product from the flyer information DB 102 via the product matching unit 106. In addition, the registration / inquiry unit 105 acquires the transmission source user ID from the header information of the image data received in step S205. In step S208, the registration / query unit 105 determines whether the product ID acquired in step S213 is managed in the user information DB 101 in association with the user ID. If the product ID is associated with the user ID, the process proceeds to step S209. On the other hand, if the product ID is not associated with the user ID, the process returns to step S205.

  In step S209, the product information search unit 107 searches the product information DB 103 for product information corresponding to the product ID acquired in step S213, and transmits the product information to the mobile terminal 200.

  FIG. 9 is a diagram schematically illustrating an example of the database structure of the product information DB 103. In FIG. 9, the product information DB 103 manages the product ID, product name, price (price 1, price 2,...), Store, inventory quantity, and product attributes in association with each other. As the product attribute, if the product is food, the production location, date of production, and the like are managed. If the product is clothing, the color, size, and the like are managed. The data managed in the product information DB 103 may be updated at an arbitrary timing in cooperation with a POS system (Point Of Sales system) not shown in FIG. For example, if the product ID acquired in step S213 is “1011”, in step S209, a record 901 is retrieved from the product information DB 103 shown in FIG. 9 and the product information registered in the record 901 is stored in the mobile terminal 200. Sent. Note that the latest information regarding the price and the stock amount included in the product information is transmitted to the mobile terminal 200. In the product information DB 103, the price of the product posted in the flyer 300 is registered as the initial price (price 1), and the price is additionally registered every time the price is updated (price 2, price 3,...・). In step S <b> 209, the last registered price is transmitted to the mobile terminal 200 from the history of prices registered in this way. As a result, the latest (current) price of the product scheduled for purchase is displayed on the mobile terminal 200. It should be noted that the product information DB 103 may manage the price history, overwrite the price itself, and manage only the latest price. In the product information DB 103, the stock quantity is overwritten at a predetermined timing, and only the latest stock quantity is managed. Accordingly, in step S209, the latest inventory amount is transmitted to the mobile terminal 200. As a result, the latest stock quantity of the product scheduled for purchase is displayed on the mobile terminal 200.

  In step S <b> 105, the mobile terminal 200 receives product information from the shopping management server 100. In step S <b> 106, the mobile terminal 200 displays the received product information computer graphics image data in combination with a real-space video. FIG. 5 is a diagram showing a display example in which computer graphics image data of product information is combined with a real space video. In the example of FIG. 5, computer graphics image data indicating product information 501 and computer graphics image data indicating a tag 502 that is an operation switch for requesting route guidance are displayed in the vicinity of the product to be purchased. .

  In step S110, the mobile terminal 200 determines whether or not there is a request for route guidance (navigation). That is, the mobile terminal 200 determines whether or not the tag 502 has been operated. If the tag 502 is operated, the process proceeds to step S107. On the other hand, if the tag 502 has not been operated, the process ends. In step S <b> 107, the mobile terminal 200 issues a route guidance request to the shopping management server 100.

  In step S <b> 210, the shopping management server 100 receives a route guidance request from the mobile terminal 200. In step S <b> 211, the map information search unit 108 acquires the current location information of the mobile terminal 200, and acquires store information selling the products scheduled for purchase from the product search unit 107. Note that the current location of the mobile terminal 200 can be acquired by using a GPS (Global Positioning System). Although not shown in FIG. 9, the product information DB 103 includes address information of stores that sell products, and the map information search unit 108 plans to purchase from the product information DB 103 via the product information search unit 107. The address information of the store that sells the product shall be acquired. In step S <b> 211, the map information search unit 108 searches for map information including the position indicated by the current location information of the mobile terminal 200 and the position indicated by the address information of the store selling the product to be purchased. In step S212, the map information search unit 108 uses the searched map information to generate guide information for guiding a route from the current location of the mobile terminal 200 to a store selling the product to be purchased, and the mobile terminal 200 is transmitted.

  In step S <b> 108, the mobile terminal 200 receives the guide information from the shopping management server 100. In step S <b> 109, the mobile terminal 200 synthesizes computer graphics image data that guides a route from the current location of the mobile terminal 200 to the store that sells the product to be purchased, using the received guidance information, into a real-space image. And displayed on the display 2007.

  In the embodiment described above, when the user wants to know the product information of the registered shopping planned product, the user performs a predetermined designation operation on the shopping planned product on the flyer, and this is performed on the imaging unit 2004. The product information such as the latest stock quantity and price of the shopping-scheduled product can be easily obtained. In addition, when the association between the user ID and the product ID of the planned purchase product is registered in the user information DB 801, the product information of the planned purchase product can be notified to the mobile terminal 200. Step S209 in FIG. 3-2 may be executed after step S204 in 3-1. That is, when the shopping management server 100 registers the association between the user ID and the product ID of the product to be purchased in the user information DB 801 in step S204, the product information corresponding to the product ID is stored in the product information DB 103 in the subsequent step S209. Are transmitted to the portable terminal 200. By doing in this way, it becomes possible to notify the portable terminal 200 of the product information of the purchase planned product at the time of registering the association between the user ID and the product ID of the planned purchase product.

  In the above-described embodiment, the user needs to hold the mobile terminal 200 over the flyer 300 again in the process of presenting the product information of the product to be purchased. However, the present invention is not limited to this. Another embodiment that does not require the portable terminal 200 to be held over the flyer 300 will be described with a focus on differences from the above-described embodiment.

  FIG. 10 is a diagram schematically illustrating a configuration of a shopping navigation system according to another embodiment. As shown in FIG. 10, the system according to the present embodiment does not include the user information DB 101 in the shopping management server 100 as compared with the system according to the above-described embodiment. Instead, the shopping list 201 is stored in the mobile terminal 200. In preparation. Note that the shopping list 201 is a file in which the product ID of a product to be purchased is recorded, and is managed in association with a product image, a product name, or the like as shown in FIG.

  FIG. 11A is a flowchart illustrating a flow of registration processing of a product planned for purchase in the shopping navigation system according to the present embodiment. Compared to FIG. 3A, step S1101 is added, and step S1201 is added instead of step S204 being deleted.

  In step S <b> 1201, the registration / inquiry unit 105 acquires a product ID associated with image data of a product determined to be a purchase planned product from the flyer information DB 102 via the product matching unit 106. Then, the acquired product ID is transmitted to the mobile terminal 200.

  In step S1101, the product ID is received from the portable terminal 200. The received product ID is stored in the ROM 2003.

  Next, FIG. 11-2 is a flowchart illustrating a process for presenting product information of a product scheduled for purchase and a navigation process from the current location to the store in the shopping navigation system according to the present embodiment. Compared to FIG. 3-2, step S <b> 1102 in FIG. 3-2 is deleted instead of step S <b> 103 and step S <b> 104. Also, step S1202 is added instead of step S208 being deleted from step S205 in FIG. Hereinafter, the present embodiment will be described focusing on differences from the above-described embodiment.

  For example, when the user wants to grasp the latest price information, inventory information, etc. of the product to be purchased that has already been registered in the mobile terminal 200 in step S1101 in FIG. 11A, in step S1102 in FIG. The product ID of the product is transmitted to the shopping management server 100. As a method for specifying the product ID of the product, for example, a shopping list is displayed on the mobile terminal 200 as shown in FIG. 12, and when the product name displayed on the shopping list is tapped, the mobile terminal corresponding to the product name is displayed. The product ID registered in 200 is selected and transmitted to the shopping management server 100.

  In step S1202, the shopping management server 100 acquires a product ID. Thereafter, the process proceeds to step S209 similar to FIG. According to the configuration of the present embodiment, the user does not need the flyer 30 when acquiring the inventory information and price information of the product to be purchased, and thus can be acquired more easily.

  In the above-described embodiment, when the user selects a purchase planned product from the products listed on the flyer 300, the product information of the purchase planned product in the store selling the purchase planned product is displayed as an AR. However, it is not limited to this. For example, the user's planned purchase product may be sold at a plurality of stores. In such a case, in step S209 described above, product information at a plurality of stores selling the product is searched, and the product information is obtained. AR display can also be performed. That is, in the flyer information DB 102, the association between the image data of the flyer products of a plurality of stores and the product ID is managed, and there are a plurality of image data of products that match or are similar to the image data of the product to be purchased. When a combination of any one of the product IDs associated with all the image data and the user ID recorded in the header of the image data of the product to be purchased exists in the user information DB 101, each of the products All product information corresponding to the ID may be acquired from the product information DB 103. Alternatively, a method may be used in which all the product IDs corresponding to image data that matches or is similar to the image data of the product to be purchased are specified from the flyer information DB 102. Alternatively, although not shown in FIG. 7, a column for managing product names is added in the flyer information DB 102, and a product name that is the same as or similar to the product name of a product that matches or is similar to the image data of the product to be purchased. A method of specifying all product IDs of products managed in the column “product name” may be used. Alternatively, instead of the image data of the product to be purchased, a method may be used in which all product IDs of products that match or are similar to the product name or product image data managed in association with the product ID received from the mobile terminal 200 may be used. . And all the acquired merchandise information is transmitted with respect to the portable terminal 200 from the shopping management server 100, and the portable terminal 200 can carry out AR display of the merchandise information of several stores regarding a purchase plan commodity.

  FIG. 6 is a diagram illustrating an example of an AR display when product information of apples that are products to be purchased by the user can be acquired for three stores. In FIG. 6, 601, 602, and 603 are product information about apples at A store, B store, and C store, respectively, and indicate current apple prices, stock quantities, and the like at each store. Thereby, the user can refer to product information of apples other than the A store that issued the flyer. For example, if an apple is sold at a cheaper price than the A store in a store other than the A store, or if the stock is out of the A store and the stock remains in the other store, buy the apple in the other store You may change your shopping destination. In this case, in step S110, the mobile terminal 200 determines whether the tag 604 relating to the product of which store is operated among the plurality of stores in which the product information is AR-displayed, and then in step S107, the mobile terminal 200 200 issues a request for route guidance to the store to the shopping management server 100. In FIG. 6, a store selection form of the tag 604 that is an operation switch for requesting route guidance is displayed in a pull-down menu, but other configurations are possible. For example, a tag for each of a plurality of stores may be provided, or voice recognition for recognizing a store name uttered by the user may be used.

  In addition, if you originally planned to purchase apples at store A, but you are selling apples at a cheaper price than store A at other stores, or if you are out of stock at store A and have stock at other stores, The management server 100 may automatically detect it, generate guidance information for navigating to the multi-store with respect to the mobile terminal 200, and transmit it to the mobile terminal 200. At this time, in step S211 and step S212, the shopping physical server 100 compares the product information in the plurality of stores searched in the above-described step S209 at an arbitrary timing, and generates guide information based on a predetermined rule. The rule is stored in the ROM 1003 or the HD (hard disk) 1009. For example, the route to the nearest store among the plurality of stores searched in the current location and in step S209 is guided or searched in step S209. The route that guides the route to the store with the most stock among the plurality of stores that have been acquired, or separately acquires information indicating the congestion status of people in each route of the plurality of stores searched in step S209, and the route that is most easily passed It may be possible to guide the user. One rule may not be fixedly applied, but a plurality of rules may be applied in combination, or the combination may be dynamically changed and applied. For example, by first applying the rule “Guide the route to the store with the shortest distance”, guide information to the store A is generated, and the user goes to the store A based on the information displayed on the mobile terminal 200 as AR. On the way, when the shopping management server 100 refers to the product information DB 103 and grasps that the product is not in stock, the purchase management server 100 purchases based on another rule “Guide the route to the store with the most stock”. The management server 100 generates new guidance information and displays the AR on the mobile terminal 200. Further, not only the route from the current location to the store but also the location where the target product is displayed in the store may be guided. Furthermore, when the user has reached the vicinity of the target product but cannot find the product, the posture may be detected by the posture detection function of the mobile terminal 200 to guide the user to see the direction.

  Further, as another example of information displayed on the mobile terminal 200 as an AR, there is an alarm that a product registered in the past is posted on the flyer currently being viewed by the user, a discount price or a discount period of the product, etc. is there. That is, once consumables such as powdered milk and tissue that are regularly required by the user are registered once by the method described with reference to FIG. 3A, the user holds the portable terminal 200 over the leaflet issued thereafter, and the leaflet If the registered product is posted, it is detected by the method described in FIG. 3-2 (steps S205 to S208), and the alarm and the discount price or discount period of the product are processed by the same processing as step S209. Or the like may be transmitted to the mobile terminal 200. As a result, information such as an alarm is displayed in an AR display while the user looks at the flyer over the mobile terminal 200.

  As another embodiment of the present invention, the shopping management server 100 automatically selects a product according to the user's preference from the flyer, and guides the user to the selected product and the store that handles the product. May be. Specifically, the user master information is registered in the shopping management server 100. Information that is registered as master information includes personal preferences (whether or not quality is important, whether or not seasonal things are important, whether or not there are allergies, the target food cost for one week, the target calorie for the day, Whether you want to increase the number of items with the same ingredients and the same cooking procedure halfway, whether you want to finish shopping in the neighborhood, family preferences (Father: oil mode, children: meat center, adults: dry curry, children: Sweet curry), cooking history (cooking success or failure, leftover history, family reaction). Moreover, the shopping management server 100 acquires the flyer information of each store. As a method of acquiring flyer information, the image of the product, the price, etc. are acquired from the electronic flyer data of each store site, or the image of the product is obtained from the image data obtained by taking a photograph of the folded flyer of the newspaper, The method of acquiring a price etc. is mentioned. Note that the flyer information may be obtained from a plurality of stores. For example, it is conceivable to obtain leaflet information about curry ingredients from a store A which is a supermarket, and to obtain leaflet information about seasonings from a store B which is a cheap liquor store. On the other hand, when the user views a cooking program, a restaurant introduction program, a cooking researcher's blog, and a recommended cooking recipe on the mobile terminal 200 and operates a button such as “Looks good” on the screen, the mobile terminal 200 displays an image. Recognition, voice recognition, character recognition, and the like are performed, and the analysis result is combined with master information and flyer information to select a food and a store.

  The shopping management server 100 generates guide information for guiding the route from the current location of the mobile terminal 200 to the sales store based on the food and the sales store thus selected, and transmits the guidance information to the mobile terminal 200. To do. In the portable terminal 200, using the guidance information, computer graphics image data for guiding from the current location to the sales store is combined with the video in the real space and displayed. Thereby, the route to the store where the food that matches the user's preference can be purchased is automatically guided. In addition, when a user moves into a sales store and holds the mobile terminal 200 in the store, a product such as the price of the food, the amount of inventory, the production area, etc. is displayed on the image of the real space in which the shelf on which the food is placed is reflected. The computer graphics image data indicating the information may be combined and displayed, or the computer graphics image data displayed in an arrow indicating the position of the shelf may be combined and displayed on the image in the real space.

  Also, the history of ingredients actually purchased by the user may be registered in the master information and used when selecting the next ingredients. Furthermore, you may make it output the log | history of the purchased foodstuff as some information of a household account book.

  In the above-described embodiment, at the time of registration of a product planned for purchase or acquisition of product information of a product planned for purchase, image data of the product planned for purchase is captured by the imaging unit 2004 and the image data is captured from the mobile terminal 200 to the shopping management server 100. However, the present invention is not limited to this. That is, when a marker corresponding to a product is posted on the leaflet, the marker may be photographed by the imaging unit 2004 and image data of the marker may be transmitted from the mobile terminal 200 to the shopping management server 100. The shopping management server 100 manages the association between the marker and the product, and identifies the planned purchase product corresponding to the marker by analyzing the received image data of the marker. Then, the shopping management server 100 compares the ID of each product listed in the leaflet 300 stored in the leaflet information DB 102 with the ID of the planned purchase product, and if there is a matching product, the purchase planned product is the flyer. And the user information DB 101 is managed by associating the user with the product to be purchased.

  In addition, when the user wants to obtain product information of the product to be purchased, the user also captures a marker corresponding to the product to be purchased with the imaging unit 2004, and the image data of the marker is transferred from the portable terminal 200 to the shopping management server 100. Send to. The shopping management server 100 identifies the product to be purchased corresponding to the marker by analyzing the received image data of the marker. Then, the shopping management server 100 compares the ID of each product listed in the leaflet 300 stored in the leaflet information DB 102 with the ID of the product to be purchased. When there is a matching product, the shopping management server 100 determines whether or not the association between the ID of the planned purchase product and the user is registered in the user information DB 101. When the association between the ID of the planned purchase product and the user is registered in the user information DB 101, it is determined that the planned purchase product has been specified by the user in the past. Therefore, the shopping management server 100 transmits the product information of the planned purchase product to the mobile terminal 200 of the user. As described above, it is also possible to easily obtain the product information of the product scheduled for purchase at an arbitrary timing by using the marker corresponding to each product posted on the flyer.

  In the above-described embodiment, the flyer has been described as an example. Needless to say, the present invention can be similarly applied to any advertising medium such as a poster or a TV commercial attached to the city.

  100: Shopping management server, 200: Mobile terminal, 300: Flyer, 101: User information DB, 102: Flyer information DB, 103: Product information DB, 104: Map information DB, 105: Registration / inquiry unit, 106: Product matching Part 107: Product information search part 108: Map information search part

Claims (13)

An information processing system that connects an augmented reality presentation device having a transmissive or non-transmissive display and a management device that manages product information about each product posted on the flyer via a network,
The augmented reality presentation device includes:
When the user of the augmented reality presentation device specifies a product to be purchased, an imaging unit that captures the image data of the flyer captured at an angle of view;
Detection means for detecting an operation of designating a product by the user for the flyer based on the image data of the flyer imaged by the imaging means;
First acquisition means for acquiring image data relating to the product that is the target of the specified action detected by the detection means;
First transmission means for transmitting image data relating to the product acquired by the first acquisition means to the management device;
The management device
Flyer information storage means for storing image data relating to each product posted on the flyer;
The image data related to the product is received from the first transmission unit of the augmented reality presentation device, and the image data related to the product is matched with the image data related to each product stored in the flyer information storage unit Identifying means for identifying the product designated by the user, and identifying means for associating the user with the identified product;
Second acquisition means for acquiring product information related to the product associated with the user;
Second transmission means for transmitting the product information acquired by the second acquisition means to the augmented reality presentation device;
The augmented reality presentation device includes:
An information processing system further comprising: presentation means for superimposing and displaying computer graphic image data indicating the product information transmitted from the second transmission means of the management device on the display so as to match the real space. . The imaging means of the augmented reality presentation device captures image data of a marker corresponding to the product as image data related to the product when a marker corresponding to the product is posted in the flyer,
The management device manages a correspondence relationship between the marker and each product listed in the leaflet, and the specifying unit of the management device is an image of the marker corresponding to the product received from the augmented reality presentation device. The information processing system according to claim 1, wherein the product specified by the user is specified by analyzing data.   3. The information processing according to claim 1, wherein the second acquisition unit of the management apparatus acquires the latest inventory information related to the product associated with the user as product information related to the product. system.   The second acquisition unit of the management device acquires the last registered product information, and the second transmission unit of the management device uses the last registered product information as the augmented reality presentation device. The information processing system according to any one of claims 1 to 3, wherein the information processing system is transmitted to the information processing system. The specifying unit of the management device further registers a correspondence between the user and the specified product in a database provided in the management device,
When the second acquisition unit of the management device receives image data related to the product posted in the leaflet from the augmented reality presentation device, the user related to the user recorded in the header of the image data It is determined whether or not the combination of the information and the image data matches the association between the user registered in the database and the identified product, and if it matches, the product related to the product associated with the user The information processing system according to claim 1, wherein information is acquired.   When the combination of the user information and the image data matches the correspondence between the user registered in the database and the specified product, the second transmission unit of the management device 6. The information processing according to claim 5, wherein information for notifying that the posted product is a product registered in the database in the past is also transmitted to the augmented reality presentation device. system.   The second acquisition unit of the management device selects a product according to the personal information of the user, acquires product information related to the product, and the second transmission unit of the management device uses a product related to the product. The information processing system according to claim 1, wherein information is transmitted to the augmented reality presentation device.   The information processing system according to any one of claims 1 to 7, wherein the management device generates the guidance information based on a predetermined rule.   The information processing system according to claim 8, wherein the predetermined rule is a rule that is dynamically selected from a plurality of rules.   The plurality of rules include a rule related to a distance from the current location of the augmented reality presentation device to a store until the product is sold, and a rule related to the inventory of the product. The information processing system described. The augmented reality presentation device includes:
It further has a detecting means for detecting the posture of the augmented reality presentation device,
The presenting means of the augmented reality presentation device presents a direction to be viewed by the user with respect to the display location of the product based on the attitude of the augmented reality presentation device detected by the detecting means. The information processing system according to any one of claims 1 to 10. Information executed by an information processing system that connects an augmented reality presentation device equipped with a transmissive or non-transmissive display and a management device that manages product information about each product posted on the flyer via a network A processing method,
The augmented reality presentation device includes:
When the user of the augmented reality presentation device specifies a product to be purchased, an imaging step of capturing the image data of the flyer captured at an angle of view;
A detection step of detecting a product designating operation by the user for the flyer based on the flyer image data imaged in the imaging step;
A first acquisition step of acquiring image data relating to the product that is a target of the designated operation detected by the detection step;
A first transmission step of transmitting image data relating to the product acquired in the first acquisition step to the management device;
The management device
Receiving image data related to the product from the augmented reality presentation device, and matching the image data related to the product with image data related to each product posted in the flyer stored in the flyer information storage unit Identifying the product designated by the user, and a step of associating the user with the identified product;
A second acquisition step of acquiring product information related to the product associated with the user;
And a second transmission step for transmitting the product information acquired in the second acquisition step to the augmented reality presentation device. Information executed by an information processing system that connects an augmented reality presentation device equipped with a transmissive or non-transmissive display and a management device that manages product information about each product posted on the flyer via a network A program for causing a computer to execute a processing method,
In the computer of the augmented reality presentation device,
When the user of the augmented reality presentation device specifies a product to be purchased, an imaging step of capturing the image data of the flyer captured at an angle of view;
A detection step of detecting a product designating operation by the user for the flyer based on the flyer image data imaged in the imaging step;
A first acquisition step of acquiring image data relating to the product that is a target of the designated operation detected by the detection step;
A first transmission step of transmitting image data relating to the product acquired in the first acquisition step to the management device;
In the computer of the management device,
Receiving image data related to the product from the augmented reality presentation device, and matching the image data related to the product with image data related to each product posted in the flyer stored in the flyer information storage unit Identifying the product designated by the user, and a step of associating the user with the identified product;
A second acquisition step of acquiring product information related to the product associated with the user;
And a second transmission step of transmitting the product information acquired in the second acquisition step to the augmented reality presentation device.

Images