JP2013235478A - Computer program and portable information terminal - Google Patents

Abstract

PROBLEM TO BE SOLVED: To support achievement of both advantage of providing merchandise information by a merchandise introduction medium, and advantage of merchandise marketing on an EC site.SOLUTION: A merchandise information holding unit 64 of a user terminal 14 holds images and attribute information of each of a plurality of merchandise items that are published on a paper medium. A camera unit 58 takes images of the merchandise items published on the paper medium. A collation unit 76 collates a plurality of merchandise images held in the merchandise information holding unit 64 with the images taken by the camera unit 58. An order processing unit 82 transmits order information of a merchandise item that matches with the images taken by the camera unit 58 to a marketing support device 12 on the basis of the attribute information of the merchandise item.

Description

  The present invention relates to a data processing technique, and more particularly to a technique for supporting purchase of a product.

  There are many EC (Electronic commerce) sites on the Internet. Companies have the advantage of reducing labor costs required for product sales by mail-ordering products using EC sites. Some EC sites provide users with web pages describing various information such as product images, so-called electronic catalogs, to assist users in selecting products.

JP 2002-342660 A

  On the other hand, there is a product display medium for introducing a product such as a catalog medium, a paper medium such as a flyer or a poster, a monitor medium such as TV shopping, and the EC site is not linked. For example, a paper medium is easy to compare a plurality of posted products, and is highly convenient for the user. In addition, since paper media is generally printed matter, product photos and the like are beautiful, and it is easy to urge users to purchase. The main object of the present invention is to realize both the merit of providing merchandise information using various media including paper media and the merit of selling merchandise using EC by ICT technology.

  In order to solve the above problems, a computer program according to an aspect of the present invention provides an image of a product captured by an imaging device from among a plurality of products displayed on a predetermined medium on a portable information terminal including the imaging device. Captured image acquisition function to be acquired, collation function for collating images of a plurality of products stored in the mobile information terminal with captured images acquired by the captured image acquisition function, and a plurality of products stored in the mobile information terminal The order function for executing the order processing of the product is realized based on the attribute information of the product that matches the captured image.

  Another aspect of the present invention is a portable information terminal. This portable information terminal includes, for a plurality of products displayed on a predetermined medium, an image and attribute information holding unit for each product, an imaging unit for imaging the product displayed on the medium, and a plurality of items held by the holding unit. A product image and a captured image obtained by the imaging unit, and an order processing unit that executes order processing for the product based on the attribute information of the product that matches the captured image.

  It should be noted that any combination of the above-described constituent elements and a representation of the present invention converted between an apparatus, a method, a system, a program, a recording medium storing the program, and the like are also effective as an aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, coexistence of the merit of merchandise information provision by a merchandise display medium and the merit of merchandise sales by EC can be supported.

It is a figure which shows the goods sales system of embodiment. It is a block diagram which shows the function structure of the sales assistance apparatus of FIG. It is a figure which shows the structure of the page information hold | maintained at a page information holding part. It is a figure which shows the structure of the merchandise information hold | maintained at a merchandise information holding part. It is a block diagram which shows the function structure of the user terminal of FIG. It is a flowchart which shows operation | movement of a user terminal. It is a flowchart which shows the page recognition process of S18 of FIG. 6 in detail. It is a figure which shows a page code imaging | photography screen. It is a figure which shows the error dialog in a page recognition process. It is a flowchart which shows the goods selection process of S22 of FIG. 6 in detail. It is a figure which shows a goods photography screen. It is a figure which shows a goods selection screen. It is a figure which shows the error dialog in a goods recognition process. It is a figure which shows the goods selection screen which displayed the publication goods of the page as a list.

Before describing the configuration of the embodiment, an outline will be described first.
The user terminal according to the embodiment is a portable information terminal (for example, a smartphone, a mobile phone, a tablet terminal, or the like) equipped with a camera. The user terminal externally manages product images showing the appearance of products printed on paper media (hereinafter collectively referred to as “paper catalogs”) such as booklets, catalogs, flyers, and posters, and attribute information about the products. Obtain in advance from the server. The user captures the product image printed on the paper catalog with the camera of the user terminal and imports it into the user terminal. The user terminal identifies a product that matches the product image captured by the user from among the product images acquired in advance from the management server, and displays an order screen indicating attribute information of the product.

  On the order screen, information on one or more related products (for example, clothes of different sizes or different colors) corresponding to the product image captured by the user is presented. The user can select a desired product on the order screen and place an order online. According to the user terminal of the embodiment, the user can examine a product to be purchased while looking at the paper catalog, and can execute the order process by an intuitive operation of photographing a desired product with a camera. As a result, offline information provision through paper catalogs and online information distribution through ECs can be organically linked through intuitive user operations, achieving both benefits and product purchases. The convenience of the user can be improved. Further, in the paper catalog, it is not necessary to print a barcode or the like in association with all of the posted products, and it is possible to suppress the loss of the design and beauty of the paper catalog.

  FIG. 1 shows a product sales system according to an embodiment. The merchandise sales system 16 includes a sales management device 10 that is a PC operated by a person in charge of a retail company, a sales support device 12 that is a web server installed in an intermediary company, and a user who is a smartphone operated by an end user. A terminal 14 is provided. The merchandise sales system 16 is an information system that realizes an image recognition shopping service in the user terminal 14.

  A person in charge at a retail company places a product image indicating the appearance of the product on a paper catalog. Also, the page information of the paper catalog and the product information posted in the paper catalog are registered in the sales support device 12 via the sales management device 10. In the paper catalog of the embodiment, an image of a unique code for uniquely identifying the paper catalog in a part of each page (for example, the upper right of the page) and uniquely identifying the page in the paper catalog. Is printed. This code is a bar code, a QR code (registered trademark) or the like as will be described later with reference to FIG. 8, and is hereinafter referred to as a “page code”.

  The basic operation of the merchandise sales system 16 will be described. The sales support apparatus 12 distributes the page information and product information registered from the sales management apparatus 10 to the user terminal 14. The user terminal 14 locally identifies a product desired by the user who has seen the paper catalog, and transmits the order information to the sales support apparatus 12, and the sales support apparatus 12 transfers the order information to the sales management apparatus 10. . The sales management device 10 transmits processing completion information corresponding to the order information to the sales support device 12, and the sales support device 12 transfers the completion information to the user terminal 14.

  FIG. 2 is a block diagram showing a functional configuration of the sales support apparatus 12 of FIG. The sales support apparatus 12 includes a page information holding unit 20, a product information holding unit 22, a management screen providing unit 24, a page information registration unit 26, a page information distribution unit 28, a product information registration unit 30, and product information. A distribution unit 32, an order information receiving unit 34, an order information providing unit 36, a completion information receiving unit 38, and a completion information providing unit 40 are provided.

  Each block shown in the block diagram of the present specification can be realized in terms of hardware by an element such as a CPU of a computer or a mechanical device, and in terms of software, it can be realized by a computer program or the like. The functional block realized by those cooperation is drawn. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by a combination of hardware and software. For example, each block in FIG. 2 may be stored in a recording medium as a program module and installed in the storage of the sales support apparatus 12. Then, the function of each block may be realized by reading the program module corresponding to each block into the main memory and executing it by the CPU. The same applies to FIG. 5 described later.

  The page information holding unit 20 is a storage area for holding attribute information (hereinafter referred to as “page information”) regarding each page included in the paper catalog. FIG. 3 shows the configuration of the page information held in the page information holding unit 20. A page information holding unit 62 of the user terminal 14 to be described later holds similar page information. In the page ID field, a page ID serving as both the paper catalog identification information and the page identification information in the paper catalog is set. Therefore, even if a paper catalog is issued regularly by the same retail company, for example, the third page of the spring issue and the third page of the summer issue are the type of paper catalog (number of volumes, number of issues, date of issue) Etc.), the page IDs are different.

  In the expiration date field, an expiration date for each paper catalog or an expiration date for each page in the paper catalog is set. In the registration date field, a time stamp indicating the date and time when the page information is registered by the retail company is set. In the page description field, the description text of the page in the paper catalog, for example, the page number, the title of the page, the outline of the product described on the page, and the number of products are set. In the image ID field, an image ID that is identification information of each of one or more product images printed on the page is set.

  Returning to FIG. 2, the product information holding unit 22 is a storage area for holding product information published in a paper catalog. FIG. 4 shows a configuration of product information held in the product information holding unit 22. A product information holding unit 64 of the user terminal 14 described later also holds similar product information. The product image ID is set in the image ID field. In the product image field, image data indicating the appearance (combination of shape, pattern, and color) of the product posted in the paper catalog is set. This image data may be data indicating a characteristic part of the product image for comparison (image matching) with a captured image in the user terminal 14 described later.

  The expiration date of the product information is set in the expiration date field. Different expiration dates may be set between a plurality of products posted on the same page. In the registration date field, a time stamp indicating the date and time when the merchandise information is registered by the retail company is set.

  In the product attribute field, attribute information related to the product is set. Specifically, a product as a unit of one order is called a “unit product”, and attributes of each of one or more unit products corresponding to one product image (the first unit product attribute and the second unit product in FIG. 4). Unit product attribute). It can be said that these unit products are products having different product attributes, and include, for example, products of different sizes and related (similar) products of different colors. In the product attribute field, attribute information such as a product ID, a thumbnail image, a product name, a size, a color, and a price is set for each unit product.

  Returning to FIG. 2, the management screen providing unit 24 provides the sales management apparatus 10 with a web page of a management screen for a person in charge of the retail company to register page information and product information. This management screen is also used to register completion information when a retail company accepts an order for a product.

  The page information registration unit 26 receives the page information input on the management screen from the sales management apparatus 10 and records the page information in the page information holding unit 20. When the page information registration unit 26 receives new page information or receives update information of existing page information, the page information registration unit 26 sets the current date and time as the registration date and time of the page information.

  The page information distribution unit 28 periodically provides the page information recorded in the page information holding unit 20 to the user terminal 14. For example, page information is distributed once a day when it is detected that a predetermined time has been reached. Further, the page information distribution unit 28 receives the page ID and registration date / time of the page information held by the user terminal 14 from the user terminal 14 as needed. When page information with a newer registration date and time is recorded in the page information holding unit 20 with the same page ID, page information with a new registration date and time is provided to the user terminal 14.

  The merchandise information registration unit 30 receives the merchandise information input on the management screen from the sales management apparatus 10 and records the merchandise information in the merchandise information holding unit 22. When new product information is received or when update information of existing product information is received, the product information registration unit 30 sets the current date and time as the registration date and time of the product information.

  The product information distribution unit 32 periodically provides the product information recorded in the product information holding unit 22 to the user terminal 14. For example, product information is distributed once a day when it is detected that a predetermined time has been reached. Further, the product information distribution unit 32 receives the image ID and registration date and time of product information held by the user terminal 14 from the user terminal 14 as needed. If product information with a newer registration date and time is recorded in the product information holding unit 22 with the same image ID, product information with a new registration date and time is provided to the user terminal 14. Further, the product information distribution unit 32 receives a request for providing product information specifying a specific image ID from the user terminal 14 as needed, and provides the user terminal 14 with product information identified by the image ID.

  The order information receiving unit 34 receives from the user terminal 14 order information including attribute information of a product to be ordered (that is, a unit product selected by the user) and personal information of the user. The order information providing unit 36 transfers the order information to the sales management device 10.

  The completion information receiving unit 38 receives completion information from the sales management apparatus 10 indicating that the order information is received and the delivery procedure for the ordered product is completed. The completion information providing unit 40 transfers the completion information to the sales management apparatus 10. The completion information providing unit 40 may notify the completion information by e-mail, or may notify the completion information by a push notification defined by the user terminal 14 (smart phone).

  FIG. 5 is a block diagram showing a functional configuration of the user terminal 14 of FIG. The user terminal 14 includes a touch screen 50, an operation detection unit 52, a display control unit 54, a communication processing unit 56, a camera unit 58, a user information holding unit 60, a page information holding unit 62, and product information holding. Unit 64 and product purchase support unit 66.

  The touch screen 50 is an input / output device in which a display device (for example, a liquid crystal display) and an input device (for example, a touch panel) are integrated. The operation detection unit 52 detects user operations input to the touch screen 50 and various buttons of the user terminal 14. The display control unit 54 controls the display content of the touch screen 50, and displays the display data output from the product purchase support unit 66 on the touch screen 50, for example. The communication processing unit 56 transmits / receives data to / from the sales support apparatus 12 via a wireless communication network or a wired communication network. The camera unit 58 is an imaging device mounted on the user terminal 14.

  The user information holding unit 60 is a storage area that holds user information necessary for purchasing a product from a retail company in the product sales system 16. In other words, the user information holding unit 60 holds personal information of a user that needs to be disclosed to a retail company in electronic commerce. For example, the user information holding unit 60 holds the user's name, date of birth, address, telephone number, credit card number, and the like. The page information holding unit 62 holds the page information explained in FIG. 3, and the product information holding unit 64 holds the product information explained in FIG.

  The product purchase support unit 66 is an application for supporting product purchase by the user (hereinafter also referred to as “product purchase application”). In the merchandise sales system 16, the brokerage company provides the merchandise purchase application to the user free of charge. For example, the product purchase application is registered in various online stores on the Internet so that it can be downloaded free of charge, and the user terminal 14 downloads and installs an installer of the product purchase application from various online stores to install the product purchase support unit 66. You may demonstrate the function. As a modification, the sales support device 12 may provide a free download service for the product purchase application.

  The product purchase support unit 66 includes a user information registration unit 68, a page information acquisition unit 70, a product information acquisition unit 72, a captured image acquisition unit 74, a matching unit 76, an order processing unit 82, and an error notification unit 88. including. The user information registration unit 68 requests the user to input personal information necessary for purchasing the product when installing the product purchase application, and stores the personal information input by the user in the user information holding unit 60.

  The page information acquisition unit 70 acquires page information that is regularly distributed by the sales support apparatus 12 and stores the page information in the page information holding unit 62. Further, the page information acquisition unit 70 inquires of the sales support device 12 whether or not the page information stored in the page information holding unit 62 is the latest information during the page recognition process by the page recognition unit 78 described later. Specifically, the combination of the page ID of each page information stored in the page information holding unit 62 and the registration date and time is transmitted to the sales support apparatus 12. If there is new page information or updated page information, the page information is acquired and stored in the page information holding unit 62.

  The product information acquisition unit 72 acquires product information that is regularly distributed by the sales support device 12 and stores the product information in the product information holding unit 64. In addition, the product information acquisition unit 72 inquires of the sales support apparatus 12 whether or not the product information stored in the product information holding unit 64 is the latest information in the product recognition process by the product recognition unit 80 described later.

  Specifically, the product information acquisition unit 72 transmits a combination of the image ID defined by the page information specified in the page recognition process and the registration date and time of the product information corresponding to the image ID to the sales support apparatus 12. If the product information has been updated, the updated product information is acquired and stored in the page information holding unit 62. Further, when the product information of the image ID specified by the page information specified in the page recognition process has not been acquired (when not stored in the product information holding unit 22), a product information acquisition request specifying the image ID is issued. It transmits to the sales support apparatus 12. Then, product information including the image ID is acquired and stored in the page information holding unit 62.

  The captured image acquisition unit 74 acquires a paper catalog image captured by the camera unit 58. Specifically, a page code image and a product image captured by the user using the camera unit 58 are acquired from the camera unit 58 and stored in a predetermined storage area such as a memory.

  The collation unit 76 collates the captured image acquired by the captured image acquisition unit 74 with the page information stored in the page information holding unit 62. Further, the captured image acquired by the captured image acquisition unit 74 and the product information stored in the product information holding unit 64 are collated. The collation unit 76 includes a page recognition unit 78 that executes page recognition processing and a product recognition unit 80 that executes product recognition processing.

  The page recognition unit 78 specifies a page ID indicated by the page code from the captured image of the page code (for example, a barcode or a QR code (registered trademark)) by a known method. Then, with reference to the expiration date of the page information associated with the specified page ID, it is determined whether or not the page information is valid (the expiration date has not expired). As described above, the expiration date of the page information may be set for each paper catalog. In this case, the page recognition unit 78 determines whether the paper catalog is valid as a whole (the expiration date has expired). It can be said that it is determined. When the page ID cannot be identified from the captured image or when the expiration date of the page information has expired, the error notification unit 88 is notified that an error has occurred in the page recognition process, and an error dialog is displayed.

  The product recognition unit 80 acquires one or more image IDs set in the image ID field of the page information specified by the page recognition unit 78. Then, one or more pieces of product information in which the acquired image ID is set in the image ID field is specified from among a plurality of pieces of product information held in the product information holding unit 64. The merchandise recognition unit 80 collates the merchandise image set for each of the merchandise information specified in this way and the captured image acquired by the captured image acquisition section 74, and includes the merchandise information including the merchandise image that matches the captured image. Is identified. In other words, the product recognizing unit 80 specifies the product that matches the captured image by the user by matching the posted image and the captured image only to the posted image of the page specified by the page recognition process, and the product information Is identified.

  The product recognition unit 80 refers to the expiration date of the specified product information and determines whether the product information is valid (whether the expiration date has expired). If the product information cannot be identified from the captured image, in other words, if the product that matches the captured image cannot be identified, or if the expiration date of the product information has passed, an error will be displayed indicating that an error has occurred in the product recognition process. The notification unit 88 is notified and an error dialog is displayed.

  Here, it supplements about image matching. The data of the plurality of product images held in the product information holding unit 64 may be feature data indicating the shape, pattern, and color of the feature portion of each product. The product recognition unit 80 generates feature data indicating the shape, pattern, and color of the feature portion in the captured image acquired by the captured image acquisition unit 74 and collates with the feature data of each product stored in the product information storage unit 64. By doing so, product information that matches the captured image may be specified. In addition, the product information holding unit 64 may hold the template data of each product image published in the paper catalog as product image data, and the product recognition unit 80 specifies a product that matches the captured image by template matching. Also good. Furthermore, image matching may be performed by a known machine learning method / SVM (Support Vector Machine).

  When the error notification unit 88 is notified from the page recognition unit 78 that an error has occurred in the page recognition process, the error notification unit 88 displays an error dialog indicating the fact on the touch screen 50. This error dialog will be described later with reference to FIG. In addition, when the product notification unit 80 notifies the product recognition unit 80 that an error has occurred in the product recognition process, the error notification unit 88 displays an error dialog indicating the fact on the touch screen 50. This error dialog will be described later with reference to FIG. In the error dialog related to the product recognition process, a button for displaying a list of listed products on the page specified in the page recognition process is displayed.

  The order processing unit 82 executes order processing for the product specified by the product recognition unit 80. The order processing unit 82 includes an order screen setting unit 84 and an order information transmission unit 86. The order screen setting unit 84 causes the touch screen 50 to display a product selection screen on which unit product information included in the product information specified by the product recognition unit 80 is arranged. This product selection screen will be described later with reference to FIG. Further, when a list display of listed products on the page specified in the page recognition process is requested, the order processing unit 82 specifies one or more image IDs indicated by the page information specified in the page recognition process. Then, a product selection screen in which unit product information included in the product information identified by each image ID is arranged is displayed on the touch screen 50. This product selection screen will be described later with reference to FIG.

  The order information transmission unit 86 transmits the order information of the product selected by the user on the product selection screen to the sales support device 12. This order information includes unit product information (product ID, number, price, etc.) to be ordered and user personal information stored in the user information holding unit 60. Although not shown in FIG. 5, the product purchase support unit 66 receives the completion information transmitted from the sales support device 12 and displays it on the touch screen 50 to indicate that the order of the product has been completed. You may further provide the user notification part which notifies to.

The operation of the user terminal 14 having the above configuration will be described below.
FIG. 6 is a flowchart showing the operation of the user terminal 14. Although not shown in the figure, the user information registration unit 68 records the personal information input by the user in the user information holding unit 60. When it is detected that the predetermined periodical information acquisition timing is reached (Y in S10), the page information acquisition unit 70 acquires the page information of the paper catalog from the sales support device 12 and records it in the page information holding unit 62. (S12). The merchandise information acquisition unit 72 acquires the merchandise information published in the paper catalog from the sales support device 12 and records it in the merchandise information holding unit 64 (S14). If it is not the periodical information acquisition timing (N of S10), S12 and S14 are skipped.

  When the product purchase tab is selected on the menu screen of the product purchase application (Y in S16), the page recognition unit 78 executes page recognition processing (S18). When a valid page is specified in the page recognition process (Y in S20), a product selection process including a product recognition process by the product recognition unit 80 is executed (S22). Then, the order processing unit 82 executes order processing (S24). If a valid page is not specified in the page recognition process (N in S20), S22 and S24 are skipped, and if the product purchase tab is not selected in the menu screen of the product purchase application (N in S16), S18 and subsequent steps are performed. skip.

  FIG. 7 is a flowchart showing in detail the page recognition process in S18 of FIG. The page recognition unit 78 displays a page code shooting screen on the touch screen 50 (S30). FIG. 8 shows a page code photographing screen. The page code shooting screen is an area that allows the user to shoot the page code image of the page of the paper catalog that contains the product image that the user wants to purchase. Is provided. Returning to FIG. 7, if a predetermined end operation is not accepted on the page code shooting screen (N in S32), the camera unit 58 waits for a page code to be captured (N in S34). When the captured image acquisition unit 74 reads the page code captured by the camera unit 58 (Y in S34), the page information acquisition unit 70 inquires of the sales support apparatus 12 whether or not the latest page information has been acquired.

  If the latest page information has not been acquired, that is, if there is unacquired page information or the acquired page information has been updated (N in S36), the page information acquisition unit 70 acquires the latest page information. Obtained from the sales support apparatus 12 (S38). If the latest page information has been acquired (Y in S36), S38 is skipped. The page recognition unit 78 specifies the page ID based on the page code imaged by the camera unit 58, and specifies the page information using the page ID as a key. A page is specified based on a captured image by the user (Y of S40), and if the expiration date of the page information has not expired (Y of S42), the flow of FIG.

  On the other hand, if the page ID cannot be identified from the captured page code (N in S40) or the identified page information has expired (N in S42), the error notification unit 88 displays an error dialog. It is displayed (S44). FIG. 9 shows an error dialog 92 in the page recognition process. As described above, by notifying the user of the expiration date of the paper catalog or the page by the error dialog, the user can be prompted to refer to the valid paper catalog. In addition, it is possible to prevent the user from ordering a product based on information that is not currently valid, and trouble can be easily avoided. When the user presses the close button of the error dialog 92, the page recognition unit 78 displays the page code shooting screen again (S30). Further, when a predetermined end operation is received on the page code photographing screen (Y in S32), the flow of FIG.

  FIG. 10 is a flowchart showing in detail the product selection process in S22 of FIG. The page recognition unit 78 displays the product photography screen on the touch screen 50 (S50). FIG. 11 shows a product photography screen. The product shooting screen includes a page explanation area 94 for displaying an explanation document for the page specified by the page recognition process, and an area for allowing a user to take a product image desired to be purchased, and stores the product image as a subject. A product photographing area 96 indicating the power range and an imaging button 98 are provided. The user causes the user terminal 14 to capture the product image by pressing the image pickup button 98 in a state where the product image desired to be purchased is displayed in the product imaging region 96 among the product images printed on the paper catalog. .

  Returning to FIG. 10, the product information acquisition unit 72 inquires of the sales support device 12 whether or not each of the one or more product information associated with the page specified by the page recognition process is the latest product information. . If there is unacquired product information, the product information is acquired from the sales support apparatus 12. This inquiry and product information acquisition is executed as a background process of the display process of the product photography screen, in other words, it starts while the user tries to photograph a desired product. In this way, by acquiring product information in the background of product photography by the user, it is difficult to make the user aware of the waiting time for product information acquisition while ensuring execution of image matching based on the latest product information. Therefore, it is possible to suppress a decrease in user convenience.

  When the latest product information has not been acquired, that is, there is unacquired product information, or when the acquired product information has been updated (N in S52), the product information acquisition unit 72 obtains the latest product information. Obtained from the sales support apparatus 12 (S54). If the latest product information has been acquired (Y in S52), S54 is skipped. Thereby, the communication amount between the user terminal 14 and the sales support apparatus 12 can be minimized, and the waiting time for obtaining product information can be reduced. Further, the product image matching the captured image by the user can always be the latest, and the order process can be executed based on the latest product information. In other words, it is possible to avoid performing image matching and order processing based on product information that is not up-to-date. As a modification, the confirmation process of whether or not the latest product information has been acquired is omitted, and the product information acquisition unit 72 always includes each product included in the page when the page is identified by the page recognition process. The latest product information may be obtained by inquiring of the sales support device 12.

  When a predetermined end operation is detected on the product photographing screen (Y in S56), the flow of FIG. If the predetermined end operation has not been detected (N in S56), the camera unit 58 waits for a product image to be captured (N in S58). When the captured image acquisition unit 74 reads a product image captured by the camera unit 58 (Y in S58), if the acquisition processing of the latest product information is incomplete, the process waits for completion (N in S60). . If acquisition processing of the latest product information has been completed (Y in S60), the product recognition unit 80 captures the product image included in each product information associated with the page specified by the page recognition processing, and the user's imaging The image is collated (S62). Thus, by limiting the comparison target with the captured image of the user to the product image posted on the page specified by the page recognition process, the efficiency of the image matching process can be realized, and the time required for the matching process can be reduced. Can be reduced.

  When a product image that matches the user's captured image is specified (Y in S64), the order screen setting unit 84 displays a product selection screen in which information on unit products associated with the product image is arranged (S66). ). FIG. 12 shows a product selection screen. On the product selection screen, attribute information of each of one or more unit products associated with one product image in the product information is displayed. In FIG. 12, information on three related products (here, knits of different colors) associated with one product image is presented to the user. In this way, by associating a plurality of unit products with one product image in the product information and displaying each unit product in a selectable manner on the product selection screen, in the paper catalog, the products to be sold (that is, unit products) are displayed. It suffices if an image of a part is posted, and it becomes easy to ensure the flexibility of the configuration of the paper catalog.

  In the product selection screen, a thumbnail image 106 and a selection button 108 are displayed in the information area of each unit product. The user presses the order button 110 after pressing the selection button 108 for the unit product to be ordered, thereby executing the order processing for the unit product. For example, as the order process in S24 of FIG. 6, the order information transmission unit 86 transmits the ID of the unit product selected by the user and the user personal information stored in the user information holding unit 60 to the sales support device 12.

  Returning to FIG. 10, if the product image that matches the user's captured image cannot be specified (N in S64) or the specified product information has expired, the error notification unit 88 displays an error dialog (S68). . FIG. 13 shows an error dialog 100 in the product recognition process. When the user presses the re-shooting button in the error dialog 100 (N in S70), the process returns to S50.

  On the other hand, when the user presses the product list button 102 in the error dialog 100 (Y in S70), the order screen setting unit 84 selects the product to be ordered from the list of products on the page specified by the page recognition process. A product selection screen for allowing the user to select is displayed (S66). Specifically, the order screen setting unit 84 specifies one or more product information associated with the page information specified by the page recognition process, and a product in which attribute information of unit products included in each product information is arranged. Display the selection screen.

  FIG. 14 shows a product selection screen displaying a list of products posted on the page. In FIG. 12, three unit products corresponding to one product image are displayed. However, in FIG. 14, a total of six product images corresponding to all product images posted on one page of the paper catalog including the three unit products are displayed. Unit products are displayed so that they can be selected. As a result, even when a product that matches the captured image by the user cannot be detected in the product recognition process, it is possible to support the user's product selection by displaying a list of products listed on the page specified in the previous page recognition process. it can.

  The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there. A modification is shown below.

A first modification will be described.
The order screen setting unit 84 may display both the product information specified by the product recognition unit 80 and all the product information included in the page specified by the page recognition unit 78 side by side on the same product selection screen. Good. For example, the product information specified by the product recognition unit 80 may be displayed at the top of the product selection screen. The product information excluding the product information specified by the product recognition unit 80 among all the product information included in the page specified by the page recognition unit 78 is displayed below the product selection screen across a predetermined separator. You may let them. According to this modification, even when a plurality of product images are printed in the vicinity of the paper catalog and the user takes a wrong product, or when the wrong product information is specified, the product information desired by the user is displayed. It can be displayed on the product selection screen, and a reduction in user convenience can be suppressed.

A second modification will be described.
When a plurality of product images are printed in the vicinity of the paper catalog, the user captures a plurality of product images in the product shooting area 96 on the product shooting screen, and the captured image acquisition unit 74 acquires one captured image. May include a plurality of product images. In this case, the product recognition unit 80 may specify a plurality of pieces of product information that match each of a plurality of product images included in one captured image. For example, when the product recognition unit 80 performs image matching based on the feature portion of the product and detects feature portions of a plurality of products from one captured image, the product recognition unit 80 executes image matching based on each feature portion, A plurality of product information may be specified. Then, the order screen setting unit 84 may display a product selection screen in which a plurality of specified product information is arranged. According to this modification, even when a plurality of product images are included in the captured image by the user, information on each product can be displayed on the product selection screen, and the user can select a desired product on the product selection screen. A reduction in user convenience can be suppressed.

A third modification related to the second modification will be described.
As described above, in the case where one captured image acquired by the captured image acquisition unit 74 includes a plurality of product images, the product recognition unit 80 includes a plurality of product images included in one captured image. A plurality of pieces of product information that match each of them may be specified. Then, product information matching the product image closest to the center position of the captured image may be passed to the order screen setting unit 84 and displayed on the product selection screen. As another aspect, when the product recognition unit 80 performs image matching based on the feature portion of the product and detects feature portions of a plurality of products from one captured image, the product recognition unit 80 is closest to the center position of the captured image. The feature portion may be used preferentially for image matching, and product information that matches the feature portion may be specified. Since the product that the user wants to purchase is likely to be close to the center of the captured image, even if the captured image by the user includes multiple product images, the product selection screen is used as a target for order processing. It becomes easy to display.

A fourth modification example related to the second modification example will be described.
As described above, in the case where one captured image acquired by the captured image acquisition unit 74 includes a plurality of product images, the product recognition unit 80 includes a plurality of product images included in one captured image. A plurality of pieces of product information that match each of them may be specified. In this case, the product recognizing unit 80 inputs a selection operation (for example, a tap operation or a touch operation, etc., hereinafter referred to as a tap operation) for a desired product image on a product shooting screen displaying a captured image by the user. A message instructing to do so may be displayed. When the user's tap operation is detected, the product recognition unit 80 may pass the product information that matches the product image closest to the tap position to the order screen setting unit 84 and display it on the product selection screen.

  As another aspect, when the product recognition unit 80 performs image matching based on a feature portion of a product and detects feature portions of a plurality of products from one captured image, the product recognition image displayed by the user is displayed. On the shooting screen, a message instructing to tap a desired product image may be displayed. When a user tap operation is detected, the product recognition unit 80 preferentially uses the feature portion closest to the tap position for image matching, and identifies product information including a product image that matches the feature portion. Also good. According to this modification, even when a plurality of product images are included in the captured image by the user, the product desired by the user can be displayed on the product selection screen as an order processing target based on the user's selection operation.

A fifth modification example related to the second modification example will be described.
As described above, in the case where one captured image acquired by the captured image acquisition unit 74 includes a plurality of product images, the product recognition unit 80 includes a plurality of product images included in one captured image. A plurality of pieces of product information that match each of them may be specified. In this case, the product recognition unit 80 may display a message instructing to perform an enlargement display operation (for example, a pinch-out operation) on a desired product image on the product shooting screen on which a captured image by the user is displayed. . When the user's enlarged display operation is detected, the product recognition unit 80 may execute the image matching process again based on the captured image after the enlarged display operation.

  As another aspect, the product recognition unit 80 performs an enlargement display operation on a desired product image when detecting feature portions of a plurality of products when performing image matching based on the feature portion of the product. A message instructing the user may be displayed. When the user's enlarged display operation is detected, the product recognition unit 80 may execute the image matching process again based on the captured image after the enlarged display operation. According to this modification, even when a plurality of product images are included in the captured image by the user, the product desired by the user can be displayed on the product selection screen as an order processing target based on the user's enlargement display operation. .

A sixth modification will be described.
In the above embodiment, a paper medium is described as a medium for displaying a product image, but the product display medium is not limited to a paper medium. As a modification, a monitor medium may be used. Specifically, even if the product display medium is a monitor (display) medium that displays television shopping or a web log, the technical idea described in the embodiment can be applied.

  For example, in the case of TV shopping, on a product display screen (for example, a product introduction screen for displaying product images in a program video), identification information of the TV shopping program (broadcast time, etc.) or product display screen identification information in the program May be displayed (corresponding to the page code in the embodiment). A viewer of TV shopping images a desired product after imaging a screen code on a product display screen. The user terminal 14 acquires the product image / product attribute displayed on the product display screen for TV shopping from the sales support device 12 and holds it. The user terminal 14 identifies the program name and the product display screen based on the screen code, matches the product image displayed on the product display screen with the captured image, and displays the product selection screen similar to that in the paper catalog. Let

  In the case of a web log, on a product display screen (for example, a pane displaying a product image on a web page), a screen code indicating the identification information (URL or the like) of the web log or the identification information of the pane may be displayed. The viewer of the web log images the desired product after capturing the screen code on the product display screen. The user terminal 14 acquires the product image / product attribute displayed on the product display screen of the weblog from the sales support device 12 and holds it. The user terminal 14 identifies the web log and the product display screen based on the screen code, matches the product image displayed on the product display screen with the captured image, and displays the product selection screen similar to the paper catalog. Let

A seventh modification will be described.
In the above embodiment, a bar code or a QR code (registered trademark) is exemplified as the page code. In the modified example, as a page code, an image showing the appearance of a part or the whole of a cover or each page of a paper medium or a product display screen displayed on a monitor medium (hereinafter also referred to as “medium appearance image”) is adopted. May be. That is, as the page recognition process, the medium may be uniquely identified by image recognition of the appearance of the product display medium. According to this modification, it is no longer necessary to display code information on each page or screen of the product display medium, the deterioration of the aesthetics of the product display medium can be further suppressed, and the production cost of the product display medium can be easily reduced. .

  For example, the user terminal 14 may acquire and hold page information including data of the medium appearance image from the sales support apparatus 12. Here, it is assumed that the user captures the entire cover of the paper catalog. The page recognition unit 78 may collate (image matching) the captured image of the cover with the previously acquired medium appearance image, and specify the medium in units of paper catalogs. The merchandise information acquisition unit 72 may acquire the latest merchandise information of all the merchandise posted in the recognized paper catalog. Thereafter, the user only needs to shoot a desired product in the paper catalog and execute the matching process between the product image and the captured image as in the embodiment. When the user captures the appearance of the entire specific page of the paper catalog, the page recognition unit 78 may specify the medium in units of pages by comparing the captured image with the medium appearance image. Henceforth, the process similar to embodiment should just be performed.

  It should also be understood by those skilled in the art that the functions to be fulfilled by the constituent elements recited in the claims are realized by the individual constituent elements shown in the embodiments and the modified examples or by their linkage. For example, the predetermined medium described in the claims includes the paper catalog described in the embodiment and the monitor medium described in the sixth modification. The display surface described in the claims includes a product listing page in a paper catalog and a product display screen on a monitor medium.

  DESCRIPTION OF SYMBOLS 10 Sales management apparatus, 12 Sales support apparatus, 14 User terminal, 16 Merchandise sales system, 58 Camera part, 62 Page information holding part, 64 Product information holding part, 70 Page information acquisition part, 72 Product information acquisition part, 74 Captured image An acquisition unit, 76 verification unit, 82 order processing unit, 88 error notification unit.

Claims (8)

In a portable information terminal equipped with an imaging device,
A captured image acquisition function for acquiring an image of a product captured by the imaging device from a plurality of products displayed on a predetermined medium;
A collation function for collating images of the plurality of products stored in the portable information terminal and the captured images acquired by the captured image acquisition function;
An order function for executing order processing of the product based on the attribute information of the product that matches the captured image among the attribute information of the plurality of products stored in the portable information terminal;
Computer program for realizing. The portable information terminal stores an image of each product in association with a display surface on which each product is displayed on the medium,
The captured image acquisition function acquires a captured image of a product displayed on the display surface after acquiring the identification image of the display surface in the medium captured by the imaging device,
The computer program according to claim 1, wherein the collation function collates a product image stored in association with a display surface specified by the identification image and a captured image of the product.   The order function includes a plurality of product images associated with the display surface, and when there is no image that matches the captured image of the product, an order target is selected from the plurality of products associated with the display surface. The computer program according to claim 2, wherein a screen for allowing the user to select a product is displayed.   Claims for further realizing a product information acquisition function for acquiring a product image associated with the display surface from an external device as background processing when the imaging device images the product displayed on the display surface. The computer program according to 2 or 3.   The product information acquisition function checks with the external device whether or not the latest product image or product attribute associated with the display surface has been acquired when an identification image of the display surface is captured, and has not been acquired. The computer program according to claim 4, wherein the latest product image or product attribute is acquired. The portable information terminal further holds information indicating an expiration date of the medium,
The identification image of the display surface includes identification information of the medium,
The identification function according to any one of claims 2 to 5, for further realizing a notification function for notifying a user when an identification image of a display surface of the medium is captured and the expiration date of the medium has expired. Computer program. The portable information terminal holds one product image and a plurality of related products in which at least a part of attribute information is different,
The order function displays a screen for allowing a user to select a product to be ordered from among related products for a plurality of related products associated with a product image that matches the captured image. The computer program according to any one of claims 1 to 6. For a plurality of products displayed on a predetermined medium, each product image and attribute information holding unit,
An imaging unit for imaging the product displayed on the medium;
A collation unit that collates a plurality of product images held in the holding unit and a captured image by the imaging unit;
An order processing unit that executes order processing of the product based on the attribute information of the product that matches the captured image;
A portable information terminal comprising: