JP2019164803A - Display control device, control method, program, and storage medium - Google Patents

Abstract

To favourably display an arrangement place of a merchandise designated by a user such as a client and detailed information on the merchandise.SOLUTION: A display control device controls a display of a display unit viewed by a seller handling merchandise, and includes a merchandise information acquisition unit, a merchandise position acquisition unit and a display control unit. The merchandise position acquisition unit acquires positional information relating to a position of the merchandise selected by a user. The display control unit makes the information about the merchandise acquired by the merchandise information acquisition unit display in the display unit apparently in association with the merchandise on the basis of the positional information acquired by the merchandise position acquisition unit.SELECTED DRAWING: Figure 8

Description

  The present invention relates to a technique for displaying information.

  2. Description of the Related Art Conventionally, in Internet shopping and the like, there is a technique for displaying an image taken inside a store on a customer terminal used by the customer. For example, in Patent Document 1, in electronic commerce, a virtual store system that allows a customer to select a product that the customer wants to purchase by simultaneously or selectively displaying images taken from a plurality of cameras installed in the store on a customer terminal. Is disclosed. Patent Document 2 discloses a communication system in which a store clerk explains a product designated by a customer in a state where a video of the store clerk is displayed by text or voice chat in electronic commerce.

JP 2009-223760 A JP 2005-071304 A

  In the electronic commerce system, when a customer designates an arbitrary product on a product list screen or the like, it is convenient that an explanation can be immediately received from a store clerk. In this case, the store clerk needs to immediately and accurately recognize and explain the product placement location and product information designated by the customer. Patent Documents 1 and 2 do not disclose anything about this point.

  The present invention has been made in order to solve the above-described problems, and is capable of suitably displaying the location of the product designated by a user such as a customer and detailed information on the product. The main purpose is to provide a device.

  The invention described in claim is a display control device, wherein information related to a product selected by a user using a user terminal is displayed on the display unit based on position information related to the selected product. A control unit that is displayed in a display unit that is visually associated with a product other than the user and a photographing unit that captures an image, and the control unit includes a person in the image. If it is reflected, the image is processed into an image in which the person cannot be specified, and the processed image is transmitted to the user terminal by a communication unit.

  Further, the invention described in claim is a control method executed by a display control device including a photographing unit for photographing an image, wherein the information related to a product selected by a user who uses a user terminal is selected. A first step of causing the display unit to display on the display unit visually associated with the selected product based on the position information related to the product, and a person in the image A second step of processing the image into an image in which the person cannot be specified when the image is reflected, and transmitting the processed image to the user terminal by the communication unit.

The invention described in the claims is a program executed by a computer of a display control apparatus including an imaging unit that captures an image, wherein information regarding a product selected by a user using a user terminal is selected by the selection. Based on the positional information related to the product, the display is caused to function as a control unit that causes the display unit to display the display unit that is visually associated with the selected product and visible to a person other than the user, When a person is reflected in the image, the control unit processes the image so that the person cannot be specified, and causes the communication unit to transmit the processed image to the user terminal.

It is a schematic block diagram of a display system. It is the figure which expanded the shelf by which goods are arranged. It is a schematic block diagram of a salesclerk terminal. It is a schematic block diagram of a customer terminal. It is a schematic block diagram of a server apparatus. The data structure of customer information DB is shown. It is a flowchart which shows the outline | summary of a process of a display system. It is a flowchart which shows the procedure of a goods description request | requirement process. It is a flowchart which shows the procedure of description goods emphasis processing. It is a flowchart which shows the procedure of a purchase related process. It is an example of a display of the half mirror of the salesclerk terminal at the time of selecting the customer for customer service. It is an example of a screen display of a customer terminal immediately after receiving a customer service inquiry signal. It is an example of a screen display of the customer terminal after shifting to the customer service mode. The state which performs operation which designates description request | requirement goods on the screen of a customer terminal is shown. It is a display example of a half mirror of a salesclerk terminal when an explanation request product is specified. It is the example of a screen display of a customer terminal when a salesclerk terminal is removed from a salesclerk. It is a display example of the half mirror of the salesclerk terminal which concerns on a modification.

  According to a preferred embodiment of the present invention, the display control device is a display control device that controls display of a display unit visually recognized by a seller who handles products, and acquires information related to the product selected by the user. Information on the product acquired by the product location information acquisition unit, information on the product location acquisition unit that acquires location information on the selected product, and information on the product acquired by the product information acquisition unit. A display control unit that is displayed on the display unit in association with the selected product.

  The display control device controls display of a display unit visually recognized by a seller who handles products, and includes a product information acquisition unit, a product position acquisition unit, and a display control unit. The “seller” is a person who handles merchandise for the purpose of sales, and corresponds to, for example, a salesperson or a clerk who can explain the merchandise. The product position acquisition unit acquires position information related to the product selected by the user. Here, the “user” refers to a person who can communicate with the seller via a network or the like and requests a description of the product or location information of the product at the store. Specifically, the user may be a customer who purchases a product, or may be a store clerk that exists in a remote place away from the seller. Further, the “position information related to the product” may be position information of an object close to the product such as a product shelf, in addition to the position information of the product. The display control unit causes the display unit to display the information related to the product acquired by the product information acquisition unit in association with the apparent product based on the position information acquired by the product position information acquisition unit. According to this aspect, the display control device can make the seller appropriately recognize the location of the product and the product information.

  In one aspect of the display control device, the display control device further includes a user information acquisition unit that acquires information about the user, and the display control unit relates to the user acquired by the user information acquisition unit. Information is displayed on the display unit. According to this aspect, the display control device can cause the seller to recognize the user information and appropriately execute the customer service.

  In another aspect of the display control device, the display control unit causes the display unit to display a frame that apparently surrounds the selected product, and associates information about the product with the frame. Display on the display. According to this aspect, the display control device can make the seller appropriately recognize the location of the product and the product information.

  In another aspect of the above display control device, the display control device receives at least one of an image generated by a photographing unit attached to the seller or an audio data generated by an audio input unit attached to the seller. It further has a communication control part which transmits by a communication part to the user terminal which the user uses. According to this aspect, the display control apparatus can suitably execute the customer service via the user terminal using video or audio.

  In another aspect of the display control apparatus, the display unit can be mounted, and the communication control unit uses at least one of the image or the audio data only when the display unit is in a mounted state. To the user terminal. According to this aspect, it is possible to suitably prevent the video and audio of a customer who is visiting the store from flowing to the user terminal when the customer service is started.

  In another aspect of the display control device, the communication unit receives information indicating whether or not the display screen of the image displayed by the user terminal is displayed in the foreground from the user terminal, The display control unit displays predetermined information on the display unit when the display screen of the image is not displayed in the foreground. According to this aspect, the display control device can suitably suppress the seller from receiving an unnecessary response when the user views or operates a screen other than the display screen for customer service.

  In another aspect of the display control device, a product recognizing unit that recognizes a product that is held by the seller, and a communication unit that transmits identification information of the product recognized by the product recognizing unit to a user terminal used by the user. And a communication control unit that transmits the data. According to this aspect, the display control apparatus can suitably notify the user of the product being explained by the seller.

  In another aspect of the display control device, when the communication unit receives a description request for a product that is different from the product recognized by the product recognition unit from the user terminal, the display control unit apparently includes the description. Information on the product is displayed on the display unit in association with the requested product. In this way, the seller can suitably execute the description of the product for which the description is requested.

  In another aspect of the display control device, the product position acquisition unit acquires position information related to related products related to the product selected by the user, and the display control unit acquires information related to the related products. Based on the position information acquired by the product position information acquisition unit, it is apparently associated with the related product and displayed on the display unit. According to this aspect, the display control device can make the seller recognize the existence of a product related to the product selected by the user, and promote the sale to the user.

  In another aspect of the display control device, the product position acquisition unit acquires position information indicating a position of a tag reader that has read information on a tag attached to the product as position information about the product. According to this aspect, the display control apparatus can appropriately recognize the position of the product and display information related to the product in association with the product.

  According to another preferred embodiment of the present invention, the display control device is a control method executed by the display control device that controls the display of the display unit visually recognized by the seller handling the product, and is selected by the user. A product information acquisition step for acquiring information about the selected product, a product position acquisition step for acquiring position information about the selected product, and information about the product acquired in the product information acquisition step. A display control step of causing the display unit to display the information in association with the selected product on the basis of the position information acquired in the step. By executing this control method, the display control device can make the seller appropriately recognize the location of the product and the product information.

  According to still another embodiment of the present invention, a program that is executed by a computer that controls display of a display unit visually recognized by a seller who handles products, and that acquires information about the product selected by the user An information acquisition unit, a product position acquisition unit that acquires position information about the selected product, and information about the product acquired by the product information acquisition unit based on the position information acquired by the product position information acquisition unit Apparently, the computer is caused to function as a display control unit that is displayed on the display unit in association with the selected product. By executing this program, the computer can make the seller appropriately recognize the location of the product and the product information. Preferably, the program is stored in a storage medium.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. Hereinafter, the head-mounted display is appropriately expressed as “HMD”.

[Display system configuration]
FIG. 1 is a schematic configuration diagram of a display system according to the present embodiment. As shown in FIG. 1, the display system is a system related to electronic commerce (EC), and is mainly a clerk terminal 1 worn by a clerk 9 in a store 5 in which commodities handled in electronic commerce are arranged. And a customer terminal 2 used by a customer of electronic commerce and a server device 3 functioning as a Web server or the like. Then, the display system transmits an image or the like taken by the clerk terminal 1 to the customer terminal 2, so that the clerk 9 responds to the customer with a product explanation on the screen of the customer terminal 2. In the present embodiment, as an example, it is assumed that the store 5 is a store that handles clothes.

  The clerk terminal 1 is a see-through type HMD configured in a spectacle shape and is configured to be attachable to the head of the clerk 9. Further, the clerk terminal 1 communicates with the server device 3 and displays an image (also referred to as “AR image”) superimposed on the landscape based on the information received from the server device 3. In the present embodiment, the clerk terminal 1 is information of a customer (also referred to as “visitor information”) browsing a screen for electronic commerce (also referred to simply as “EC screen”) on which the server device 3 provides display information. Is received from the server device 3, and the store clerk 9 is made to select a customer to be served. Thereafter, the clerk terminal 1 establishes communication with the customer terminal 2 at the start of customer service, and transmits a photographed image to the customer terminal 2 or a product for which the customer requests explanation (also referred to as “explanation request product”). And receiving the information on the product and displaying the product location and product information. Hereinafter, an operation mode in which the store clerk terminal 1 and the customer terminal 2 establish communication to exchange customer-facing video and audio data is also referred to as “customer service mode”.

  The customer terminal 2 is a portable terminal such as a personal computer or a smartphone used by a customer at home, and displays an EC screen based on display information provided by the server device 3. Further, the customer terminal 2 receives a signal (also referred to as “customer service inquiry signal”) for inquiring whether or not a response is required from the store clerk terminal 1 via the server device 3 during display of the EC screen, and responds based on the user input. A signal is transmitted to the clerk terminal 1. After the transition to the customer service mode, the customer terminal 2 receives and outputs video and audio data from the clerk terminal 1 or transmits the generated audio data to the clerk terminal 1. The customer terminal 2 is an example of the “user terminal” in the present invention.

  The server device 3 communicates with the clerk terminal 1 and the customer terminal 2 via the network. The server device 3 transmits EC screen display information to the clerk terminal 1 based on a request signal related to the Web page of the customer terminal 2. In addition, the server device 3 transmits visitor information and information related to products to the customer terminal 2 and performs processing for transferring a customer service inquiry signal.

  In the store 5, products with tags 7 such as RFID (Radio Frequency Identifier) are placed on shelves, and the tag reader 6 that performs wireless communication with the tags 7 on the shelves on which the products are placed. Is arranged. FIG. 2 is an enlarged view of a shelf on which products are arranged. As shown in FIG. 2, each tag reader 6 has a communication range indicated by a broken line, and a signal indicating identification information (that is, a product number) of the corresponding product is transmitted from the tag 7 existing in the communication range by wireless communication. Receive. In the example of FIG. 2, three tag readers 6 are arranged at approximately equal intervals on each of the upper shelf, the middle shelf, and the lower shelf, and are close enough that their communication ranges do not overlap each other. Then, the tag reader 6 transmits a signal (also referred to as “tag detection signal”) indicating the product number received from the tag 7 and the identification information of the tag reader 6 to the server device 3. By doing in this way, as will be described later, the server device 3 preferably specifies the location where the product after movement exists based on the tag detection signal received from the tag reader 6 even when the product moves. can do. In addition, the place where goods are arrange | positioned may be arbitrary places, such as not only in a goods shelf but in a showcase. In this case, a tag reader 6 may be provided for each installation location of the product, and the server device 3 may store information on the installation location of the product in advance.

[Structure of clerk terminal]
FIG. 3 shows a schematic configuration of the clerk terminal 1. As shown in FIG. 2, the clerk terminal 1 includes a light source unit 10, a half mirror 11, a communication unit 12, an input unit 13, a storage unit 14, a camera 15, a tag reader 16, a control unit 17, A speaker 18 and a measurement unit 19 are included.

  The light source unit 10 includes a laser light source, an LCD light source, and the like, and emits light from such a light source. The half mirror 11 reflects the light from the light source unit 10 toward the user's eyeball. Thereby, the virtual image corresponding to the image formed by the salesclerk terminal 1 is visually recognized by the user. The half mirror 11 has substantially the same transmittance and reflectance. However, instead of using such a half mirror 11, a mirror (so-called beam splitter) in which the transmittance and the reflectance are not equal may be used. . The salesclerk terminal 1 or the half mirror 11 is an example of the “display unit” in the present invention.

  The communication unit 12 performs data communication with the customer terminal 2 and the server device 3 based on the control of the control unit 17. The input unit 13 generates an input signal based on a user operation and transmits it to the control unit 17. The input unit 13 is a remote controller having, for example, a button for operation by a user, a cross key, and a button. In addition, the input unit 13 includes a microphone 130 and transmits audio data generated by the microphone 130 to the control unit 17. The microphone 130 generates sound data from the sound input in the customer service mode. The storage unit 14 stores a program executed by the control unit 17. Under the control of the control unit 17, the camera 15 generates an image (also referred to as “captured image Im”) obtained by capturing the front of the clerk terminal 1, and supplies the generated captured image Im to the control unit 17.

  The tag reader 16 reads the product number from the tag 7 attached to the product, and supplies the read product number information to the control unit 17. The tag reader 16 reads the product number from the tag 7 when the store clerk 9 picks up and explains the product and the tag 7 attached to the product enters the communicable range.

  The speaker 18 outputs audio data received from the customer terminal 2 based on the control of the control unit 17. The measuring unit 19 is a sensor or the like that detects the state of the clerk terminal 1, and includes a GPS receiver that generates position information indicating the current position, a geomagnetic sensor that detects the direction, an acceleration sensor that detects the inclination of the clerk terminal 1, and the like. Have.

  The control unit 17 includes a processor such as a CPU (not shown) and a memory such as a RAM and a ROM, and performs overall control of the clerk terminal 1. For example, in the customer service mode, the control unit 17 transmits the captured image Im generated by the camera 15 and the audio data generated by the microphone 130 to the customer terminal 2 or reproduces the audio data received from the customer terminal 2. Further, the control unit 17 displays an AR image representing customer information and product information of the customer who is serving the customer based on information transmitted from the server device 3, or the current position measured by the measurement unit 19 and the clerk terminal 1. By detecting the orientation, an AR image indicating the placement location of the specified explanation request product is displayed. When the half mirror 11 is regarded as a “display unit” in the present invention, the control unit 17 is an example of a “display control device” in the present invention and a computer that executes a program.

[Configuration of customer terminal]
FIG. 4 is a schematic configuration of the customer terminal 2. The customer terminal 2 includes a communication unit 22, an input unit 23, a storage unit 24, a control unit 27, and a display unit 29 such as a display.

  The communication unit 22 performs data communication with the clerk terminal 1 and the server device 3 based on the control of the control unit 27. The input unit 23 includes a microphone 230. The microphone 230 generates sound data from the sound input during the customer service mode. The input unit 23 includes input devices such as a mouse, a keyboard, and a touch panel in addition to the microphone 230. The storage unit 24 stores a program executed by the control unit 27.

  The control unit 27 includes a processor such as a CPU (not shown) and a memory such as a RAM and a ROM, and performs overall control of the customer terminal 2. For example, the control unit 27 causes the display unit 29 to display an EC screen based on the display information received from the server device 3. In the customer service mode, the control unit 27 outputs the captured image Im and audio data received from the store clerk terminal 1, and transmits the audio data input by the microphone 230 to the store clerk terminal 1.

[Configuration of server device]
FIG. 5 is a schematic configuration of the server device 3. The server device 3 includes a communication unit 32, a storage unit 34, and a control unit 37. The communication unit 32 performs data communication with the clerk terminal 1 and the customer terminal 2 based on the control of the control unit 37.

  The storage unit 34 stores a program executed by the control unit 37. The storage unit 34 includes a customer information DB 341, a product information DB 342, tag reader arrangement information 343, and explained list information 344.

  The customer information DB 341 stores information such as a communication address of the customer terminal 2 and an access history to the EC screen for each customer. The customer information DB 341 will be described later with reference to FIG. The product information DB 342 stores information on each product displayed on the EC screen. The product information registered in the product information DB 342 includes product images, product sizes, colors, product names, serial numbers (SN), and the like. The tag reader arrangement information 343 is information in which the identification information of each tag reader 6 installed in the store 5 is associated with the arrangement location. The already-explained list information 344 is information in which products that the clerk 9 explained to the customer at the time of customer service are listed for each customer.

  The control unit 37 includes a processor such as a CPU (not shown) and a memory such as a RAM and a ROM, and performs overall control of the server device 3. For example, the control unit 37 monitors the time during which the customer displays the EC screen on the customer terminal 2 (also simply referred to as “visit time”), and information on customers whose visit time exceeds a predetermined threshold is displayed as a visitor. The information is transmitted to the clerk terminal 1 as information. In this case, the visit time may be a display time for each Web page (site), or may be a total display time for all sites managed by the server device 3. In addition, when receiving a customer service inquiry signal designating a specific customer from the clerk terminal 1, the control unit 37 transfers the customer service inquiry signal to the corresponding customer terminal 2. And the control part 37 transmits the information required for the salesclerk terminal 1 and the customer terminal 2 to communicate directly to the salesclerk terminal 1 and / or the customer terminal 2 at the time of start of customer service mode. When the clerk terminal 1 is regarded as a “display unit” in the present invention, the control unit 37 is an example of a “display control device” in the present invention and a computer that executes a program.

  Here, the customer information DB 341 will be supplementarily described. FIG. 6 shows the data structure of the customer information DB 341. The customer information DB 341 shown in FIG. 6 includes “Status” indicating the connection status of each customer terminal 2 with the server device 3, “IP Address” indicating the IP address used by the customer terminal 2, and “Date Time” indicating the access date and time. , “Visit Duration” indicating the visit time, “Request” indicating the address information of the accessed EC screen, “User-agent” indicating the program information such as the browser used by the customer terminal 2, and the country specified by the IP address “Country” indicating “Country”. In the example of FIG. 6, the control unit 37 uses the items “IP Address” and “User-agent” as customer identification information, and analyzes the access by the customer terminal 2 to the EC screen in real time, thereby the customer information DB 341. Update. For example, the control unit 37 updates “Status” based on the presence / absence of a connection state with the customer terminal 2. In addition, each time the customer terminal 2 accesses the EC screen, the control unit 37 adds each item such as “Date Time” and “Visit Duration” by measuring the connection date and time and the visit time with the customer terminal 2. Or update.

[Processing flow]
Next, a processing flow executed by the display system according to the present embodiment will be described with reference to FIGS.

(1) Processing Overview FIG. 7 is an example of a flowchart showing an overview of processing of the display system according to the present embodiment. The display system repeatedly executes the process of the flowchart shown in FIG. 7 every time the customer terminal 2 accesses the server device 3 in order to display the EC screen. Note that when the plurality of customer terminals 2 are accessing the server apparatus 3 in order to display the EC screen, the server apparatus 3 executes the process of FIG.

  First, the customer terminal 2 transmits a request signal for display information of an EC screen to be displayed on the Web browser to the server device 3 based on a user operation for specifying a URL or the like (step S101). And the server apparatus 3 transmits the display information of a corresponding EC screen to the customer terminal 2 based on the received request signal (step S102). Then, the customer terminal 2 displays an EC screen based on the received display information (step S103).

  Next, the server device 3 updates the customer information DB 341 related to the access history (step S104). Specifically, the server device 3 measures the date and time of visit and the visit time of the customer terminal 2, and associates it with the IP address and program information of the customer terminal 2 recognized by communication with the customer terminal 2 as the customer information DB 341. Remember. Then, the server device 3 determines whether or not the visit time during measurement exceeds a predetermined threshold (step S105). The above threshold value is a threshold value for determining whether or not the customer is confused about purchasing a product on the EC screen, for example, and is set to a preliminarily adapted value based on an experiment or the like, for example.

  When the server device 3 determines that the visit time exceeds the threshold (step S105; Yes), the server device 3 transmits the visitor information to the clerk terminal 1 (step S106). Here, for example, the server device 3 displays, as visitor information, the IP address of the customer terminal 2, country information extracted from the customer information DB 341, visit time information, etc., and the customer terminal 2 being displayed on the EC screen. The product information and the like are transmitted to the clerk terminal 1. On the other hand, if the visit time does not exceed the threshold (step S105; No), the server device 3 continuously monitors whether the visit time does not exceed the threshold.

  Next, the clerk terminal 1 displays an AR image indicating visitor information (step S107). A specific display example of visitor information will be described later with reference to FIG. Then, the clerk terminal 1 determines whether or not a user input indicating that the clerk 9 is serving is detected (step S108). Then, when the store clerk terminal 1 detects a user input for customer service (step S108; Yes), the customer service terminal 1 transmits a customer service inquiry signal designating the customer's customer service IP address or the like to the server device 3 (step S109). In this case, the server device 3 transfers the received customer service inquiry signal to the customer terminal 2 of the corresponding customer (step S110).

  The customer terminal 2 that has received the customer service inquiry signal displays that the store clerk 9 is willing to serve customers, and accepts an input for allowing the user to select whether or not to receive the customer service (step S111). Then, the customer terminal 2 transmits a response signal indicating whether or not to receive a customer service to the server device 3 (step S112), and the server device 3 transfers the response signal to the clerk terminal 1 (step S113). And the salesclerk terminal 1 determines whether the received response signal is a customer service request (step S114). When the received response signal is a customer service request (step S114; Yes), the clerk terminal 1 establishes communication with the customer terminal 2, and the captured image Im generated by the camera 15 and the audio data generated by the microphone 130 are established. Is transmitted to the customer terminal 2 (step S116), or voice data generated by the customer terminal 2 is received and reproduced by the speaker 18 (step S117).

  Similarly, when the customer terminal 2 makes a response to receive a customer service (step S115; Yes), the customer terminal 2 receives and reproduces the captured image Im and voice data generated by the clerk terminal 1 (step S116), or a microphone. The voice data generated by 230 is transmitted to the clerk terminal 1 (step S117).

  On the other hand, if the received response signal does not indicate a customer service request (step S114; No), or if the response signal cannot be received even after a predetermined time has passed since the customer service inquiry signal is transmitted, End the process. Similarly, when the customer terminal 2 does not make a response to receive a customer service (step S115; No), the process of the flowchart ends. Then, the clerk terminal 1 determines that the customer service should be terminated when there is a predetermined operation from the clerk 9 or when there is a notification from the customer terminal 2 that the customer service is terminated (step S118; Yes). Then, the process of the flowchart ends. Similarly, the customer terminal 2 determines that the customer service by the store clerk 9 has ended when there is a predetermined user operation to end the customer service or when there is a notification from the store clerk terminal 1 that the customer service is ended. Then (step S119; Yes), the process of the flowchart is terminated.

(2) Product Description Request Process FIG. 8 is a flowchart showing the procedure of the product description request process executed in the customer service mode when steps S116 and S117 in FIG. 7 are executed. In the product explanation request process, when the customer performs an operation of designating an arbitrary product on the EC screen as the explanation request product during customer service, the salesclerk terminal 1 displays the position and detailed information of the designated explanation request product. It is processing. The display system repeatedly executes the process of the flowchart shown in FIG. 8 in the customer service mode. At the time of executing the product description request process, the EC screen includes a display area for the captured image Im generated by the salesclerk terminal 1 and an area for displaying a list of products, as shown in FIGS. It shall have.

  First, the customer terminal 2 determines whether or not there is a user input that designates one product as an explanation request product from the list of products displayed on the EC screen (step S201). A specific method for accepting user input will be described later with reference to FIG. And when there is user input which designates one goods as an explanation demand goods from the list of goods displayed on EC screen (Step S201; Yes), customer terminal 2 is the goods number of the explanation demand goods designated. Is transmitted to the server device 3 (step S202). On the other hand, when there is no user input specifying one product as an explanation request product from the list of products displayed on the EC screen (step S201; No), the customer terminal 2 continues to monitor the presence or absence of the above-described user input. .

  Next, when the server device 3 receives a signal including the product number of the explanation request product, the server device 3 extracts product information corresponding to the received product number from the product information DB 342, and the same product number as the received product number. By identifying the position of the tag reader 6 that transmits the tag detection signal including the tag reader arrangement information 343, the arrangement location of the explanation request product is recognized (step S203). Then, the server device 3 transmits the product information of the explanation request product extracted or recognized and the information on the arrangement location to the clerk terminal 1 (step S204). In this case, the clerk terminal 1 displays an AR image that highlights the placement location of the received explanation request product and also displays an AR image that indicates the received product information (step S205). In this case, for example, the clerk terminal 1 recognizes and recognizes the range of view of the clerk 9 through the half mirror 11 by recognizing the position and orientation of the clerk terminal 1 based on the measurement result by the measuring unit 19. When there is a location where the explanation request product is located within the field of view, an AR image surrounding the location is displayed. As a result, the clerk 9 can preferably recognize the location and product information of the explanation request product specified by the customer, and can suitably explain the explanation request product.

(3) Explanation Product Emphasis Processing FIG. 9 is a flowchart showing a procedure of explanation product enhancement processing executed during the customer service mode. The explanation product highlighting process is a process of highlighting on the EC screen displayed on the customer terminal 2 the product that the clerk 9 explains to the hand. The display system repeatedly executes the process of the flowchart shown in FIG. 9 in the customer service mode. At the time of executing the explanation product enhancement process, the EC screen has a display area for the captured image Im generated by the salesclerk terminal 1 and an area for displaying a list of products, as in the case of the product explanation request process. To do.

  First, the clerk terminal 1 determines whether the tag reader 16 has read the product number from the tag 7 (step S206). When the tag reader 16 reads the product number from the tag 7 (step S206; Yes), the clerk terminal 1 transmits the read product number to the customer terminal 2 (step S207). On the other hand, if the tag reader 16 has not read the product number from the tag 7 (step S206; No), the clerk terminal 1 continuously monitors whether the tag reader 16 has read the product number from the tag 7.

  After receiving the product number, the customer terminal 2 highlights the product indicated by the received product number (step S208). Thereby, the customer terminal 2 can make a customer recognize easily the goods which the shop assistant 9 is explaining for the hand.

(4) Purchase-related processing FIG. 10 is a flowchart showing a procedure of purchase-related processing executed after the start of the customer service mode. In the purchase-related processing, the display system performs generation / update processing of the explained list information 344 and also displays the explained product list indicated by the explained list information 344 on the customer terminal 2, thereby We accept designation smoothly. The display system repeatedly executes the process of the flowchart shown in FIG.

  First, the clerk terminal 1 determines whether or not the clerk 9 has explained the product (step S301). For example, if the tag reader 16 reads the same product number from the tag 7 continuously for a predetermined time or more, or if there is a predetermined input during reading of the product number by the tag reader 16, the store clerk terminal 1 It is determined that the store clerk 9 explained the product indicated by. When the clerk terminal 1 determines that the clerk 9 has explained the product (step S301; Yes), the clerk terminal 1 transmits a request signal for updating the explained list information 344 to the server device 3 (step S302). In this case, the clerk terminal 1 transmits the product number read by the tag reader 16 or the IP address for identifying the customer terminal 2 in the above request signal.

  Then, the server device 3 updates the explained list information 344 (step S303). Specifically, the product indicated by the product number specified in the request signal in step S302 is regarded as a product that the clerk 9 has already explained to the customer of the customer terminal 2, and is added to the explained list information 344. And the salesclerk terminal 1 and the server apparatus 3 repeatedly perform the process of step S301-S303 until customer service is complete | finished.

  Then, the customer terminal 2 determines whether or not the customer service has ended (step S304). If the customer terminal 2 determines that the customer service has ended (step S304; Yes), the customer terminal 2 transmits a request signal for the explained list information 344 to the server device 3 (step S305). In this case, the server device 3 transmits the explained list information 344 to the customer terminal 2 (step S306).

  Thereafter, the customer terminal 2 displays the explained list on the EC screen based on the explained list information 344, and receives an input for designating a product that the customer wants to purchase from the list (step S307). Thereafter, when the customer terminal 2 detects that an arbitrary product is designated from the explained list, the customer terminal 2 communicates with the server device 3 or the like, and executes a settlement process or the like necessary for purchase of the designated product.

  Preferably, the clerk terminal 1 transmits the coupon information of the product to the customer terminal 2 at an arbitrary timing based on the user operation of the clerk 9 or the like. For example, the salesclerk terminal 1 transmits an instruction to transmit coupon information regarding the already described product to the server device 3 at a timing based on the user input in order to promote the purchase of the product during the customer service mode or at the end of the customer service mode. In this case, the server device 3 transmits the designated coupon information to the customer terminal 2. And the customer terminal 2 displays the received coupon information, and determines the presence or absence of utilization of coupon information based on the input by a customer.

[Display example]
Next, display examples based on the present embodiment will be described with reference to FIGS.

  FIG. 11A shows a display example of the half mirror 11 when visitor information corresponding to five customers is transmitted to the clerk terminal 1 in step S106 of FIG. In the example of FIG. 11 (A), the clerk terminal 1 includes the broken line frames 40A and 40B surrounding the position of the product estimated to be interested by each customer, the number of customers interested in the product in the broken line frame, and Windows 41A and 41B showing product information such as product images and prices are displayed as an AR image over the landscape on the half mirror 11. Note that the AR images such as the windows 41A and 41B shown in FIG. 11A are drawn with 0% transparency for convenience, but actually have a predetermined transparency.

  In this case, the clerk terminal 1 uses as the visitor information the product information such as the product number (SN number) and the product name of the product that each customer is displaying on the EC screen or selected with the cursor, and the location of the product. Are received from the server device 3. Then, the clerk terminal 1 determines that the product numbers (SN: 5180-15-300 in this case) included in the visitor information for the two customers match, and based on the information on the arrangement location of the products, the broken line frame 40A And a window 41A are displayed. Further, the store clerk terminal 1 determines that the product numbers (SN: 4380-12-300 in this case) included in the visitor information for the three customers match, and displays a broken line frame 40B and a window 41B.

  When the clerk 9 selects a customer to be a customer, the clerk terminal 1 selects a window (referred to as a “customer list”) indicating detailed customer information by selecting one of the windows 41A and 41B. To display.

  FIG. 11B shows a display example of the half mirror 11 after the window 41B is selected by the store clerk 9. In FIG. 11B, the clerk terminal 1 detects that the window 41B has been selected based on the input to the input unit 13, and displays a window 42B showing a customer list for three persons corresponding to the window 41B. ing. The store clerk terminal 1 recognizes that the position of the hand of the store clerk 9 is image-recognized by the captured image Im instead of the input to the input unit 13, and determines that the window 41B is selected. May be.

  The customer list shown in the window 42B of FIG. 11B has items such as an IP address, a visit time (VISIT DURATION), and a country name (COUNTRY). Then, the store clerk 9 refers to the country name or the like from the customer list for three persons shown in the window 42B, and selects a list of customers that can be handled. In this case, the store clerk 9 may select a list of customers who are determined to be in need of customer service, further considering visit time. In this case, the clerk terminal 1 accepts selection of one list from the customer list indicated by the window 42B, as in the selection method of the window 41B. Then, the clerk terminal 1 generates a customer service inquiry signal for the customer corresponding to the selected list and transmits it to the server device 3. Then, the server device 3 transmits the received customer service inquiry signal to the customer terminal 2 of the customer designated by the clerk terminal 1. The customer list shown in the window 42B includes arbitrary information held as the customer information DB 341 by the server device 3 such as the last access date, purchase history, and customer address (location area), in addition to the items of visit time and country name. May be included.

  FIG. 12 is a display example of an EC screen displayed on the display unit 29 of the customer terminal 2 immediately after receiving the customer service inquiry signal. In the example of FIG. 12, the customer terminal 2 displays an EC screen including a product display area 70 and a message area 71 provided above the product display area 70 based on the display information received from the server device 3. Yes. Here, the customer terminal 2 displays, on the product display area 70, product images 45A to 45F representing each product, purchase selection buttons 49A to 49F, characters describing each product, and the like. Further, the customer terminal 2 has a message field 43 for displaying a message indicating that the customer is willing to serve on the message area 71, and an accept button 44A for responding to the clerk terminal 1 that the customer service is received. A refusal button 44B for responding to the clerk terminal 1 that the customer service is not received and an icon 46 representing the state of the clerk 9 are displayed.

  When the customer terminal 2 detects that either the accept button 44A or the reject button 44B is selected by a mouse operation or the like, the customer terminal 2 transmits a response signal corresponding to the selected button to the server device 3. In the example of FIG. 12, the customer terminal 2 detects that the acceptance button 44 </ b> A has been selected, and transmits a response signal indicating that the customer service is received to the server device 3. Then, the server device 3 transmits the response signal to the clerk terminal 1. Thereafter, the customer terminal 2 and the store clerk terminal 1 establish a communication by receiving notification of both communication addresses and the like from the server device 3, and shift to the customer service mode.

  FIG. 13 is a display example of an EC screen displayed by the display unit 29 after the clerk terminal 1 and the customer terminal 2 shift to the customer service mode. In FIG. 13, after the clerk terminal 1 and the customer terminal 2 establish communication, the customer terminal 2 arranges the video display area 72 for displaying the captured image Im received from the clerk terminal 1 at the upper part, and the merchandise display area An EC screen with 70 at the bottom is displayed.

  Then, the customer terminal 2 displays the captured image Im in the video display area 72 and outputs the audio data received from the store clerk terminal 1 through the speaker 28. Further, the customer terminal 2 transmits the voice data generated by the microphone 230 to the clerk terminal 1. Thereby, the customer who uses the customer terminal 2 can receive an explanation about the product or ask a question with the clerk 9 through the video or audio.

  In the example of FIG. 13, the store clerk 9 holds the product 48 for explanation. In this case, the clerk terminal 1 reads the product number from the tag 7 attached to the product 48 by the tag reader 16 and notifies the customer terminal 2 of the product number via the server device 3. Accordingly, the customer terminal 2 highlights the product image 45D corresponding to the notified product number by the border effect. Thereby, the customer can recognize suitably the goods which the salesclerk 9 takes and explains.

  FIG. 14 shows a state in which an operation for designating an explanation request product is performed on the EC screen shown in FIG. In the example of FIG. 14, the customer wants to receive an explanation of a product indicated by a product image 45E different from the product 48 currently being explained, and selects the product image 45E by a click operation or the like. In this case, the customer terminal 2 detects that a product image 45E different from the highlighted product image 45D is selected, and displays a window 45E that prompts whether or not to receive an explanation. When the customer terminal 2 detects that the YES button in the window 45E has been selected, the customer terminal 2 transmits the product number corresponding to the product image 45E to the salesclerk terminal 1.

  FIG. 15A shows a display example of the half mirror 11 when an explanation request product is designated on the EC screen in FIG. In FIG. 15 (A), the clerk terminal 1 displays, as an AR image, a window 51 indicating information on a customer who is serving customers, a broken line frame 52 indicating the location of the explanation request product, and a window 53 showing information on the explanation request product. Is displayed.

  After the transition to the customer service mode, the salesclerk terminal 1 receives customer service target customer information from the server device 3 and displays it on the window 51. In FIG. 15A, the clerk terminal 1 displays the customer country name and purchase history registered in the customer information DB 341 on the window 51 by receiving from the server device 3. When the server device 3 records detailed customer registration information in the customer information DB 341 or the like due to member registration or the like, the salesclerk terminal 1 may receive the registration information and display it on the window 51. Here, the registration information corresponds to, for example, a name, age or age, sex, occupation, and the like.

  Further, the clerk terminal 1 follows the merchandise explanation request process of FIG. 8 based on the arrangement location of the explanation requested merchandise acquired from the server device 3 and the position and inclination of the clerk terminal 1 recognized by the measuring unit 19. By enclosing the requested product. Further, the clerk terminal 1 displays a window 53 displaying the product information of the explanation request product acquired from the server device 3 in association with the broken line frame 52. In this case, the clerk terminal 1 displays on the window 53 the product name, product number, price, color, size, number of products in stock, upper limit of possible discount rate, etc. as product information. Thereby, the store clerk 9 can grasp the arrangement location and the product content of the explanation request product specified by the customer on the EC screen, and can explain the product suitably for the customer.

  FIG. 15B is a display example of the half mirror 11 in which there is no explanation request product and an alternative product similar to the explanation request product is presented as a recommended product. In the example of FIG. 15B, the clerk terminal 1 displays a window 53A showing the product information of the alternative product in association with the broken line frame 52A surrounding the place of the alternative product.

  In this case, the server device 3 recognizes that the explanation request product specified by the customer terminal 2 does not exist in the store 5 based on the signal transmitted from the tag reader 6 and searches for an alternative product of the explanation request product, The location of the product and product information are transmitted to the clerk terminal 1. In this case, for example, the server device 3 stores in advance a table or a map that associates similar products with each other, and determines an alternative product with reference to the table or the like.

  The clerk terminal 1 may display, as a recommended product, a product that is preferably worn with the explanation request product or a product that the store 5 wants to actively sell, instead of displaying the substitute product of the explanation request product as a recommended product. Good. In this case, for example, the server device 3 stores in advance a table or a map that associates products recommended to be worn at the same time, determines recommended products with reference to the tables, and stores information on the products. Transmit to the clerk terminal 1. In this case, the clerk terminal 1 displays the recommended product information as the AR image together with the explanation request product information.

  FIG. 16 is a display example of the EC screen when the store clerk 9 removes the store clerk terminal 1 to respond to the store visitor in the customer service mode. In the example of FIG. 16, the clerk terminal 1 detects that the clerk terminal 1 is tilted above a horizontal plane by a predetermined angle or more by the measurement unit 16, or the clerk terminal 1 It is detected that the camera has been removed from the position, and transmission of the captured image Im and audio data is stopped. Then, the clerk terminal 1 notifies the customer terminal 2 that it cannot respond. In this case, the customer terminal 2 displays “SORRY.” In the video display area 72 instead of the captured image Im, and notifies the customer that the captured image Im cannot be displayed. Thereby, it can suppress suitably that the customer who visited the customer is displayed on the EC screen.

  In FIG. 16, the customer terminal 2 displays the icon 46 differently by changing the background color from the case of FIG. Thus, the customer terminal 2 changes the display mode of the icon 46 according to the detachment state of the clerk terminal 1 of the clerk 9 wearing the clerk terminal 1, so that the customer who uses the customer terminal 2 is in the state of the clerk 9. Can be suitably recognized.

  The salesclerk terminal 1 may process the display portion of the person in the photographed image Im so that the person cannot be specified and send it to the customer terminal 2 to display the processed photographed image Im on the customer terminal 2. . In this case, the clerk terminal 1 recognizes that an arbitrary person's face is reflected in the captured image Im and its display area by a known image recognition technique. Also by this, it can suppress suitably that the customer who visited the store is displayed on the EC screen.

[Modification]
Hereinafter, modified examples suitable for the above-described embodiments will be described. The following modifications may be applied in any combination to the above-described embodiments.

(Modification 1)
In FIG. 15 (A), instead of or in addition to displaying the position and product information of the explanation request product, the store clerk terminal 1 is in the case where the cursor position is over the product image on the EC screen. The position and product information of the product (also referred to as “cursor superimposed product”) indicated by the product image may be displayed.

  FIG. 17A shows a display example of the half mirror 11 according to this modification. In the example of FIG. 17A, the clerk terminal 1 displays, as an AR image, a dashed frame 52B indicating the placement location of the cursor superimposed product and a window 53B indicating the product information of the cursor superimposed product in association with each other. .

  In this case, the customer terminal 2 detects that the cursor position has overlapped with the display position of the product image displayed on the EC screen, and the server device 3 uses the product number of the product indicated by the product image as the product number of the cursor superimposed product. To notify. In this case, the customer terminal 2 may recognize the product indicated by the product image in which the cursor position overlaps for a predetermined time or more as the cursor superimposed product. Then, the server device 3 acquires the notified location and product information of the cursor superimposed product by referring to the product information DB 342 and the tag reader arrangement information 343 and notifies the salesclerk terminal 1 of the information. And the salesclerk terminal 1 displays the broken line frame 52B which shows the arrangement | positioning place of object goods, and the window 53B which shows the goods information of object goods similarly to step S205 of the goods description request | requirement process of FIG.

  By doing in this way, the salesclerk terminal 1 can make the salesclerk 9 recognize suitably the arrangement | positioning location and merchandise information of the goods estimated that a customer is interested, and can perform a customer service smoothly.

(Modification 2)
When the EC screen of the customer terminal 2 becomes inactive (that is, on the back of another screen), the salesclerk terminal 1 determines that the customer is not in a state of receiving a customer and notifies the salesclerk 9 to that effect. Also good.

  FIG. 17B shows a display example of the half mirror 11 in this modification. In the example of FIG. 17B, the clerk terminal 1 displays a window 54 indicating that the EC screen of the customer terminal 2 is not active as an AR image. In this case, after starting the customer service mode, the customer terminal 2 monitors whether or not the EC screen is active. If the EC screen becomes inactive, the customer terminal 2 transmits a signal indicating that to the salesclerk terminal 1. Note that the clerk terminal 1 may notify the clerk 9 that the EC screen is not active by voice instead of displaying the window 54.

  According to this aspect, the clerk terminal 1 can suitably prevent the clerk 9 from continuing the product description unnecessarily when the customer is browsing a screen other than the EC screen.

(Modification 3)
The server device 3 may store a database of position information of each product in the store 5 in the storage unit 34 instead of the tag reader arrangement information 343. In this case, in step S203 of the product description request process of FIG. 8, the server device 3 refers to the above-described database stored in the storage unit 34 and acquires the location where the description request product is arranged.

(Modification 4)
In the flowchart of FIG. 7 showing the processing outline of the embodiment, the server device 3 transmits visitor information to the clerk terminal 1 in step S106 regardless of whether there is an inquiry from the clerk terminal 1. Instead, the server device 3 may transmit the generated visitor information to the clerk terminal 1 only when there is a visitor information inquiry from the clerk terminal 1. For example, in this case, when the store clerk terminal 1 detects a predetermined operation from the store clerk 9 or recognizes that the store clerk 9 wears the store clerk terminal 1, the store clerk terminal 1 inquires the visitor information and Receive information.

(Modification 5)
The server device 3 monitors whether or not the store clerk 9 is available for customer service. When the store clerk 9 is available for customer service and there is an EC screen acquisition request, the display information of the EC screen shown in FIG. 12 is displayed. You may transmit to the customer terminal 2.

  In this case, the clerk terminal 1 recognizes the detachment state of the clerk terminal 1 based on the measurement unit 16 or a detachable sensor (not shown) and transmits the recognition result to the server device 3 at predetermined intervals. Then, based on the above recognition result, the server device 3 determines that the store clerk 9 is ready to serve when the store clerk 9 wears the store clerk terminal 1. Then, when there is an EC screen acquisition request when the clerk 9 is in the customer service enabled state, the server device 3 transmits the display information of the EC screen shown in FIG. 12 to the customer terminal 2 and displays a message field 43 on the EC screen. And the accept button 44A are displayed.

(Modification 6)
In step S <b> 105 of FIG. 7, the server device 3 transmits visitor information corresponding to the customer whose visit time exceeds the threshold to the clerk terminal 1. Instead of this, or in addition to this, the server device 3 may determine whether or not the visitor information should be transmitted to the clerk terminal 1 based on information other than the visit time.

  For example, the server device 3 is a regular customer or a customer who is confused about purchasing when the number of accesses to the EC screen recognized based on the customer information DB 341, the previous access date and time, and the like satisfy predetermined conditions. The visitor information may be transmitted to the clerk terminal 1.

(Modification 7)
In step S <b> 106 of FIG. 7, the server device 3 may simultaneously transmit visitor information to the clerk terminal 1 that exists in the same series of stores as the store 5. And the server apparatus 3 starts the direct connection with the customer terminal 2 with respect to the salesclerk terminal 1 which transmitted the customer service inquiry signal as a response with respect to the transmitted visitor information, and makes it shift to customer service mode. Similarly to the modified example 5, when the server device 3 monitors whether or not the clerk 9 of each store 5 is in a serviceable state, only the clerk terminal 1 of the clerk 9 in the serviceable state is provided. Visitor information may be transmitted.

(Modification 8)
The salesclerk terminal 1 may have some functions of the server device 2 described in the embodiment. For example, the clerk terminal 1 may store various types of information such as the product information DB 342 and the tag reader arrangement information 343 and execute the product description request process of FIG. 8 without communicating with the server device 2. In this case, the clerk terminal 1 directly receives the product number of the description request product from the customer terminal 2 and performs the display processing related to the product description request processing shown in FIG. 15 without communicating with the server device 2. In this case, the store clerk terminal 1 receives a signal from the tag reader 6 instead of the server device 3 and recognizes an arrangement location of each product in the store 5 based on the tag reader arrangement information 343.

(Modification 9)
The display system may include a mobile terminal such as a smartphone having a camera instead of the clerk terminal 1. In this case, the mobile terminal displays an image captured by the camera on the display, and displays the AR image shown in FIGS. 11 and 15 in a superimposed manner on the displayed image. In this modification, the mobile terminal or the display of the mobile terminal is an example of the “display unit” in the present invention.

(Modification 10)
Even when the customer meets the store clerk 9 and the store 5 and receives a customer, the store clerk terminal 1 may display the arrangement location and the product information of the explanation request product specified by the customer by the AR image.

  In this case, for example, the clerk terminal 1 transmits the voice data of the customer input by the microphone 130 to the server device 3, and the server device 3 extracts a phrase specifying a product such as a product name from the voice data. Then, the server device 3 recognizes the product related to the extracted phrase as an explanation request product, and transmits the location of the product and product information to the store clerk terminal 1, thereby placing the product on the store clerk terminal 1. Display location and product information. In this case, the server device 3 may store in advance a map or the like indicating the correspondence between the phrase that can be extracted and the product. Note that the clerk terminal 1 may execute voice recognition processing such as phrase extraction described above instead of the server device 3.

(Modification 11)
In the customer service mode, the clerk terminal 1 and the customer terminal 2 communicated without going through the server device 3. Instead, the clerk terminal 1 and the customer terminal 2 may exchange data via the server device 3 even in the customer service mode.

(Modification 12)
The salesclerk terminal 1 receives information on the location of each product (also referred to as “product location information”) from a terminal (also referred to as “headquarter terminal”) used by the headquarters of the store 5, and determines the location of each product. An AR image may be displayed.

  In this case, the headquarters terminal generates product placement information indicating the placement (display) location of each product for each product number, and transmits the information to the clerk terminal 1. When the clerk terminal 1 reads the product number from the tag 7 by the tag reader 16, the store clerk terminal 1 refers to the product placement information received from the headquarters terminal and recognizes the placement location of the corresponding product. Then, the clerk terminal 1 displays an AR image surrounding the recognized arrangement location, as in the example of FIG. Thereby, the store clerk can display goods appropriately according to the instruction of the headquarters. In this modification, the user of the headquarters terminal is an example of the “user” in the present invention.

  Instead of the headquarters terminal, a terminal used by a store manager or the like in the store 5 may transmit the product arrangement information to the clerk terminal 1.

1 Clerk terminal 2 Customer terminal 3 Server device 5 Store 6 Tag reader 7 Tag 9 Clerk

Claims (1)

Information on the product selected by the user using the user terminal is associated with the selected product on the display unit based on the positional information on the selected product, and is a person other than the user. A control unit to be displayed on the display unit visually recognized by
A photographing unit for photographing images,
When the person is reflected in the image, the control unit processes the image into an image in which the person cannot be specified, and causes the communication unit to transmit the processed image to the user terminal. apparatus.

Images