WO2021176552A1

WO2021176552A1 - User terminal and program

Info

Publication number: WO2021176552A1
Application number: PCT/JP2020/008864
Authority: WO
Inventors: 好孝西田; 由樹源
Original assignee: 株式会社ＡＳＩＡＮＦｒｏｎｔｉｅｒ
Priority date: 2020-03-03
Filing date: 2020-03-03
Publication date: 2021-09-10

Abstract

A user terminal (100) has: an imaging means; a display means (130) that displays an image captured by the imaging means; a storage means (150) that stores information about a commodity and a condition concerning the commodity; a specification means (121) that, on the basis of the image captured by the imaging means and the information about the commodity, specifies the imaged commodity; and a display control means (122) that, when the specified article satisfies the condition, processes the image of the article and displays the image on the display means.

Description

User terminal and program

The present invention relates to a search for displayed products.

For example, at a place where alcoholic beverages are served, such as a bar, there is a need to easily find a product that suits one's taste from among the many products on display. In relation to this point, in order to support the inventory management of products, there is a technique for identifying the brand of each product by photographing the display shelves of the store (Patent Document 1 and the like).

JP-A-2017-117362

With the technique of Patent Document 1, even if an existing product can be grasped, it is not possible to present a product that suits the taste of the customer.
An object of the present invention is to support finding a product that suits a customer's taste from among the displayed products.

In one embodiment, the present invention is photographed by a photographing means, a display means for displaying an image photographed by the photographing means, a storage means for storing product information and conditions related to the product, and the photographing means. Based on the image and the information of the product, the specific means for identifying the photographed product and, when the specified article satisfies the above conditions, the image of the article is processed and displayed on the display means. A user terminal having a display control means is provided.

According to the present invention, the work of finding a product that suits a customer's taste from among the displayed products is supported.

The functional diagram of the user terminal 100. Example of user preference information. An example of a user's order history. Example of product identification information. Example of detailed product information. An example of inventory information. An operation example of the user terminal 100. An operation example of the user terminal 100 (No. 2). An example showing the usage state of the user terminal 100. An example of a screen displayed on the user terminal 100 (No. 1). An example of a screen displayed on the user terminal 100 (No. 2). An example of a screen displayed on the user terminal 100 (No. 3). An example of a screen displayed on the user terminal 100 (No. 4). An example of a screen displayed on the user terminal 100 (No. 5).

<Example>
FIG. 1 is a functional diagram of the user terminal 100. The user terminal 100 is a device carried by the user, and includes a photographing means 110, a control unit 120, a display means 130, an input means 140, a storage means 150, a touch screen 160, a communication means 170, and the like. Includes position acquisition means 180. Hereinafter, the case where the user terminal 100 is a smartphone will be described as an example, but the call function is not essential, and conversely, the user terminal 100 may be a device to which a function not related to the present invention is added.

The photographing means 110 is mounted as a module including a camera, a lens, a light receiving element, and other optical elements, and photographs the surroundings of the user terminal 100 according to the user's operation. Specifically, the photographing means 110 generates data indicating a photographed image and supplies the data to the specific means 121. The acquired image may be a still image or a moving image. Further, an infrared image may be acquired instead of the visible light image.

The display means 130 is realized by, for example, a liquid crystal screen, and displays an image according to an instruction from the control unit 120.

The input means 140 is an input device such as a touch pad or a mouse, and accepts a user's designation of a position on the screen. Preferably, the functions of the display means 130 and the input means 140 are integrated to form the touch screen 160.

The communication means 170 is realized as a wireless communication module such as WiFi (registered trademark), 4G, 5G, etc., is connected to another device via a communication network such as the Internet, and is necessary for identifying a product as needed. Get information and product information.

The position acquisition means 180 is implemented as a module for receiving radio waves from GPS satellites or a communication module for receiving information including position information from a base station or a wireless router, and obtains the current position of the user terminal 100. Get the information shown.

The storage means 150 stores at least product information and conditions regarding product selection. In a preferred embodiment, the storage means 150 stores user preference information, order history, product identification information, product detail information, and inventory information, as shown in FIGS. 2 to 6. The above conditions are generated at least based on user preference information.
In addition, the storage means 150 stores an application program (hereinafter, referred to as an application) that is executed by the processor of the user terminal 100 and realizes the function of the user terminal 100.

FIG. 2 is an example of user preference information. In this example, for each type of beverage, in addition to the characteristics of the user's preference of the user terminal 100 from the viewpoint of alcohol content, aroma, taste, flavor, and throat, the price (budget amount assumed by the user), Information such as brand, place of origin, etc. may be included. In short, it contains information used to determine to what extent the user's preferences match (or do not) the displayed product. Specifically, a condition for determining whether or not a certain product is suitable for this user is generated based on the user preference information.

FIG. 3 is an example of the user's order history. The store visited by the user of the user terminal 100 in the past, the date and time of the visit, the product ordered at that store, and the product registered by the user (for example, a product that did not reach the order but was of concern to the user and was ordered. Includes information (which may be irrelevant to the presence or absence of). This product is identified by, for example, a number assigned according to GTIN (Global Trade Item Number), which is an international standard for uniquely identifying a product. Alternatively, it may be identified by the name of the manufacturing company, the brand name, the place of origin, the age, the type, and the like.

FIG. 4 is an example of product identification information. In this example, each ID that uniquely identifies the product includes information necessary for distinguishing the product from its appearance (package or container). As this ID, for example, a GTIN-compliant 13-digit number string can be adopted. In this example, as information about the label affixed to the product, the characters on the label (typically, a trademark for identifying the product such as a brand name), the pattern of the label, the shape of the label, and the label In addition to the size and the position where the label is attached to the container, the information on the container (bottle shape, color, etc.) can be included. The example in the figure is just an example, and any information may be used as long as it is information that characterizes the appearance of the product.

FIG. 5 is an example of detailed product information. The detailed product information may be any information related to the product, such as the characteristics and properties of the product, but is preferably information that can be referred to when the user decides to order the product. In this example, text data, figures, illustrations, photographs, graphs and other image information for explaining the characteristics of the product, and the manufacturer, retailer, wholesaler, etc. of the product are associated with the product ID such as GTIN. Information (URL link) for accessing the information of the provided product is included.

FIG. 6 is an example of inventory information. For each store, the location information of the store and the information for identifying all the products handled in the store are stored in association with each other.

The control unit 120 is realized as one or more general-purpose or image processing dedicated processors, and includes a specific means 121 and a display control means 122.

The specifying means 121 identifies the product photographed by the photographing means 110 based on the image photographed by the photographing means 110 and the product information read from the storage means 150. When the specifying means 121 determines that the product cannot be specified due to the camera work of the user, the specifying means 121 may output a signal to that effect to the display controlling means 122. Here, "identifying" a product is not limited to specifying all publicly available information such as the producer, place of origin, model number, type, brand, brand, version, variation, etc. of the product, for example, the brand is It is specified to the extent that it can be specified but it is not possible to specify which of the multiple variations exists, or the manufacturer can be specified, but the candidates for the applicable model number can be narrowed down to three. It means to include the case. In short, it suffices if the information that contributes to the decision-making when the user selects the product can be obtained from the photographed image, and even if it can be said that the product cannot be uniquely identified or is incompletely identified, the product is described as " Corresponds to "identify".

When a product is specified, the identification means 121 determines whether or not each of the specified products satisfies the condition. Specifically, the specific means 121 refers to the user preference information and the product identification information stored in the storage means 150 to determine the degree of conformity with the conditions, and the determination result is information indicating an image area of the product. Is output to the display control means 122 in association with. The degree of conformity can be calculated based on the number of matching items, for example, when the condition consists of a plurality of items (matching / not matching).

The specific means 121 determines whether or not the user's camera work and shooting conditions exceed a predetermined value based on the image data, and allows the user's camera work (for example, the degree of camera shake caused by a sudden camera operation). When the product cannot be specified due to (when it is out of the range), a flag indicating that fact is output to the display control means 122.

When the article specified by the specific means 121 satisfies the condition, the display control means 122 processes the image of the article and displays it on the display means 130. Specifically, the image area to be processed and the processing content are determined based on the information supplied from the specific means 121.
In a preferred embodiment, the processing process is a process that makes the identified article stand out relative to other commodities. For example, it is an effect process (hereinafter referred to as "light emission process") that imitates the appearance of the product emitting light (= shining). Further, the processing process is a process (in other words, a process of making the image area of an article other than the specified article (for example, a product can be identified but the goodness of fit to the condition is less than or equal to a predetermined value) relatively inconspicuous). It may be a process for reducing visibility). Specifically, the image area includes blackening processing, blurring processing, processing for reducing lightness or saturation (hereinafter referred to as mask processing), and the like.

In a preferred embodiment, the display control means 122 may change the content of the effect processing according to the degree of matching with the conditions. Specifically, at least one of color, brightness (luminance), and light emission timing regarding light emission may be changed according to the degree of conformity with the conditions. For example, an image having a higher brightness (for example, an image that looks dazzling in gold) is added to a product having a higher degree of conformity with the conditions. Alternatively, when this effect is an effect process for drawing an image that imitates the blinking of a predetermined object, the blinking time interval is set to be shorter as the degree of matching is higher. Similarly, with respect to masking, the degree to which the image of the product becomes inconspicuous is changed by controlling other drawing methods such as brightness, saturation, lightness, and edge blurring, depending on the degree of conformity to the conditions. May be good.
By changing the mode of light emission in this way, the user can intuitively understand the degree of conformity with the conditions.
The display control means 122 supplies the image data applied to the image data acquired from the photographing means 110 by the processing processed in this way to the display means 130.

Further, the display control means 122 may generate image data so as to superimpose and display the information about the specified product on the image taken by the photographing means 110. Further, when the display control means 122 receives a flag from the identification means 121 indicating that the product cannot be specified due to the user's camera work or the like, a message to that effect (for example, "the product cannot be specified due to large blurring"). You may generate image data in which the text information) "Please move it a little more slowly" is superimposed. This encourages the user to perform appropriate camera operations.

FIG. 7 is an operation example of the user terminal 100. The user who visits the store starts the above-mentioned application. Then, first, the position acquisition means 180 acquires the position information (S501), and refers to the inventory information stored in the storage means 150 to identify the store where the user is currently located. As shown in FIG. 9, the user points the camera lens of the photographing means toward the display shelf in the store (here, a bar serving alcoholic beverages), performs a predetermined operation, and starts photographing. After that, the user terminal 100 continuously acquires image data at a predetermined frame rate. As shown in FIG. 10, the acquired data is displayed on the display means 130 in real time (S502). When the user changes the direction of the user terminal 100 to change the shooting target as necessary, an image corresponding to the shooting target is displayed.

The identification means 121 sequentially analyzes the acquired image data and attempts to identify each photographed product (S503). At this time, for example, when it is not possible to specify one product but a plurality of candidates are found, the identification means 121 refers to the storage means 150 as necessary, and the user is present from the acquired position information. You may perform processing to improve the accuracy of product identification, such as identifying a store and using the information of the products handled at the store to narrow down to one from the plurality of candidates. In addition, there may be a product that cannot be specified or is specified incorrectly.

Subsequently, the specifying means 121 determines whether or not the specified conditions are met for each of the specified products (S505). For each product whose degree of matching is equal to or higher than a predetermined value (S506: YES), as shown in FIG. 11, the specific means 121 determines a light emitting mode according to the degree of matching, and performs light emitting effect processing on the product. While superimposing and displaying on the screen (S507), the information of the product is read out from the storage means 150 and superimposing and displaying (S509). In the example of FIG. 11, it is determined that the product of the product image OB1-1 satisfies the condition, and as a result, the light emitting object OB4 and the product information window OB2-1 are displayed in the vicinity of the product image OB1-1. ing. In the product information window OB2-1, for example, a photograph showing a photograph of the product, a description, and a chart showing the characteristics of the flavor are displayed.
On the other hand, for each product whose degree of matching is equal to or less than a predetermined value (S506: NO), the image of the product is masked (S508). In the example of FIG. 11, the masking objects OB3-1, OB3-2, and OB3-3 are masked as a result of determining that the degree of matching is a product of a predetermined value or less.

The control unit 120 performs light emission processing or other conspicuous processing on the product image, the target product on which the light emission processing or other conspicuous processing is performed on the product image, mask processing, or the like. The target product that has been subjected to the inconspicuous processing is stored in the storage means 150 (S510).

When the user performs some operation on the touch screen 160 (S504: YES), the process proceeds to S550 in FIG. When the operation is a long-press operation (S550: YES), the specifying means 121 identifies the product reflected in the position by the long-pressing operation, reads the information of the specified product from the storage means 150, and takes a picture. It is superimposed and displayed on the image (S551). FIG. 12 shows an example of a screen displayed at this time. In this example, when the long press operation is performed on the product image OB1-2, the product information window OB2-2 is displayed. In this way, by specifying the photographed product by the user, it is possible to acquire information about the product that does not meet the conditions but is of interest to the user.

Further, when the user performs a flick operation (for example, the direction from the edge of the screen to the center) (S552: YES), the guidance information window OB5 is superimposed and displayed as shown in FIG. 13 (S553). The information displayed in the guidance information window includes, for example, information about the current store generated based on the order history and inventory information, and information about the products that the user has placed an order for in the past at this store. A list of products that have been registered as favorites (described later), links to websites that provide further detailed information on products, links to websites that provide services for purchasing products by mail order, and conditions. Various guidance information, such as information for changing the information, may be included. The guidance information window OB5 is closed by flicking in the opposite direction.

Further, when the user performs a tap operation (S554: YES), the display control means 122 superimposes and displays a pin image on the position on the screen where the tap operation is performed (S555). FIG. 14 is an example of the screen displayed at this time. In this example, when the trademark image OB1-3 is tapped, the pin object OB6 is displayed. Then, the display control means 122 stores in the storage means 150 that the product is registered as a favorite product (S556). Then, the process returns to S502 of FIG. 7, and the processes of S502 to S510 (movement is taken, the photographed product is specified, and the image processing according to the condition is met) are repeated. When a flick operation is performed on a pin object, the display of the pin object disappears.

According to the above embodiment, the user can easily find a product that suits his / her taste among a large number of displayed products. At this time, for example, even if the product is a brand that the user does not know, if the conditions are met, the product is visually notified, so that the hurdle to try an unknown product is lowered. Further, for example, it is not necessary to ask the clerk whether the product suits his / her taste, and it is only necessary to inform the clerk of the product to be ordered determined by the user based on the product presented by the method of the present invention. , It is convenient for users who do not want unnecessary communication with the clerk or who want to feel free to try products of brands that they have never tried before. On the other hand, the clerk does not need to have abundant knowledge to answer user's questions about all the products he handles, so for example, he should concentrate on work other than providing product knowledge to customers. Can be done.

Also, even if the displayed product is known to the user, the presence of the product should be visually recognized due to circumstances such as a small label, small characters on the label, or dim lighting. Even in situations where it is difficult (for example, in a place such as a bar counter), if the shooting conditions (illuminance, resolution, zoom function, etc.) that can perform image analysis are satisfied, the user can display the product. Just take a picture of the shelf and you will be presented with a product that suits your taste.

Even if the number of displayed products is too large to fit in the camera frame, you can search the entire displayed products by changing the shooting range by operating pan, tilt, zoom, etc. Can be done. When a product that meets the conditions is detected, that fact is reflected on the screen display in real time, so the user can exit the application when one product that meets the conditions is found, and for the time being, the entire displayed product. If multiple conforming products are detected, it is possible to examine the purchased products from among them. At this time, the user can narrow down the products by referring to the difference in the light emitting mode reflecting the degree of conformity with the conditions. In addition, products with low (or non-conforming) degree of conformity are masked, and many products with similar appearances are densely displayed, making it difficult to see individual products. Even so, it becomes easy to find the product that the user wants. Note that the "display" does not necessarily mean only a state in which the products are arranged in an orderly manner, but in short, it is sufficient that a plurality of products are present at a position where they can be captured by the photographing means.

<Modification example>
The above-mentioned smartphone is an example of a user terminal, and as the user terminal, for example, smart glasses (glasses type terminals) using so-called augmented reality (AR) technology may be used. In short, it suffices to have a function of superimposing and displaying an image of the user's surroundings (for example, a display shelf) and an image subjected to the above image processing on the image and providing the user with the image. In this case, in order to specify the position on the screen, an existing input mechanism such as line-of-sight detection or voice input may be used.

In the above embodiment, the product to be searched was a beverage, but it may be a food product, daily necessities, or an electric appliance. In short, the present invention can be applied as long as it is possible to support the search and selection of products from a large number of displayed products.

The above-mentioned long press operation, flick operation, and tap operation are merely examples of operations for inputting commands such as product specification, registration instruction holding, and information window display.

The above-mentioned "light emission processing" does not have to be something that everyone perceives that a product emits light, but in short, it visually distinguishes a product that meets (or has a high degree of conformity) with other products. Any image processing may be used to support recognition by distinguishing between the two. Further, in the above embodiment, the product that satisfies the conditions is subjected to the light emission processing and the product information is displayed, but only the light emission processing may be performed and the display of the product information may be omitted.
Alternatively, depending on the degree of conformity with the conditions, it may be selected whether to perform only the light emission processing or to perform the light emission processing and the display of the product information. For example, the degree of conformity to the conditions is set in three stages ("high", "medium", "low"), and when the degree of conformity is "high", light emission processing and product information are displayed, and the degree of conformity is "". If it is "medium", only light emission processing is performed, and if the goodness of fit is "low", masking processing is performed.
Alternatively, whether or not to execute the display of the product information may be determined according to the order history information. For example, if the order history information identifies that the item has been ordered in the past, the user is likely to already know the information about the item, depending on the goodness of fit of the condition. No product information is displayed.
Alternatively, the user may set in advance whether or not to display the product information in addition to the light emission processing. For example, an advanced mode and a beginner mode are provided, and one of the modes is specified in advance by the user. In the case of the advanced mode, product information is provided regardless of the degree of conformity with the conditions. In the case of beginner mode without displaying, always perform light emission processing and product information display as a set.

Further, when notifying the existence of a product that meets the conditions, in addition to the notification by light emission or other image processing, the notification is made by vibrating the housing of the user terminal 100 and / or outputting a predetermined sound. You may.

The product information and / or guidance information window that is displayed when the conditions are met or when the user specifies a product does not need to be displayed superimposed on the captured image, and is on a screen different from or on the same screen as the captured image. May be displayed in different display areas. For example, when the user terminal 100 physically has a plurality of display screens, the captured image and the product information are displayed on different screens. Alternatively, the product information may be displayed on a device different from the user terminal 100. For example, when a user carries a plurality of devices, only the photographed image is displayed on the smartphone, and only the product information is displayed on the device worn on the wrist. Alternatively, the product information may be displayed on a device not owned by the user (for example, a screen provided in the store). In this case, wireless communication is performed between different devices, and a product information display command is transmitted from the user terminal 100 to another device.

The user preference information may be input by the user himself or may be automatically generated by the user terminal 100. Specifically, the specific means 121 automatically generates preference information according to a predetermined algorithm based on the user's order history. At this time, information on the preferences of other users and order history may be used. For example, order history and preference information are acquired for a plurality of users, the relationship between order history and preference information is learned using a method such as machine learning to generate a learning model, and order history information is input to this learning model. By doing so, the conditions may be determined.

A part of the function of the user terminal 100 may be possessed by another device (for example, a server connected to the user terminal 100 via a network). For example, all or part of the functions of the control unit 120 and the storage means 150 are omitted from the user terminal 100. The user terminal 100 need at least have a photographing means, a display screen, an input means and a communication means, and for identification of the photographed product and / or image processing, an external device via the communication means. May receive the processing result. At this time, instead of storing the information necessary for identifying the product and determining the image processing content in the storage means 150, an external storage device stores it, and the user terminal 100 acquires it as needed. Alternatively, the user terminal 100 may inquire of the search server for detailed information on the product or information necessary for identifying the product, and may acquire the search result.
In short, the present invention includes a step of inputting conditions related to product selection, a step of acquiring product information, and a step of acquiring an image taken by a user in an information processing system including one or a plurality of information processing devices. , The step of specifying the photographed product based on the image and the information of the product, and the step of processing and displaying the image of the article when the specified product satisfies the above conditions. It should be executed.

100: User terminal 110: Imaging means 120: Control unit 121: Specific means 122: Display control means 130: Display means 140: Input means 150: Storage means 160: Touch screen 170: Communication means 180: Position acquisition means

Claims

Shooting means and
A display means for displaying an image taken by the shooting means and a display means.
A storage means for storing product information and conditions related to product selection,
A specific means for identifying the photographed product based on the image photographed by the photographing means and the information of the product, and
A user terminal having a display control means that processes an image of the article and displays it on the display means when the specified article satisfies the above conditions.
The user terminal according to claim 1, wherein the processing process is an effect process that imitates a state in which the specified article emits light.
The user terminal according to claim 1 or 2, wherein the processing is a process for reducing the visibility of articles other than the specified article.
The user terminal according to claim 2, wherein at least one of the color, the brightness, and the timing of light emission differs depending on the degree of matching with the above conditions.
Further equipped with a means for acquiring the position information of the own user terminal,
The storage means further stores the location information of the store and the product information handled by the store.
The user terminal according to any one of claims 1 to 4, wherein the specific means further specifies the photographed product based on the product information.
The photographing means continuously captures images and
The user terminal according to any one of claims 1 to 5, wherein the specific means notifies the user when the product cannot be specified due to the camera work of the user.
The user terminal according to any one of claims 1 to 6, wherein the processing process is a process of displaying information about the specified product.
It also has an input means that accepts the user's specification of the position on the display screen.
The identification means identifies the product in the designated position and
The user terminal according to any one of claims 1 to 7, wherein the display control means displays information on the specified product.
The user terminal according to any one of claims 1 to 8, wherein the condition includes a condition relating to a user's preference.
For computers
Steps to enter conditions for product selection and
Steps to get product information and
Steps to get the image taken by the user,
Based on the image and the information of the product, the step of identifying the photographed product and
A program for executing a step of processing and displaying an image of the specified product when the specified product satisfies the above conditions.