JP2020057107A - Article information providing system, article information providing method, and program - Google Patents

Abstract

To provide an article information providing system, an article information providing method, and a program capable of providing article information corresponding to an article from a description of the article.SOLUTION: An article information providing system 10 includes a character information extraction unit that extracts character information about an article from an image including a description of the article, a diagram information extraction unit that extracts diagram information about the article from the image, and an article information acquisition unit that acquires article information corresponding to the article based on at least one of the character information extracted by the character information extraction unit and the diagram information extracted by the diagram information extraction unit.SELECTED DRAWING: Figure 1

Description

  The present disclosure generally relates to a product information providing system, a product information providing method, and a program. The present disclosure particularly relates to a product information providing system, a product information providing method, and a program for providing product information corresponding to a product.

  Patent Literature 1 discloses an information presentation system that presents information using a product catalog. The information presenting system includes an information presenting device connected to each other via a network, and an information presenting server. According to the identification information transmitted from the information presenting device, the information presenting system retrieves the image-related information from the storage means in the information presenting server and transfers the image-related information to the information presenting device to which the identification information has been transmitted via the network. The image-related information transmitted from the server is transmitted by the information presenting device of the information presenting apparatus.

JP-A-2004-54307

  In Patent Literature 1, various product images (descriptions of products) are printed on a product catalog, and in each page, identification information on products on the entire page is printed as images. That is, in Patent Literature 1, it is not possible to acquire product information only by describing a product such as a product image, and it is necessary to print identification information on the product in advance.

  The problem is to provide a product information providing system, a product information providing method, and a program that can provide product information corresponding to a product from the description of the product.

  A product information providing system according to an embodiment of the present disclosure includes a character information extraction unit, a diagram information extraction unit, and a product information acquisition unit. The character information extracting unit extracts character information on the product from an image including a description of the product. The diagram information extracting unit extracts diagram information on the product from the image. The product information obtaining unit obtains product information corresponding to the product based on at least one of the character information extracted by the character information extracting unit and the diagram information extracted by the diagram information extracting unit. .

  A method for providing product information according to an aspect of the present disclosure includes extracting character information related to the product from an image related to the product, extracting diagram information related to the product from the image, and comparing the character information with the diagram information. The product information corresponding to the product is obtained using at least one of the products.

  A program according to an embodiment of the present disclosure is a program for causing one or more processors to execute the product information providing method.

  According to the aspect of the present disclosure, it is possible to provide product information corresponding to a product from the description of the product.

FIG. 1 is a block diagram of an entire system including a product information providing system according to an embodiment. FIG. 2 is an explanatory diagram of a process in the product information providing system. FIG. 3 is another explanatory diagram of the processing in the product information providing system. FIG. 4 is another explanatory diagram of the processing in the product information providing system. FIG. 5 is another explanatory diagram of the processing in the product information providing system. FIG. 6 is another explanatory diagram of the processing in the product information providing system. FIG. 7 is another explanatory diagram of the processing in the product information providing system. FIG. 8 is a flowchart of the operation of the product information providing system.

1. Embodiment 1.1 Overview FIG. 1 shows a product information providing system 10. The product information providing system 10 includes a character information extracting unit F12, a diagram information extracting unit F13, and a product information obtaining unit F14. The character information extraction unit F12 converts the character information L11 to L13, L21 to L23, L31 to L35, L41 to L43, L51 to L53, L61 to L63, L61 to L63 (see FIG. Extraction). The diagram information extraction unit F13 extracts the diagram information I10 to I60 (see FIG. 5) regarding the product from the image G10. The merchandise information acquisition unit F14 performs merchandise information P10 to P60 (FIG. 7) corresponding to the merchandise based on at least one of the character information extracted by the character information extraction unit F12 and the diagram information extracted by the diagram information extraction unit F13. See).

  In the product information providing system 10, not only character information about the product but also figure information about the product can be extracted from the image G10 including the description of the product. Then, product information corresponding to the product is obtained based on at least one of the character information and the diagram information. Therefore, according to the product information providing system 10, it is possible to provide the product information P10 to P60 corresponding to the product from the description of the product. In particular, in order to obtain product information, it is not necessary to prepare identification information on the product separately from the description of the product. Therefore, the merchandise information providing system 10 can acquire merchandise information based on a general-purpose image instead of a special image.

1.2 Details Hereinafter, the product information providing system 10 of the present embodiment will be described in more detail with reference to FIGS. The product information providing system 10 of the present embodiment is a system for providing product information based on an image (product image) including a description of a product. Here, "commodity" is an article to be purchased or sold. Commodities can include not only products but also natural resources such as crops and marine products. The “description of the product” is information describing the product (for example, explaining the shape and properties of the product) and may include visual information such as characters and drawings.

  FIG. 2 shows an example of an image including a description of a product. An image G10 illustrated in FIG. 2 is an image of a list of products including descriptions of a plurality of products. The image G10 is an image of a list of the lighting fixtures installed in the building. In the present embodiment, the product is a lighting fixture, and the description of the product includes a figure, which is a type of external view of the product. As shown in FIG. 2, the image G10 includes a plurality (six in FIG. 2) of columns R10, R20, R30, R40, R50, and R60 in each of which a product is described.

  FIG. 7 shows an example of an image including product information. The image G20 shown in FIG. 7 shows the product list W10. The product list W10 is a list of product information. The product list W10 is a list of a plurality (six in FIG. 7) of product information P10, P20, P30, P40, P50, and P60. Examples of product information include a product number, a product name, a price, a dimension, a mass, a specification, and notes. In this embodiment, the product information includes a product number and a product name, as shown in FIG.

  The product information providing system 10 can be connected to the recognition system 20, the product search system 30, the product estimation system 40, and the terminal device 50 via the communication network 60. The product information providing system 10 forms a system together with the recognition system 20, the product search system 30, the product estimation system 40, and the terminal device 50 via the communication network 60.

  Communication network 60 may include the Internet. The communication network 60 can be composed of not only a network conforming to a single communication protocol but also a plurality of networks conforming to different communication protocols. The communication protocol may be selected from various well-known wired and wireless communication standards. Although simplified in FIG. 1, the communication network 60 may include data communication devices such as repeater hubs, switching hubs, bridges, gateways, routers, and the like.

  The terminal device 50 is used for inputting the image G10 to the product information providing system 10 and displaying product information from the product information providing system 10. The terminal device 50 includes an input / output unit 51, a communication unit 52, and a processing unit 53, as shown in FIG. The terminal device 50 is an information terminal, for example, a desktop computer. Note that the terminal device 50 can be realized by a laptop computer or a mobile terminal (smartphone, tablet terminal, wearable terminal, or the like).

  The input / output unit 51 includes an input device for operating the terminal device 50. Input devices may include, for example, a keyboard, a mouse, a trackball, a touchpad, and the like. The input / output unit 51 includes an image display device for displaying information. The image display device may include a thin display device such as a liquid crystal display and an organic EL display.

  The communication unit 52 is a communication interface. The communication unit 52 is connectable to the communication network 60 and has a function of performing communication through the communication network 60. The communication unit 52 complies with a predetermined communication protocol. The predetermined communication protocol may be selected from various known wired and wireless communication standards.

  The processing unit 53 is configured to control the entire terminal device 50, that is, the input / output unit 51 and the communication unit 52. The processing unit 53 can be realized by, for example, a computer system including one or more processors (microprocessors) and one or more memories. That is, the one or more processors function as the processing unit 53 by executing one or more programs (applications) stored in one or more memories. The program is recorded in the memory of the processing unit 53 in advance here, but may be provided through an electric communication line such as the Internet, or recorded on a non-temporary recording medium such as a memory card.

  The processing unit 53 has a function of transmitting the image G10 input by the user to the product information providing system 10 through the communication network 60 in accordance with the operation of the input / output unit 51 described above.

  As shown in FIG. 3, the processing unit 53 displays the image G10 input by the user on the image display device of the input / output unit 51, and inputs / outputs an area (cell) where a product is described in the image G10. It has a function to be specified by the input device of the unit 51. In FIG. 3, the region C10 is set for the column R10, but the region may be set for any of the columns. When an area is set, the processing unit 53 has a function of transmitting area information to the product information providing system 10 via the communication network 60. The area information may include the size of the area and the position on the image.

  Further, the processing unit 53 has a function of allowing a user to select a candidate for product information when there are a plurality of candidates for product information for an area (see FIG. 7). This will be described later.

  The product information providing system 10 includes a communication unit 11 and a processing unit 12, as shown in FIG. The product information providing system 10 can be realized by a server.

  The communication unit 11 is a communication interface. The communication unit 11 is connectable to the communication network 60 and has a function of performing communication through the communication network 60. The communication unit 11 conforms to a predetermined communication protocol. The predetermined communication protocol may be selected from various known wired and wireless communication standards.

  The processing unit 12 is a control circuit that controls the operation of the product information providing system 10. The processing unit 12 can be realized by, for example, a computer system including one or more processors (microprocessors) and one or more memories. In other words, one or more processors function as the processing unit 12 by executing one or more programs (applications) stored in one or more memories. Here, the program is recorded in the memory of the processing unit 12 in advance, but may be provided through an electric communication line such as the Internet, or recorded on a non-temporary recording medium such as a memory card.

  As illustrated in FIG. 1, the processing unit 12 includes an area setting unit F11, a character information extracting unit F12, a diagram information extracting unit F13, a product information obtaining unit F14, and a product list creating unit F15. . In FIG. 1, the area setting unit F11, the character information extracting unit F12, the diagram information extracting unit F13, the product information obtaining unit F14, and the product list creating unit F15 do not show a substantial configuration. 2 illustrates functions implemented by the processing unit 12.

  The region setting unit F11 sets a plurality of regions respectively corresponding to descriptions of a plurality of products in the image G10. More specifically, upon receiving the image G10 and the information on the area from the terminal device 50, the area setting unit F11 sets a plurality of areas respectively corresponding to the descriptions of the plurality of products in the image G10. For example, as shown in FIG. 4, the region setting unit F11 sets a plurality (six in FIG. 4) of regions C10 to C60 so as to correspond to the columns R10 to R60 in which the description of the product is respectively provided. The region setting unit F11 extracts the columns R10 to R60 from the image G10 based on the information on the region, and sets the regions C10 to C60 in the columns R10 to R60, respectively. As an example, the area setting unit F11 extracts a contour candidate forming a column from the image G10 by edge detection. Since the columns in the image G10 are arranged in a matrix, the region corresponding to each column can be estimated based on the region information (region size and region position) from the terminal device 50. Therefore, the region setting unit F11 selects the contour forming the column from the candidates for the contour forming the column so as to match the information of the region (the size of the region and the position of the region). As a result, the region setting unit F11 sets the regions C10 to C60 in the columns R10 to R60, respectively, as shown in FIG.

  The character information extracting unit F12 extracts character information relating to the product from the image including the description of the product. When a plurality of areas are set in the area setting unit F11, the character information extracting unit F12 extracts character information for each of the plurality of areas. The character information extraction unit F12 gives the character information extracted from the image to the product information acquisition unit F14.

  The diagram information extraction unit F13 extracts diagram information about the product from the image including the description of the product. When a plurality of regions are set in the region setting unit F11, the diagram information extracting unit F13 extracts diagram information for each of the plurality of regions. The figure information extraction unit F13 provides the figure information extracted from the image to the product information acquisition unit F14.

  In the present embodiment, the character information extracting unit F12 and the diagram information extracting unit F13 extract character information using the recognition system 20.

  The recognition system 20 is a system that recognizes a character and a figure from an image. The recognition system 20 can be realized by a conventionally known image processing technique. As an example, the recognition system 20 reads characters from an image by optical character recognition (Optical Character Recognition). The recognition system 20 reads, as a diagram, an area that cannot be read as a character in the image. Such a recognition system 20 can be realized by a server.

  The character information extraction unit F12 acquires character information from the result of inputting the image G10 to the recognition system 20. In the present embodiment, the character information extraction unit F12 extracts a character string relating to details of a product as character information. That is, the character information extraction unit F12 ignores character strings that are not related to the details of the product. For example, the character information extraction unit F12 may treat a character string whose number of characters is equal to or less than a predetermined value (for example, 2) as a character string that is not related to details of a product. Since a character string having a small number of characters is not considered to contribute to the specification of the product information, it may be treated as a character string having no relation to the details of the product. The details of the product may include a product number and a product specification. The character information extracting unit F12 may determine whether the character information is a part number. As an example, the character information extracting unit F12 determines character information that is a character string of alphanumeric characters having a predetermined number of digits or more as part number information. The character information extraction unit F12 may handle character information on product specifications as supplementary information. Note that the character string extracted as the supplementary information may be a character string including a keyword related to the product specification. In the case of a lighting fixture, examples of keywords include a keyword related to shape, a keyword related to brightness, a keyword related to light color, and other keywords. Examples of keywords related to shape include “directly attached”, “embedded”, “without shade”, “with shade”, “square”, “bottom”, “panel”, “louver”, “open”, “guard” , "Fuji type", "Single reflection", "Round type". Examples of keywords relating to brightness include a symbol indicating brightness and a wattage. Examples of keywords relating to light color include “daylight color”, “lunch white”, “white”, “warm white”, and “bulb color”. Examples of other keywords include “moisture proof”, “waterproof”, “rainproof”, “battery”, “dimming”, “emergency”, “power”, and “battery”. Preferably, these keywords are suitable for a search in the product search system 30. Further, the keyword may be appropriately changed depending on the type of the product. When the extracted character information includes a predetermined keyword, the character information extraction unit F12 may determine the extracted character information as supplementary information. The character information extraction unit F12 gives the character information extracted from the image to the product information acquisition unit F14.

  The figure information extraction unit F13 acquires figure information from the result of inputting the image G10 to the recognition system 20. In the present embodiment, the diagram information extracting unit F13 extracts a diagram of the appearance of the product (the figure in the present embodiment) as diagram information. That is, the diagram information extraction unit F13 ignores a diagram that is not related to the appearance of the product. For example, the figure information extracting unit F13 may treat the figure information whose size is equal to or smaller than a predetermined value as the figure information that is not related to the appearance of the product. In the figure list, since the figure of the appearance of the product is drawn in a certain size, the figure information having a small size may be treated as figure information that is not related to the appearance of the product.

  FIG. 5 shows the results of the extraction by the character information extraction unit F12 and the diagram information extraction unit F13. In FIG. 5, six regions C10 to C60 are set in the image G10.

  Three pieces of character information L11, L12, and L13 and one piece of figure information I10 are extracted from the area C10. The character information L11 is a character string “LED embedded panel” and indicates the type of product. The character information L12 is a character string “56W 450 square” and indicates the rated power and size of the product. The character information L13 is a character string “XL575PFVJLA9” and indicates a product number. Character information L13 is part number information, and character information L11 and L12 are supplementary information. The figure information I10 is a figure and shows an LED embedded panel.

  From the area C20, three pieces of character information L21, L22, L23 and one piece of figure information I20 are extracted. The character information L21 is a character string “LED embedded panel” and indicates the type of product. The character information L22 is a character string “43W 450 square” and indicates the rated power and size of the product. The character information L23 is a character string “XL574PFVJLA9” and indicates a product number. Character information L23 is part number information, and character information L21 and L22 are supplementary information. The figure information I20 is a figure and shows an LED embedded panel.

  From the area C30, five pieces of character information L31, L32, L33, L34, L35 and one piece of figure information I30 are extracted. The character information L31 is a character string “LED embedded louver” and indicates the type of product. The character information L32 is a character string “LDL40 × 2 (3800Lm)” and indicates the light source of the product. The character information L33 is a character string “power consumption 64 W” and indicates the power consumption of the product. The character information L34 is a character string “multi-comfort 15” and indicates the type of product. The character information L35 is a character string “XFL323CBLE9” and indicates a product number. Character information L35 is part number information, and character information L31 to L34 is supplementary information. The figure information I30 is a figure and shows the LED embedded louver.

  From the area C40, three pieces of character information L41, L42, L43 and one piece of figure information I40 are extracted. The character information L41 is a character string “LED bracket” and indicates the type of product. The character information L42 is a character string “equivalent to one 20-type straight tube fluorescent lamp” and indicates the brightness of the product. The character information L43 is a character string “LGB85039LE1” and indicates a product number. Character information L43 is part number information, and character information L41 and L42 are supplementary information. The figure information I40 is a figure and shows an LED bracket.

  From the area C50, three pieces of character information L51, L52, L53 and one piece of figure information I50 are extracted. The character information L51 is a character string “LED Fuji type” and indicates the type of product. The character information L52 is a character string “LDL40 × 2 (2600Lm)” and indicates the light source of the product. The character information L53 is a character string “NNF42001LE9” and indicates the product number. Character information L53 is part number information, and character information L51 and L52 are supplementary information. The figure information I50 is a figure and shows a Fuji-type LED lighting fixture.

  From the area C60, three pieces of character information L61, L62, L63 and one figure information I60 are extracted. The character information L61 is a character string “LED direct-attached type” and indicates the type of product. The character information L62 is a character string “32.5W iD series direct attachment type” and indicates the rated power and the type of the product. The character information L63 is a character string “W150 × L1250” and indicates the size of the product. Each of the character information L61 to L63 is supplementary information. The figure information I60 is a figure and shows a direct-attached LED lighting fixture.

  The merchandise information acquisition unit F14 obtains merchandise information corresponding to the merchandise based on at least one of the character information L11 to L63 extracted by the character information extraction unit F12 and the diagram information I10 to I60 extracted by the diagram information extraction unit F13. P10 to P60 (see FIG. 7) are acquired.

  More specifically, the product information acquisition unit F14 has a function of performing a first process and a second process.

  The first process is the product information P10 corresponding to the product using the character information L11 to L13, L21 to L23, L31 to L35, L41 to L43, L51 to L53, and L61 to L63 extracted by the character information extraction unit F12. This is the process of acquiring P60. The product information acquisition unit F14 uses the product search system 30 in the first processing. The product search system 30 is configured to output product information in response to input of one or more character strings. The product search system 30 has a database of product information. The product search system 30 searches a database of product information with the input one or more character strings, and outputs product information to which the input one or more character strings apply. If the character string input to the product search system 30 does not include a character string sufficient to specify the product information, the product search system 30 cannot sufficiently narrow down the product information, and thus cannot specify the product information. . In this case, the product search system 30 may output a notification that the product information cannot be specified. The product search system 30 can be configured by a conventionally known database construction technique, and can be realized by a server.

  In the first process, the product information acquisition unit F14 acquires the product information corresponding to the product from the product search system 30 by giving the character information extracted by the character information extraction unit F12 to the product search system 30. In particular, in the first process, the product information acquisition unit F14 acquires the product information corresponding to the product from the product search system 30 by providing the product search system 30 with the product number information extracted by the character information extraction unit F12. If the character information extracting unit F12 cannot extract the product number information, the product information acquiring unit F14 may not be able to acquire the product information corresponding to the product in the first process.

  The second process is a process of acquiring product information P10 to P60 corresponding to a product using the diagram information I10 to I60 extracted by the diagram information extracting unit F13. In the second processing, the product information acquisition unit F14 uses the product search system 30 and the product estimation system 40. The product estimation system 40 is configured to output the type of the product in response to the input of the diagram information. As an example, if the figure information is the figure of the lighting fixture, the type of the lighting fixture corresponding to the figure may be associated. Examples of types of lighting fixtures include base lights, down lights, ceiling lights, guide lights, and emergency lights.

  The product estimation system 40 uses a learned model (product information estimation model). The product information estimation model is a learned model that has learned the relationship between the diagram information and the type of product. As an example, such a product information estimation model uses an artificial intelligence program (algorithm) to provide a program (algorithm) for the appearance of a product using a learning data set including a diagram of the product appearance and a product type corresponding to the product appearance. It can be generated by learning the relationship between the figure and the type of product. The artificial intelligence program is a machine learning model, for example, a neural network that is a type of hierarchical model. That is, a learned model is generated by causing the neural network to perform machine learning (for example, deep learning) using a learning data set. The product estimation system 40 can be realized by a server.

  In the second process, the product information acquisition unit F14 uses the diagram information extracted by the diagram information extraction unit F13 as the product estimation system 40 (that is, a learned model that has learned the relationship between the product appearance diagram and the product type). , The type of the product corresponding to the diagram information is specified. Then, the product information obtaining unit F14 obtains the product information by giving the specified product type to the product search system 30. At this time, if there is supplementary information, it is provided to the product search system 30 together with the type of product obtained from the diagram information. By using the supplementary information, the product information can be narrowed down, and the acquisition of the product information can be performed with high accuracy.

  The product information acquisition unit F14 executes the first process before the second process. Then, if the product information corresponding to the product can be obtained in the first process, the product information obtaining unit F14 does not execute the second process. In other words, the product information acquisition unit F14 executes the second process when the product information corresponding to the product cannot be obtained in the first process. As a result, an improvement in the efficiency of the process of acquiring the product information corresponding to the product from the product description can be expected. Further, when the extraction of the character information by the character information extraction unit F12 fails, the product information acquisition unit F14 executes the second process. Here, if the extraction of the diagram information by the diagram information extraction unit F13 also fails, the product information acquisition unit F14 may determine that there is no description of the product.

  In the case of the area C10, the product information acquisition unit F14 acquires the product information corresponding to the product from the product search system 30 by giving the three pieces of character information L11, L12, and L13 to the product search system 30. In the area C10, the character information L13 indicates the product number, so the product search system 30 outputs the product information P10 (see FIG. 7) specified by the product number. The product information P10 includes a product number of “XL575PFVJLA9” and a product name of “ceiling embedded LED (lunch white) integrated LED base light”.

  In the case of the area C20, the product information acquisition unit F14 acquires the product information corresponding to the product from the product search system 30 by giving the three pieces of character information L21, L22, and L23 to the product search system 30. In the area C20, the character information L23 indicates the product number, so the product search system 30 outputs the product information P20 (see FIG. 7) specified by the product number. The product information P20 includes a product number of “XL574PFVJLA9” and a product name of “Ceiling recessed LED (lunch white) integrated LED base light”.

  In the case of the area C30, the product information acquisition unit F14 acquires the product information corresponding to the product from the product search system 30 by giving the five pieces of character information L31, L32, L33, L34, and L35 to the product search system 30. In the area C30, the character information L35 indicates the product number, so the product search system 30 outputs the product information P30 (see FIG. 7) specified by the product number. The product information P30 includes a product number of “XFL323CBLE9” and a product name of “ceiling embedded type 40 type straight tube LED lamp base light”.

  In the case of the area C40, the product information acquisition unit F14 acquires the product information corresponding to the product from the product search system 30 by giving the three pieces of character information L41, L42, and L43 to the product search system 30. In the area C40, since the character information L43 indicates the product number, the product search system 30 outputs the product information P40 (see FIG. 7) specified by the product number. The product information P40 includes a product number of “LGB85039LE1” and a product name of “ceiling-mounted / wall-mounted LED (bulb color) mirror light”.

  In the case of the area C50, the product information acquisition unit F14 acquires the product information corresponding to the product from the product search system 30 by giving the three pieces of character information L51, L52, and L53 to the product search system 30. In the area C50, the character information L53 indicates the product number, so the product search system 30 outputs the product information P50 (see FIG. 7) specified by the product number. The product information P50 includes a product number of “NNF42001LE9” and a product name of “ceiling-mounted type 40 straight tube LED lamp base light”.

  In the case of the area C60, the product information acquisition unit F14 gives the three pieces of character information L61, L62, and L63 to the product search system 30. If the three pieces of character information L61, L62, and L63 do not include a character string sufficient to specify the product information, the product search system 30 is notified that the product information cannot be specified. In this case, the product information acquisition unit F14 determines that the extraction of the character information by the character information extraction unit F12 has failed, and executes the second process. That is, the product information acquisition unit F14 provides the diagram information I60 to the product estimation system 40. The product estimation system 40 outputs product information P60 for the diagram information I60. Here, the product information P60 includes a plurality of product information candidates D11 to D13. The product information candidates D11 to D13 are indistinguishable in the appearance diagram of the product, but have different electrical performances. The product number of the product information candidate D11 is “XLX450AENZLE9”, the product information candidate D12 is “XLX450AENZRZ9”, and the product information candidate D13 is “XLX450AENZLR9”. In FIG. 7, the content of the candidate D11 is represented in the product information P60.

  FIG. 6 shows a result of obtaining the product information by the product information obtaining unit F14. In the image G10 shown in FIG. 6, in the regions C10 to C50, the product information is determined to be one. On the other hand, the area C61 corresponds to the area C60 in FIG. 3, but is displayed differently from the other areas C10 to C50 because there is a candidate for product information. The areas C10 to C50 and C61 are provided with labels N10 to N60 indicating the identification numbers of the areas.

  The product list creation unit F15 creates a product list for each of the plurality of areas based on the product information acquired by the product information acquisition unit F14. FIG. 7 is an example of an image including a product list created by the product list creation unit F15. The image G20 illustrated in FIG. 7 includes a product list W10 and a candidate list W20. The product list W10 includes product information P10 to P60 obtained by the product information obtaining unit F14 for each of the plurality of regions C10 to C60. In FIG. 7, the product information P10 to P60 are displayed together with the identification numbers of the corresponding areas (cell numbers in FIG. 7).

  The product information providing system 10 transmits the output of the product information acquisition unit F14 (result of acquisition of the product information) and the output of the product list creation unit F15 (information of the product list W10 and the candidate list W20) to the terminal device via the communication network 60. Send to 50.

  In this case, the terminal device 50 may cause the image display device of the input / output unit 51 to display the image G10 illustrated in FIG. From the image G10 in FIG. 6, it is possible to confirm whether the diagram information is correctly extracted. In the terminal device 50, the processing unit 53 may cause the image display device of the input / output unit 51 to display the image G20. Thereby, it is possible to confirm the details of the product information. Here, in the candidate list W20, a button B10 for selecting the product information candidates D11 to D13 is displayed. By operating the input device of the input / output unit 51 of the terminal device 50 and selecting a desired button B10, the user can determine the product information corresponding to the area C60 from among the product information candidates D11 to D13. .

1.3 Operation Next, the operation of the product information providing system 10 will be briefly described with reference to FIG.

  First, the user uses the terminal device 50 to input data of an image G10 including a description of a product as shown in FIG. 2 (S11). Further, the user uses the terminal device 50 to specify a cell (here, the area C10) (S12) as shown in FIG.

  Thereby, in the product information providing system 10, the area setting unit F11 sets the cells (areas) C10 to C60 for the image G10 as shown in FIG. 4 (S13). Next, in the product information providing system 10, as shown in FIG. 5, the character information extracting unit F12 extracts the character information L11 to L63 using the recognition system 20, and the figure information extracting unit F13 uses the recognition system 20. Then, the diagram information I10 to I60 are extracted (S14).

  The product information acquisition unit F14 performs a process for acquiring corresponding product information for each of the plurality of cells. First, the product information acquisition unit F14 determines whether or not there is character information in a cell from which product information is to be obtained (S15).

  If the character information extraction unit F12 can extract the character information (S15; Yes), the product information acquisition unit F14 gives the character information extracted by the character information extraction unit F12 to the product search system 30, and outputs the character information. To search for a product (S16). If the product information (product number) is obtained from the product search system 30 (S17; Yes), the product information obtaining unit F14 determines whether the process for obtaining the product information has been completed for all the cells (areas). (S18). If the process for acquiring the product information has been completed for all cells (areas) (S18; Yes), the product list creation unit F15 creates a product list and outputs it to the terminal device 50, and the terminal device 50 A product list is displayed (S19). Thus, the operation of the product information providing system 10 ends. On the other hand, if the process for acquiring the product information has not been completed for all the cells (areas) (S18; No), the product information acquisition unit F14 also has the character information for the cell for which the process has not been completed. It is determined whether or not (S15).

  If character information cannot be extracted by the character information extracting unit F12 (S15; No), the product information obtaining unit F14 determines whether or not there is diagram information in a cell from which the product information is to be obtained (S20). Alternatively, when the product information (product number) is not obtained from the product search system 30 (S17; No), the product information acquisition unit F14 determines whether or not the cell for which the product information is to be acquired has diagram information. (S20).

  When the diagram information can be extracted by the diagram information extraction unit F13 (S20; Yes), the product information acquisition unit F14 gives the diagram information extracted by the diagram information extraction unit F13 to the product estimation system 40, and Is used to estimate the product (S21). If the product information (product number) is obtained from the product estimation system 40 (S22; Yes), the product information acquisition unit F14 determines whether the process for acquiring the product information has been completed for all the cells (areas). (S18).

  When the diagram information cannot be extracted by the diagram information extraction unit F13 (S20; No), the product information acquisition unit F14 concludes that the product information (part number) cannot be specified for the cell from which the product information is to be obtained. (S23). Alternatively, when the product information (product number) is not obtained from the product estimation system 40 (S22; No), the product information acquisition unit F14 outputs the product information (product number) to the cell from which the product information is to be acquired. It is concluded that it cannot be specified (S23). Then, it is determined whether or not the processing for acquiring the product information has been completed for all the cells (areas) (S18).

1.4 Summary The product information providing system 10 described above includes a character information extraction unit F12, a diagram information extraction unit F13, and a product information acquisition unit F14. The character information extraction unit F12 converts the character information L11 to L13, L21 to L23, L31 to L35, L41 to L43, L51 to L53, L61 to L63, L61 to L63 (see FIG. Extraction). The diagram information extraction unit F13 extracts the diagram information I10, I20, I30, I40, I50, and I60 (see FIG. 5) on the product from the image G10. The merchandise information acquisition unit F14, based on at least one of the text information extracted by the text information extraction unit F12 and the graphic information extracted by the graphic information extraction unit F13, product information P10, P20, P30, P40, P50, and P60 (see FIG. 7) are acquired. As described above, according to the product information providing system 10, it is possible to provide the product information corresponding to the product from the description of the product.

  In other words, it can be said that the product information providing system 10 executes the following method (product information providing method). The method of providing product information is based on the character information L11 to L13, L21 to L23, L31 to L35, L41 to L43, L51 to L53, L61 to L63 (see FIG. 5) on the product from the image G10 (see FIG. 2) including the description of the product. ) Extraction. The merchandise information providing method extracts figure information I10, I20, I30, I40, I50, and I60 (see FIG. 5) on the merchandise from the image G10. The merchandise information providing method includes merchandise information P10, P20, P30, P40 corresponding to the merchandise based on at least one of the character information extracted by the character information extracting unit F12 and the graphic information extracted by the graphic information extracting unit F13. , P50, P60 (see FIG. 7). As described above, according to the product information providing method, similarly to the product information providing system 10, it is possible to provide the product information corresponding to the product from the description of the product.

  The product information providing system 10 is realized by a computer system (one or more processors). That is, the product information providing system 10 is realized by one or more processors executing a program (computer program). This program is a program for causing one or more processors to execute the product providing method. According to such a program, the product information corresponding to the product can be provided from the description of the product, similarly to the product information providing method.

2. Modifications Embodiments of the present disclosure are not limited to the above embodiments. The above embodiment can be variously modified according to the design and the like as long as the object of the present disclosure can be achieved. Hereinafter, modified examples of the above embodiment will be listed.

  In the above embodiment, the product is a lighting fixture, but is not limited to this. The merchandise may be an article to be bought and sold, and may include not only electric appliances such as lighting equipment but also products such as food and medicine. In addition, commodities can also include natural resources such as crops and marine products. Examples of the electric products include sensor-related equipment, air-conditioning-related equipment, housing / equipment-related equipment, cooking / housework-related equipment, health care-related equipment, management / operation-related equipment, and AV-related equipment. Examples of the sensor-related devices include a camera, a fire sensor, a human body detection sensor, a temperature sensor, a CO2 sensor, and a current sensor. Examples of air conditioning related equipment include air conditioners, fans, ventilation fans, air purifiers, hot carpets, and oil fan heaters. Examples of equipment related to houses and facilities include electric blinds, electric curtains, water heaters, electric locks, smart meters, solar power generation equipment, storage batteries, fuel cells, general lighting, single-function lighting, and emergency lights. Examples of cooking / housework-related equipment include an IH cooker (cooking heater), a microwave oven, a dishwasher, a dish dryer, a washing machine, a clothes dryer, and an automatic vacuum cleaner. Examples of health management-related devices include a weight scale, a body fat scale, a thermometer, a blood pressure meter, and a blood glucose meter. Examples of the management / operation-related devices include a controller and a switch (HA device). Examples of the AV-related equipment include a TV and a display.

  In one variation, the external view of the product may be a figure other than the figure, for example, a front view, a rear view, a plan view, a bottom view, a left side view, a right side view, a perspective view, a cross-sectional view, and the like. It may be. However, it is preferable that the figure representing the product be a commonly used figure.

  In a modified example, the product information providing system 10 may include a character information extracting unit F12, a diagram information extracting unit F13, and a product information obtaining unit F14.

  In a modified example, the area setting unit F11 may set an area by image processing even if there is no area information from the terminal device 50.

  In a modified example, the character information extraction unit F12 may have the function of the recognition system 20. In other words, the product information providing system 10 can include the recognition system 20.

  In a modification, the diagram information extracting unit F13 may have the function of the recognition system 20. In other words, the product information providing system 10 can include the recognition system 20.

  In one modification, the product information acquisition unit F14 may have a function of the product search system 30. In other words, the product information providing system 10 can include the product search system 30.

  In one modification, the product information acquisition unit F14 may have a function of the product estimation system 40. In other words, the product information providing system 10 can include the product estimation system 40.

  In one variation, the product estimation system 40 may use a learned model that has learned the relationship between a diagram of the appearance of a product and one or more pieces of product information. Products may have the same appearance but different characteristics. Examples of such products include products having the same appearance but different power consumption and usable light sources. Since the diagram information is a diagram of the appearance of the product, the product estimation system 40 may be configured to output one or more product information in response to the input of the diagram information. As an example, such a product information estimation model includes a learning data set including a diagram of the appearance of a product and one or more pieces of product information corresponding to the appearance of the product. It can be generated by learning the relationship between the appearance diagram and the product information. The artificial intelligence program is a machine learning model, for example, a neural network that is a type of hierarchical model. That is, a learned model is generated by causing the neural network to perform machine learning (for example, deep learning) using a learning data set. The product estimation system 40 can be realized by a server.

  In a modification, the product information acquisition unit F14 may execute the first processing and the second processing in parallel, or may execute the second processing before the first processing. That is, the product information acquisition unit F14 may perform the first process when the product information cannot be obtained in the second process. Further, the product information acquisition unit F14 may specify the product information by combining the character information and the diagram information. In particular, when the character information does not include a product number, the product information acquisition unit F14 obtains the product information by performing a logical product of the product information candidate acquired by the character information and the product information candidate acquired by the diagram information. May be specified. Alternatively, the product information obtaining unit F14 may specify the product information by narrowing down the product information candidates obtained from the diagram information by character information.

  In a modified example, the product list creation unit F15 may create a product list by extracting only specific items without including all items included in the product information in the product list. Examples of specific items include a product number and a price. Thereby, the product list creation unit F15 can create a product list that can be used as an estimate.

  In one modification, the product information providing system 10 may be configured by a plurality of computers. For example, the functions of the product information providing system 10 (particularly, the area setting unit F11, the character information extraction unit F12, the diagram information extraction unit F13, the product information acquisition unit F14, and the product list creation unit F15) are distributed to a plurality of devices. You may. Further, at least a part of the functions of the product information providing system 10 may be realized by, for example, a cloud (cloud computing).

  The execution entity of the product information providing system 10 described above includes a computer system. The computer system has a processor and a memory as hardware. When the processor executes the program recorded in the memory of the computer system, the function as the execution entity of the product information providing system 10 according to the present disclosure is realized. The program may be recorded in a memory of the computer system in advance, or may be provided through a telecommunication line. Further, the program may be provided by being recorded on a non-transitory recording medium such as a memory card, an optical disk, or a hard disk drive that can be read by a computer system. A processor of a computer system includes one or more electronic circuits including a semiconductor integrated circuit (IC) or a large-scale integrated circuit (LSI). Field programmable gate array (FGPA), ASIC (application specific integrated circuit), or reconfigurable, which can be programmed after the manufacture of the LSI, to enable the reconfiguration of junction relations inside the LSI or the setup of circuit sections inside the LSI Logical devices can be used for the same purpose. The plurality of electronic circuits may be integrated on one chip, or may be provided separately on a plurality of chips. The plurality of chips may be integrated in one device, or may be provided separately in a plurality of devices.

3. Aspects As is apparent from the above embodiments and modifications, the present disclosure includes the following aspects. In the following, reference numerals are given in parentheses only for clarifying the correspondence with the embodiment.

  A first aspect is a product information providing system (10), which includes a character information extraction unit (F12), a diagram information extraction unit (F13), and a product information acquisition unit (F14). The character information extraction unit (F12) extracts character information (L11 to L63) related to the product from the image (G10) including the description of the product. The diagram information extraction unit (F13) extracts diagram information (I10 to I60) related to the product from the image (G10). The product information acquisition unit (F14) includes the character information (L11 to L63) extracted by the character information extraction unit (F12) and the figure information (I10 to I60) extracted by the figure information extraction unit (F13). ) Is obtained based on at least one of the items (P10 to P60). According to this aspect, product information (P10 to P60) corresponding to the product can be provided from the description of the product.

  A second aspect is based on the merchandise information providing system (10) of the first aspect. In a second aspect, the product information acquisition unit (F14) has a function of performing a first process and a second process. The first process is a process of acquiring product information (P10 to P60) corresponding to the product using the character information (L11 to L63) extracted by the character information extracting unit (F12). The second process is a process of acquiring product information (P10 to P60) corresponding to the product using the diagram information (I10 to I60) extracted by the diagram information extracting unit (F13). According to this aspect, product information (P10 to P60) corresponding to the product can be provided from the description of the product.

  A third aspect is based on the merchandise information providing system (10) of the second aspect. In a third aspect, the product information acquisition unit (F14) executes the first processing before the second processing. According to this aspect, an improvement in the efficiency of the process of acquiring the product information corresponding to the product from the description of the product can be expected.

  A fourth aspect is based on the merchandise information providing system (10) of the third aspect. In a fourth aspect, the product information acquisition unit (F14) does not execute the second process when the product information (P10 to P60) corresponding to the product can be acquired in the first process. According to this aspect, an improvement in the efficiency of the process of acquiring the product information corresponding to the product from the description of the product can be expected.

  A fifth aspect is based on the merchandise information providing system (10) according to the third or fourth aspect. In a fifth aspect, the product information acquisition unit (F14) executes the second process when the extraction of the character information (L11 to L63) by the character information extraction unit (F12) fails. According to this aspect, an improvement in the efficiency of the process of acquiring the product information corresponding to the product from the description of the product can be expected.

  A sixth aspect is based on the merchandise information providing system (10) according to any one of the first to fifth aspects. In a sixth aspect, the character information extraction unit (F12) extracts a character string relating to details of the product as the character information (L11 to L63). According to this aspect, the accuracy of obtaining the product information (P10 to P60) can be improved.

  A seventh aspect is based on the merchandise information providing system (10) of the sixth aspect. In a seventh aspect, the product information acquisition unit (F14) converts the character information (L11 to L63) extracted by the character information extraction unit (F12) into input of one or more character strings. By giving (P10 to P60) to the output product search system (30), the product information (P10 to P60) corresponding to the product is acquired from the product search system (30). According to this aspect, the accuracy of obtaining the product information (P10 to P60) can be improved.

  An eighth aspect is based on the merchandise information providing system (10) according to any one of the first to seventh aspects. In an eighth aspect, the diagram information extraction unit (F13) extracts a diagram of the appearance of the product as the diagram information (I10 to I60). According to this aspect, the accuracy of obtaining the product information (P10 to P60) can be improved.

  A ninth aspect is based on the merchandise information providing system (10) of the eighth aspect. In a ninth aspect, the product information acquisition unit (F14) uses the learned model (40) that has learned the relationship between the appearance diagram of the product and the type of the product, and uses the learned information model (40). The product information (P10 to P60) corresponding to the product is obtained based on the diagram information (I10 to I60) extracted in (1). According to this aspect, the accuracy of obtaining the product information (P10 to P60) can be improved.

  A tenth aspect is based on the product information providing system (10) according to any one of the first to ninth aspects. In the tenth aspect, the image (G10) is an image of a product list including descriptions of a plurality of products. According to this aspect, a plurality of product information items (P10 to P60) can be provided at one time.

  An eleventh aspect is based on the product information providing system (10) of the tenth aspect. In an eleventh aspect, the product information providing system (10) further includes an area setting unit (F11) for setting a plurality of areas (C10 to C60) respectively corresponding to descriptions of the plurality of goods in the image (G10). Prepare. The character information extraction unit (F12) extracts the character information (L11 to L63) for each of the plurality of regions (C10 to C60). The diagram information extraction unit (F13) extracts the diagram information (I10 to I60) for each of the plurality of regions (C10 to C60). The merchandise information acquisition unit (F14) is configured to extract the character information (L11 to L63) extracted by the character information extraction unit (F12) and the diagram information extraction for each of the plurality of regions (C10 to C60). The product information (P10 to P60) corresponding to the area is obtained using at least one of the diagram information (I10 to I60) extracted by the section (F13). According to this aspect, a plurality of product information items (P10 to P60) can be provided at one time.

  A twelfth aspect is based on the merchandise information providing system (10) of the eleventh aspect. In a twelfth aspect, a product list (W10) is created for each of the plurality of areas (C10 to C60) based on the product information (P10 to P60) acquired by the product information acquisition unit (F14). It further includes a product list creating unit (F15) to create. According to this aspect, a plurality of product information items (P10 to P60) can be provided at one time.

  A thirteenth aspect is based on the merchandise information providing system (10) according to any one of the first to twelfth aspects. In a thirteenth aspect, the product information (P10 to P60) includes a product number. According to this aspect, product information (P10 to P60) including the product number can be provided.

  A fourteenth aspect is a method for providing product information, wherein character information (L11 to L63) relating to the product is extracted from an image (G10) including a description of the product, and the image information (G10) is used to extract the character information related to the product. The figure information (I10 to I60) is extracted. The product information providing method obtains product information (P10 to P60) corresponding to the product using at least one of the character information (L11 to L63) and the diagram information (I10 to I60). According to this aspect, product information (P10 to P60) corresponding to the product can be provided from the description of the product.

  A fifteenth aspect is a program, which is a program for causing one or more processors to execute the product information providing method of the fourteenth aspect. According to this aspect, product information (P10 to P60) corresponding to the product can be provided from the description of the product.

10 Product information providing system F12 Character information extraction unit F13 Figure information extraction unit F14 Product information acquisition unit F15 Product list creation unit G10 Image (image including description of product)
30 product search system 40 product estimation system (learned model)
L11-L63 Character information I10-I60 Graphic information P10-P60 Product information C10-C60 area

Claims (15)

A character information extraction unit that extracts character information related to the product from an image including a description of the product,
A diagram information extraction unit that extracts diagram information on the product from the image;
A product information acquisition unit that acquires product information corresponding to the product based on at least one of the character information extracted by the character information extraction unit and the diagram information extracted by the diagram information extraction unit;
Comprising,
Product information providing system. The product information acquisition unit,
A first process of obtaining product information corresponding to the product using the character information extracted by the character information extraction unit;
A second process of acquiring product information corresponding to the product using the diagram information extracted by the diagram information extracting unit;
Having the function of performing
The product information providing system according to claim 1. The product information acquisition unit executes the first process before the second process.
The product information providing system according to claim 2. The product information acquisition unit does not execute the second process when the product information corresponding to the product can be obtained in the first process.
The product information providing system according to claim 3. The product information acquisition unit executes the second process when the extraction of the character information by the character information extraction unit fails.
The product information providing system according to claim 3 or 4. The character information extraction unit extracts a character string related to details of the product as the character information,
The product information providing system according to claim 1. The product information acquisition unit provides the character information extracted by the character information extraction unit to a product search system that outputs product information in response to input of one or more character strings. Get product information corresponding to the product,
The product information providing system according to claim 6. The diagram information extraction unit extracts a diagram of the appearance of the product as the diagram information,
The product information providing system according to claim 1. The product information acquisition unit uses the learned model that has learned the relationship between the appearance diagram of the product and the type of the product, and corresponds to the product based on the diagram information extracted by the diagram information extraction unit. Get product information,
The product information providing system according to claim 8. The image is an image of a list of products including descriptions of a plurality of products,
The product information providing system according to claim 1. The image processing apparatus further includes an area setting unit that sets a plurality of areas corresponding to descriptions of a plurality of products in the image,
The character information extraction unit, for each of the plurality of regions, performs the extraction of the character information,
The diagram information extraction unit, for each of the plurality of regions, performs the extraction of the diagram information,
The product information acquisition unit, for each of the plurality of regions, using at least one of the character information extracted by the character information extraction unit and the diagram information extracted by the diagram information extraction unit, Acquiring product information corresponding to the area,
The product information providing system according to claim 10. Based on the product information acquired by the product information acquisition unit for each of the plurality of regions, further comprising a product list creation unit that creates a product list,
The product information providing system according to claim 11. The product information includes a product number.
The product information providing system according to claim 1. From the image including the description of the product, extract the character information on the product,
From the image, perform the extraction of the diagram information on the product,
Using at least one of the character information and the diagram information, obtain product information corresponding to the product,
How to provide product information. Claims for causing one or more processors to execute the product information providing method according to claim 14:
program.

Images