WO2017199389A1

WO2017199389A1 - Information provision system, information provision method, and information provision program

Info

Publication number: WO2017199389A1
Application number: PCT/JP2016/064840
Authority: WO
Inventors: 冨井　隆春
Original assignee: 株式会社ａｍｕｓｅｏｎｅｓｅｌｆ
Priority date: 2016-05-19
Filing date: 2016-05-19
Publication date: 2017-11-23
Also published as: JPWO2017199389A1

Abstract

[Problem] To implement an information provision system which is capable of providing information to a user of a portable terminal with a method which is simple for the user to use while using diverse foodstuff information. [Solution] Provided is an information provision system in which a portable terminal 10 and a system server 20 are connected by the Internet 30. The portable terminal 10 comprises an OCR processing unit 16 and a foodstuff recognition unit 17 which carry out an analysis upon a photographic image which is photographed with the portable terminal 10 and extract parameters. The system server 20 comprises a database 22 which outputs information which corresponds to the extracted parameters. The information which is outputted from the database 22 is transmitted from the system server 20 to the portable terminal 10, and is displayed and presented to the user by the portable terminal 10.

Description

Information providing system, information providing method, and information providing program

The present invention relates to an information providing system, an information providing method, and an information providing program for providing information on ingredients and dishes using a mobile terminal such as a smartphone or a tablet.

Patent Document 1 discloses that a printed advertisement or catalog is photographed using a digital camera or the like provided in a mobile terminal, and a product is ordered using information on the photographed image. Patent Document 1 discloses a menu ordering method as a similar ordering service in which a camera is used for a restaurant menu and a customer directly transfers his / her order to the restaurant kitchen.

Special Table 2004-514225

Various media such as magazines, TV, and the Internet send out various information about food. Such information includes, for example, food information that is highly effective for beauty and health management, or food material that should be avoided for specific diseases and diseases. Due to the presence of such food material information, the consumer side is increasingly incorporating food material information into menu selections during meals.

In the food service industry such as restaurants, and the food service industry such as bento restaurants and side dishes restaurants, customers are increasingly providing calorie information on their menus. However, food information as described above is extremely diverse, and needs for food information vary from customer to customer. For this reason, it is almost impossible for the store providing the menu to provide all kinds of food information to customers. For this reason, there is a problem that it is difficult for consumers to make full use of various kinds of food information even if they try to select a menu with reference to the food information.

Patent Document 1 merely reduces the burden on restaurant staff by transferring customers' own orders to the restaurant kitchen, and can provide a menu selection method incorporating food information to the customers. Absent.

This invention is made | formed in view of the said subject, Realizing the information provision system which can provide information with a simple method for a user, utilizing a variety of foodstuff information with respect to the user of a portable terminal. With the goal.

In order to solve the above-described problems, the present invention provides an information providing system that provides information using a portable terminal including a photographing unit and a display unit, and analyzes a photographed image photographed by the photographing unit. The image analysis means for extracting the parameters by performing the process, the parameter extracted by the image analysis means as an input parameter, the database for outputting information corresponding to the input parameter, and the information output from the database for the display It is characterized by being displayed in the part.

According to the above configuration, when a user shoots a menu, food, or dish using the imaging unit of the mobile terminal, parameters are extracted from the captured image by image analysis processing, and information corresponding to the extracted parameters is stored in the database. Is output from. The output information is displayed on the display unit of the mobile terminal. As a result, the user can obtain various information simply by photographing the menu, ingredients, dishes, and the like.

In the information providing system, the portable terminal includes an input unit, and a disease name can be input from the input unit, and the image analysis unit performs OCR processing on the captured image, An OCR processing unit for converting a character portion in the photographed image into text data; and a food material recognition unit for extracting a portion corresponding to a food material name from the text data. The database includes a food material name and the food material. Is stored in association with the information on whether or not it is a restricted ingredient whose intake should be restricted for each disease, and the database includes the name of the ingredient extracted by the ingredient recognition unit and the input unit When the input disease name is input as an input parameter, it can be configured to output information on whether the input food is a restricted food for the input disease. .

According to the above configuration, when the user shoots a restaurant menu table or the like, the OCR processing unit and the food material recognition unit extract a portion corresponding to the food material name from the captured image. Moreover, in the portable terminal, the disease name that the user has can be input from the input unit. In the database, when the ingredient name and disease name extracted by the ingredient recognition unit are input as input parameters, information on whether or not the input ingredient is a restricted ingredient for the input disease is output. The Since this information is displayed on the mobile terminal side, the user can easily confirm the restricted foods for himself among the ingredients on the menu table.

In the information providing system, the image analysis unit includes an image analysis unit that performs image analysis processing on the captured image and recognizes / extracts foodstuffs reflected in the captured image. Is stored in association with the name of the dish and the ingredients used for the dish, and when the ingredient name extracted by the image analysis unit is inputted as an input parameter, the database inputs the ingredient It can be set as the structure which outputs the information of the dish which uses as a used ingredient.

According to the above configuration, when the user photographs the food material itself, the image analysis unit extracts the food material shown in the photographed image. In the database, when the food extracted by the image analysis unit is input as an input parameter, information on a dish using the input food as a used food is output. Since this information is displayed on the mobile terminal side, the user can easily obtain information on dishes made using the photographed ingredients.

In the information providing system, the image analysis unit includes an image analysis unit that performs image analysis processing on the photographed image, and recognizes and extracts a dish or food that is reflected in the photographed image, The database stores a dish or ingredient and the calories in the dish or ingredient in association with each other, and the database is inputted when the dish or ingredient extracted by the image analysis unit is inputted as an input parameter. It is possible to output a calorie equivalent to a dish or food.

According to the above configuration, when the user photographs a cooked dish or the like, the image analysis unit extracts the dish or ingredients reflected in the photographed image. When a dish or food extracted by the image analysis unit is input as an input parameter in the database, calories corresponding to the input food or food are output. If the calories of all dishes or ingredients extracted from the photographed image are added, the total calories of the dish can be calculated. Since this information is displayed on the mobile terminal side, the user can easily recognize the cooked total calories.

In order to solve the above-described problem, an information providing method of the present invention is an information providing method for providing information using a portable terminal including a photographing unit and a display unit, and is a photographed image photographed by the photographing unit. An image analysis process for analyzing and extracting parameters, and inputting the parameters extracted in the image analysis process to the database as input parameters and outputting information corresponding to the input parameters from the database And a display step of displaying the information output in the information output step on the display unit.

In order to solve the above-described problem, an information providing program of the present invention is an information providing program for providing information using a portable terminal including a photographing unit and a display unit, and the control unit included in the portable terminal In addition, an image analysis process for analyzing the captured image captured by the imaging unit and extracting parameters, and the parameters extracted in the image analysis process are input to the database as input parameters. An information output process for outputting information corresponding to the information from the database and a display process for displaying the information output in the information output process on the display unit are performed.

The information providing system, the information providing method, and the information providing program of the present invention can present various information to the user simply by photographing the menu, ingredients, food, and the like using the mobile terminal. There are effects such as.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 illustrates an embodiment of the present invention and is a block diagram illustrating a schematic configuration of an information providing system according to a first embodiment. 1, showing an embodiment of the present invention, is a block diagram showing another configuration of the information providing system in Embodiment 1. FIG. 3 is a diagram illustrating a data configuration example of a database in Embodiment 1. FIG. 6 is a diagram illustrating an example of an information display screen in the information providing system according to Embodiment 1. FIG. It is a figure which shows the other example of the information display screen in the information provision system of Embodiment 1. FIG. 1, showing an embodiment of the present invention, is a block diagram showing another configuration of the information providing system in Embodiment 1. FIG. 1, showing an embodiment of the present invention, is a block diagram illustrating a schematic configuration of an information providing system according to Embodiment 2. FIG. 6 is a diagram illustrating a data configuration example of a database according to Embodiment 2. FIG. 10 is a diagram illustrating an example of an information display screen in the information providing system of Embodiment 2. FIG. FIG. 20 is a diagram illustrating a data configuration example of a database in the third embodiment. FIG. 10 is a diagram illustrating an example of an information display screen in the information providing system according to the third embodiment.

[Embodiment 1]
Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing a schematic configuration of an information providing system (hereinafter referred to as the present system) according to the first embodiment. This system includes a mobile terminal 10 that is a smartphone or a tablet owned by a user, a system server (information providing apparatus) 20 that provides a service of the system, and an Internet 30 that connects the information terminal 10 and the system server 20. Is done.

The mobile terminal 10 includes, as basic functional units, a photographing unit 11 such as a camera, a display unit 12 such as a display, an input unit 13 such as a touch panel for a user to perform operation input, and an Internet 30 for connecting and communicating with the Internet 30. The communication part 14 and the control part 15 which controls operation | movement of the portable terminal 10 are provided. Further, the control unit 15 includes an OCR processing unit (image analysis unit) 16, a food material recognition unit (image analysis unit) 17, a position storage unit 18, and a setting storage unit 19 in order to use the present system.

The control unit 15 includes a CPU (Central Processing Unit) and a memory included in the mobile terminal 10. Further, the OCR (Optical Character Recognition) processing unit 16, the food material recognition unit 17, the position storage unit 18 and the setting storage unit 19 install predetermined application software in the mobile terminal 10, and the control unit 15 operates according to the software. It is realized with.

The system server 20 includes a control unit 21, a database 22, and a communication unit 23. The control unit 21 includes a data extraction unit 24 that extracts and extracts desired data from the database 22. Various kinds of food information are stored in the database 22 in association with various parameters. Further, the system server 20 is connected to the Internet 30 via the communication unit 23.

In the above description, the database 22 is provided in the system server 20, and the mobile terminal 10 and the system server 20 constitute this system. However, the present invention is not limited to this, and the present system can also be realized by using the mobile terminal 10 alone (stand-alone). In this case, the control unit 15 also functions as the data extraction unit 24 by application software installed in the mobile terminal 10 and the database 22 is also downloaded to the mobile terminal 10 side. That is, all the components for realizing the present system may be provided in the mobile terminal 10 (see FIG. 2).

Subsequently, a specific example of the system utilization method in the first embodiment will be described. In this specific example, for example, when a user with a disease such as diabetes or gout places an order from a menu of a restaurant or the like, the user should carry ingredients that should be careful about the disease (the intake should be restricted) to the user. It can be presented from the terminal 10. Further, in the following description, the case where the present system is configured by the mobile terminal 10 and the system server 20 will be described as an example (FIG. 1), but the case where the mobile terminal 10 is used in a stand-alone manner (FIG. 2) is also basic. The operation is similar.

The user photographs the menu table in the restaurant by the photographing unit 11 of the mobile terminal 10. The OCR processing unit 16 performs OCR processing on the captured image, and converts the character portion in the captured image into text data. Further, the food material recognition unit 17 recognizes and extracts a portion corresponding to the food material name from the text data. That is, when the user takes a picture of a restaurant menu, the names of ingredients described in the menu are recognized and extracted by the OCR processing unit 16 and the ingredients recognition unit 17. In addition, the display position of the extracted food name on the display unit 12 is stored in the position storage unit 18.

In addition, although the said process by the OCR process part 16, the foodstuff recognition part 17, and the position memory | storage part 18 may be performed with respect to a still image, it may be processed with respect to a moving image in real time. . When real-time processing is performed on a moving image, the mobile terminal 10 is preferably used in a stand-alone manner.

The setting storage unit 19 stores a user's disease and chronic illness as user information. User information stored in the setting storage unit 19 can be set and registered in advance by the user via the input unit 13. In addition, the setting storage unit 19 may store the user's age, sex, height, weight, lifestyle, and the like as user information.

Thus, when the user shoots the restaurant menu table, the names of ingredients recognized and extracted by the OCR processing unit 16 and the ingredients recognition unit 17 and the user information stored in the setting storage unit 19 are transmitted via the Internet 30. It is transmitted to the system server 20. The system server 20 determines ingredients (hereinafter, restricted ingredients) that the user should restrict intake based on the ingredient names and user information transmitted from the mobile terminal 10 and the ingredient information stored in the database 22. .

Here, a configuration example of the database 22 will be described. FIG. 3 is a diagram illustrating an example of a data configuration of the database 22 according to the first embodiment. In the database 22, information on whether or not each disease is a restricted food is set by an intake restriction flag. Specifically, the intake restriction flag is set to “1” if it is a restricted food, and the intake restriction flag is set to “0” if it is not a restricted food. For example, the intake restriction flag for ingredients A, D, and F is “1” for diabetes, and the intake restriction flag for ingredients B and E is “1” for gout. That is, it is determined that ingredients A, D, and F are restricted ingredients for diabetes, and ingredients B and E are restricted ingredients for gout. In the configuration in which the mobile terminal 10 is used standalone, it is not necessary to download all the information in the database 22 to the mobile terminal 10 side. For example, if the user only needs information on gout and does not need other information, only the data on restricted foods for gout may be selectively downloaded.

When the food name and user information are sent from the mobile terminal 10, the data extraction unit 24 inputs them to the database 22. And the restricted foodstuff with respect to the disease contained in user information is extracted among the inputted foodstuff names. The restricted food material extracted in this way is sent back to the mobile terminal 10 via the Internet 30.

When the information on the restricted food is sent back, the control unit 15 in the mobile terminal 10 presents to the user information about which food is the restricted food in the first photographed image (that is, the menu table image). Display control to do this. Specifically, as shown in FIG. 4, in the captured image displayed on the display unit 12, a strike line is superimposed and displayed on the name of the ingredient that is the restricted ingredient (information display screen). The display location of the erased line can be specified based on the display position of the ingredient name stored in the position storage unit 18. Thereby, the user can easily recognize that the foodstuff displayed in the cross-line is a restricted foodstuff.

In this way, in this system, the user simply takes a picture of the menu table with the mobile terminal 10 and only the restricted ingredients are displayed in a cross-hatched manner from among the ingredients listed in the menu table. Thereby, foodstuff information can be provided easily, without requiring a complicated operation | work to a user. In particular, AR (Augmented Reality) in which the mobile terminal 10 is configured to be used in a stand-alone manner and the above information is superimposed on the captured video is used for a user who uses this system in a sense of play. It is possible to have a feeling and to expect many users to use this system.

In the information display screen shown in FIG. 4, a disease name display area 50 may be provided in the screen to display the set disease names at the same time. By simultaneously displaying the disease name on the information display screen, the user can recognize without error what information the restricted food being presented is the restricted food for.

Furthermore, when a disease name is displayed in the disease name display area 50, the user may scroll the disease name display area 50 to change the display to another disease name. In this case, when the disease name to be displayed is changed, the display of the restricted food on the information display screen is also changed to that corresponding to the disease name. For example, when a restricted ingredient corresponding to gout is displayed on the information display screen, if the user changes the disease name displayed by scrolling the disease name display area 50 to diabetes, the data extraction unit 24 will detect diabetes. Are extracted from the database 22. And the portable terminal 10 changes an information display screen according to the information. In this configuration, the user can easily obtain information on restricted foods for various diseases without changing the user information set and stored in the setting storage unit 19.

In addition, in the above description, the case where information on restricted foods that should be restricted for each disease is presented has been described. In addition to this, foods that should be recommended for each disease (hereinafter referred to as recommended foods) It is also possible to present information. In this case, as shown in FIG. 3, in the database 22, food information indicating whether or not the food is a recommended food for each disease is set in the recommended intake flag. Specifically, the intake restriction flag is set to “1” if it is a recommended food, and the intake restriction flag is set to “0” if it is not a recommended food. For example, the intake restriction flag for food G is “1” for diabetes, and the intake restriction flag for food F is “1” for gout. That is, it is determined that the food G is a recommended food for diabetes and the food F is a recommended food for gout. In the information display screen shown in FIG. 4, for the recommended food, for example, a circle is superimposed on the food name and displayed.

This system stores the food information in the database 22 as widely and accurately as possible (increase of stored food and disease types, accurate setting of intake restriction flag and recommended intake flag) for the user. Convenience is improved. However, such food information is extremely diverse, and new knowledge is often discovered through new research. For this reason, it is difficult for the service provider of this system to keep updating the information in the database 22 appropriately.

For this reason, in this system, it is desirable that the information stored in the database 22 in the system server 20 is automatically updated by feedback of information from the user. For example, a certain user has knowledge that a certain ingredient X is a restricted ingredient for disease Y (or a recommended ingredient), but such a display is made on the ingredient information screen. If not, the user can send knowledge (food material information) he / she knows to the system server 20.

In the system server 20, the ingredients information sent from the user is pooled, and when a certain number of similar ingredient information is accumulated, the ingredient information is recognized as correct and the stored information in the database 22 is updated. . In this case, since the information stored in the database 22 is not updated by a small number of provision information, erroneous information from a small number of users is not reflected, and the reliability of the present system is ensured. Moreover, in the structure which uses the portable terminal 10 by stand-alone, the information of the database 22 memorize | stored in the portable terminal 10 side can be updated by downloading the database 22 stored in the system server 20 again.

In updating the information in the database 22, weighting is performed depending on whether the user who provided the information is an expert or a general user, and the information provided by the expert is reflected as having higher accuracy. It may be easy to be done. In this case, it is conceivable that occupations are included in the user information stored in the setting storage unit 19 and that occupations such as doctors and nutrition managers are regarded as experts.

In the above description, as an example of use of this system, a case where restricted food (or recommended food) is extracted and displayed from a restaurant menu table is exemplified, but various other usage methods are conceivable. For example, when a user makes a purchase at a supermarket or a convenience store, the label portion of the product is photographed with the mobile terminal 10 and, as shown in FIG. 5, the restricted ingredient (or recommended ingredient) from the ingredient name written on the label part. May be extracted and displayed. In this case, the label portion to be photographed may be a display portion of a trade name or a display portion of raw materials.

Furthermore, this system is not limited to those that present restricted foods or recommended foods for diseases, but can also present restricted foods for pregnant women in lactation, supplementary foods for pets, etc. .

Also, in this system, the OCR processing unit 16 is compatible with various foreign languages so that the names of foreign language ingredients recognized by the OCR processing unit 16 can be translated into a specific language (for example, Japanese) and displayed. May be. In this case, the translation dictionary may be stored in the database 22 of the system server 20.

In addition, the restricted food is a food that should be restricted and not meant to be prohibited. For this reason, excessive food intake should be refrained from restricted foods, but naturally, foods that can be moderately consumed are also included. For this reason, in this system, in combination with health care, restriction display may be performed including determination as to whether or not to eat depending on the day. Specifically, the following control can be considered.

First, when the user uses this system, it will be possible to record the total amount of ingredients and ingredients ingested. On the other hand, in the database 22, information on the intake frequency of the restricted foods for each disease (for example, food A can be taken up to once a week, food B can be taken up to twice a month) can be databased. Keep it. And the cumulative record of the foodstuff which the user ingested and the information regarding the ingestion frequency memorize | stored in the database 22 are compared, and when the said frequency | count is exceeded, the strike-through display mentioned above as a limitation foodstuff, If it does not exceed the limit, it may be possible to perform control such as not displaying as a restricted food.

Fig. 6 shows a functional block diagram of this system for performing the above control. In the block diagram of FIG. 6, communication between the control unit 15 and the database 22 is described with a configuration not using the Internet 30 (a configuration in which the mobile terminal 10 is used in a stand-alone manner), but the database 22 is arranged on the system server 20 side. And the structure which communicates via the internet 30 may be sufficient. This is the same in other embodiments.

The control unit 15 includes a cumulative data storage unit 41 and a determination unit 42 in addition to the OCR processing unit 16, the food material recognition unit 17, the position storage unit 18, and the setting storage unit 19 shown in FIG. The accumulated data storage unit 41 is a storage area for accumulating and recording the ingredients and components consumed by the user in the memory in the control unit 15. The determination unit 42 is a means for comparing and determining the cumulative record of the food taken by the user and the information regarding the intake frequency stored in the database 22.

In the configuration of FIG. 2, the cumulative data storage unit 41 is set as a storage area in the memory in the control unit 15, but may be set as a storage area in the database 22. Further, in the configuration in which the database 22 is arranged on the system server 20 side, the cumulative data is stored in association with the user ID, and the cumulative data is written / read out upon receiving the user ID notification from the mobile terminal 10. Just do it.

Further, in order to store the cumulative data of the food taken by the user in the cumulative data storage unit 41, it is necessary to input the food taken by the user in the mobile terminal 10. As this input method, for example, the following method can be considered.

As a first method, on the information display screen (touch panel display) shown in FIG. 4, for example, a shaded frame or a surrounding frame is superimposed and displayed on all the ingredients recognized by the ingredient recognition unit 17. If the display location of the food taken by the user is touched, it is stored in the cumulative data storage unit 41 together with the date of intake, assuming that the food is the food taken by the user.

As a second method, the food before the user ingests is directly photographed, the food (or dish) ingested by the user is recognized based on the photographed image, and stored in the cumulative data storage unit 41 together with the date of ingestion. Is the method. In the second method, the control unit 15 includes an image analysis unit (image analysis unit) 43 described in the second embodiment. That is, the image analysis unit 41 analyzes an image obtained by photographing a dish, and recognizes the photographed dish or the ingredients contained in the dish. The dish or food recognized by the image analysis unit 41 is stored in the cumulative data storage unit 41 together with the date of intake, assuming that the food or food taken by the user is taken.

When displaying the restricted food, the determination unit 42 determines whether or not to display a strikethrough as the restricted food based on the comparison between the intake frequency information stored in the database 22 and the cumulative data. For example, if there is a food B in the displayed food and the information on the intake frequency of the food B is set to “can be taken twice a month” in the database 22, the food B in the most recent month in the cumulative data Check the number of intakes. Then, if the number of intakes of food B in the most recent month exceeds two, a strikethrough is displayed as a restricted food, but if it is not exceeded, display as a restricted food is not performed.

Also, recommended ingredients may be emphasized from the above cumulative record. For example, if conditions such as “how much vitamins are needed per week” are set, information on “how much vitamins are needed this week” can be derived from the above cumulative record. As a result, when it is determined that active vitamin intake is necessary to satisfy the first setting condition, it is possible to highlight a food with a high vitamin content as a recommended food.

In this case, for example, a condition such as “how much vitamin should be taken per week (a necessary vitamin amount)” is set in the setting storage unit 19. Furthermore, in the database 22, information such as vitamin content in each food material is also made into a database. In the cumulative record, the total amount of vitamins ingested during the week (ingested vitamin amount) is also accumulated based on the food ingested by the user. Then, the determination unit 42 compares the necessary vitamin amount set in the setting storage unit 19 with the cumulative amount of intake vitamin, and if the difference is equal to or greater than a predetermined value, the food with a high vitamin content is selected. Highlight as recommended food. For example, it is possible to display the shaded foods that have a high vitamin content among the displayed foods, or to display a flag that indicates “recommended intake”. In addition, the said control can be performed similarly about components other than a vitamin.

In addition, when the ingredients and components ingested by the user who uses this system are accumulated and recorded, the calorie intake, ingredients and components in the meal of the day may be sent as a daily report by e-mail. In this case, when the user starts using the system, the e-mail address or the like of his / her own mobile terminal 10 may be registered in the system server 20.

In addition, when inputting ingested food and ingested ingredients for performing cumulative recording, a photographed image may be recorded so that it can be input later. For example, a recording icon is displayed at a predetermined location on an information display screen (touch panel display) that allows the user to input the food ingested, and if the user touches the icon, the information display screen is displayed on the mobile terminal 10. To be stored in the memory. Then, the information display screen may be read later by a predetermined operation, and input may be performed from the read information display screen.

[Embodiment 2]
In recent years, the ability of image recognition software has been remarkably improved, and for example, the types of dishes and foods can be recognized from images. In the second embodiment, ingredients are recognized and extracted from a captured image, and information such as a recommended menu recipe is provided to the user using the information.

FIG. 7 is a block diagram showing a schematic configuration of an information providing system (hereinafter referred to as the present system) according to the second embodiment. This system includes a mobile terminal 10 that is a smartphone or a tablet owned by a user, a system server 20 that provides services of the system, and an Internet 30 that connects the information terminal 10 and the system server 20. Of course, in the second embodiment, as in the first embodiment, the mobile terminal 10 can be used alone (stand-alone) to realize the present system. In the following description, the case where this system is comprised by the portable terminal 10 and the system server 20 is demonstrated as an example.

The portable terminal 10 includes a photographing unit 11, a display unit 12, an input unit 13, a communication unit 14, and a control unit 15 as basic functional units as in the first embodiment. Furthermore, the control unit 15 includes an image analysis unit (image analysis unit) 43 in order to use this system. The image analysis unit 43 is realized by installing predetermined application software on the mobile terminal 10 and the control unit 15 operating according to the software.

The system server 20 includes a control unit 21, a database 22, and a communication unit 23. The control unit 21 includes a data extraction unit 24 that extracts and extracts desired data from the database 22.

Subsequently, a specific example of the system utilization method in the second embodiment will be described. In this specific example, menu information recommended from the ingredients in the refrigerator (hereinafter, recommended menu) can be presented to the user from the mobile terminal 10.

The user photographs the inside of the refrigerator with the photographing unit 11 of the mobile terminal 10. The image analysis unit 43 analyzes the captured image, and recognizes / extracts food in the refrigerator (that is, the captured food). That is, when the user photographs the inside of the refrigerator, the food stored in the refrigerator is recognized and extracted by the image analysis unit 43. As described above, in recent years, the ability of image recognition software has improved remarkably, and the image analysis unit 43 can use known image recognition software as an API (Application Programming Interface).

The ingredients recognized and extracted by the image analysis unit 43 are transmitted to the system server 20 via the Internet 30. The system server 20 extracts a recommended menu based on the ingredients transmitted from the mobile terminal 10 and the information stored in the database 22.

Here, a configuration example of the database 22 in the system server 20 will be described. FIG. 8 is a diagram illustrating an example of a data configuration of the database 22 according to the second embodiment. The database 22 stores the names of dishes and the ingredients (used ingredients) used for the dishes in association with each other.

The data extraction unit 24 in the system server 20 inputs information about the food from the mobile terminal 10 to the database 22. And the input foodstuff and the used foodstuff stored in the database are contrasted, and the dish with much correspondence with the input foodstuff and the used foodstuff is extracted. Note that the number of dishes to be extracted is not limited to one, and it is preferable to extract a plurality of dishes as candidates. For example, if the input ingredients are ingredients a, b, d, and f, in the database 22 shown in FIG. 8, the food A corresponds to three ingredients used, and the dishes B and C correspond to two ingredients used. To do. In this case, it is possible to extract dish A as a first candidate and extract dishes B and C as second candidates. The dish extracted in this way is sent back to the portable terminal 10 via the Internet 30.

When the information on the dish extracted by the system server 20 is sent back, the control unit 15 in the mobile terminal 10 performs display control for presenting the dish as a recommended menu to the user. Specifically, as shown in FIG. 9, in the captured image displayed on the display unit 12, a button display of “view recommended recipe” is superimposed and displayed, and when the user touches this button, The dish sent from the system server 20 is displayed as a recommended menu. Furthermore, when the user selects one dish from the displayed recommended menu, the recipe for the selected dish may be displayed.

As for the recipe display of the dish, the recipe information itself may be stored in the database 22. For example, when the user selects a dish in cooperation with an existing dish recipe providing site, the dish on the site is displayed. It is possible to jump to the recipe page and display it. Of course, in this case, it is assumed that dish recipe information is also stored in the database 22 in association with the dish name.

In the system according to the second embodiment, the recognition / extraction of food by the image analysis unit 43 may be performed on a still image, but may be performed on a moving image in real time. preferable. In this case, while the user is shooting with the mobile terminal 10, the recognized food is displayed on the screen of the display unit 12 with an arrow display or the like. Thereby, the user can easily know which food is image-recognized by the image analysis unit 43. Furthermore, the user can take a picture while bringing the food item that the user wants to recognize the image to where it is easy to photograph. In addition, although the image analysis unit 43 could not recognize the food, the user may manually input from the input unit 13 the ingredients that the user wants to use for cooking. In this way, in this system, food material data in the refrigerator can be collected.

In the above description, a method for obtaining a recommended menu from ingredients in the refrigerator is illustrated as an example of use of the present system, but various other usage methods are conceivable. For example, when a user performs shopping at a supermarket or the like, it is possible to continuously photograph products (foodstuffs) that the user intends to purchase and obtain a recommended menu from the plurality of photographed foodstuffs.

Furthermore, in this case, the system server 20 may not only present the recommended menu but also present the ingredients that are insufficient when creating the recommended menu. For example, when the ingredients transmitted to the system server 20 are ingredients a, b, and d, and the system server 20 presents the dish A as a recommended menu based on the database of FIG. May be presented. By presenting the insufficient food, the user can purchase the food c.

In addition, in this system, it is also possible to present foods to be added in combination with health management for the whole family. Specifically, the following control can be considered.

First, when the user uses this system, the ingredients of the ingredients taken in the home are accumulated (records that must be taken / preferred to avoid) and recorded. Enter how many items you want to make in advance, and in a simple case, consider a recipe that optimally separates the ingredients from the above ingredients according to the age structure of the family. In the future, if it becomes possible to link the results of an individual Ningen Dock, it is possible to provide more detailed recipes.

The functional block diagram of this system for performing the above control is basically the same as that shown in FIG. However, the accumulated data storage unit 43 can record the ingested foods and ingested food materials of all the family members as well as individual users. In order to make this possible, for example, on the information display screen (touch panel display) shown in FIG. 4, for example, shading or a surrounding frame is superimposed on all the ingredients recognized by the ingredient recognition unit 17 and this information display is performed. If you touch the display location of the food that ingested the user on the screen, the family list will be displayed, and you can specify the person who ingested the food in the family from this family list. It is done.

At the time of presenting the recommended menu, the determination unit 42 determines the intake restriction food or the recommended intake food for each individual in the family by the method described in the first embodiment, and the dish using the recommended intake food is highly recommended. On the other hand, it is conceivable to perform display control to set a recommended menu having a low recommendation level for a dish using an intake restriction ingredient while setting the recommended menu.

[Embodiment 3]
In the third embodiment, ingredients are recognized and extracted from a photographed image of cooked dishes, and information such as the presence of calories and restricted ingredients is provided to the user using the information. The outline of the system in the third embodiment is shown in FIG. 7 as in the second embodiment.

Subsequently, a specific example of the system usage method in the third embodiment will be described. In this specific example, it is possible to recognize and extract the used ingredients by analyzing the captured image of the cooked dish, and to present information such as the presence of calories and restricted ingredients to the user from the mobile terminal 10. Become.

The user photographs the cooked dish with the photographing unit 11 of the mobile terminal 10. The image analysis unit 43 analyzes the captured image, and recognizes and extracts the ingredients used for the dish (that is, the photographed ingredients). Moreover, in this Embodiment 3, the image analysis part 43 shall obtain | require not only the kind of foodstuff but the usage-amount in the dish by analysis.

Each ingredient recognized and extracted by the image analysis unit 43 and its usage amount are transmitted to the system server 20 via the Internet 30. The system server 20 calculates the number of calories of the shot food based on the ingredients transmitted from the mobile terminal 10 and the amount of use thereof and the information stored in the database 22.

Here, a configuration example of the database 22 in the system server 20 will be described. FIG. 10 is a diagram illustrating an example of a data configuration of the database 22 according to the third embodiment. The database 22 stores foodstuffs and calories per unit gram (for example, 100 grams) of the foodstuff in association with each other.

The data extraction unit 24 in the system server 20 inputs information on the ingredients and the amount of use thereof from the mobile terminal 10 to the database 22. And the calorie equivalent to the usage-amount of each foodstuff used with the said dish is calculated, and the total calorie of the said dish input by calculating the calorie in all the foodstuffs is calculated. The calculation of the total calories is performed by the control unit 21 of the system server 20. The calculated total calories of the dish are sent back to the mobile terminal 10 via the Internet 30.

When the total calorie of the dish calculated by the system server 20 is sent back, the control unit 15 in the portable terminal 10 displays the calorie information of the dish in the first photographed image (that is, the image of the cooked dish). Display control for presentation to the user is performed. Specifically, as shown in FIG. 11, an information display area 51 is provided in the captured image displayed on the display unit 12, and the total calories of the captured food are displayed in the information display area 51 (information display screen). . The information display area 51 may also display the recognized types of ingredients.

In the system according to the third embodiment, the ingredients recognized and extracted by the image analysis unit 43 are naturally limited to those reflected in the photographed image, and the ingredients are engraved in the cooked dish. In many cases, the recognition by the image analysis unit 43 becomes impossible due to the change of the. However, this system does not assume the use for medical purposes, but assumes a health management application that can be used by the user in a playful sense, and only needs to be able to recognize and extract foods to the extent possible.

Further, instead of individually recognizing ingredients in the cooked dish, the type of dish itself may be recognized and the calories of the whole dish may be displayed. For example, when the dish is a general dish such as curry, the image analysis unit 43 can recognize the type of the dish itself. In this case, it is assumed that the database 22 stores the general number of calories of a dish that can be recognized and extracted by the image analysis unit 43. Even if the curry is not surely recognized, a plurality of similar items such as curry / stew / soup are displayed, and the user selects a corresponding dish (curry) from the displayed plurality. Anyway. In addition, when some food is topped in the curry, the image analysis unit 43 may recognize and extract the curry as a dish and the topped food separately.

Also in the present system according to the third embodiment, when the total calories of the displayed dish are incorrect, the user may be able to correct it. In addition, when the calorie value corrected by the user is too large or too small, the control unit 21 of the system server 20 determines that the correction by the user is inappropriate and does not adopt the corrected value. You can also.

Further, in the system according to the third embodiment, the restricted food can be displayed in combination with the system according to the first embodiment. That is, the database 22 has not only the data table shown in FIG. 10 but also the data table shown in FIG. 3, and the ingredients transmitted to the system server 20 are input to the data table shown in FIG. I do. For example, when the food B is included in the food transmitted to the system server 20, since the food B (for example, soybean) is a restricted food for gout, this is displayed on the information display screen (see FIG. 11). You can also.

In the above description, as an example of use of the system, a method of obtaining the calories of the cooked food from the cooked food image is illustrated, but various other usage methods are conceivable. For example, it is possible to store a data table that associates brands of alcohol and sugar content in the database 22 and displays the sugar content of the liquor when the user has photographed a label or the like of the sake bottle. is there.

The information providing system according to the present invention can provide various services to the user of the mobile terminal 10 by the above-described usage method, and can also be used for acquiring big data by using this system by a large number of users. For example, in the usage method of the first embodiment, when information is transmitted from the mobile terminal 10 to the system server 20, the location information of the store where the user has taken the menu is included, so that the store menu and the location of the store are obtained. Corresponding data can be collected. Further, even when the user uses the system, even if the user uses the system, if the mobile terminal 10 transmits information on the store menu and the store location to the system server 20, the data Can be collected. If the user information includes the age and sex of the user, it is possible to collect data on what kind of store the person of the age and sex prefers.

In order to acquire such big data, it is necessary for a large number of users to use this system, but in order to increase the number of users of this system, it is important to simplify user operations as much as possible. . In this system, the user can obtain a variety of information simply by photographing a menu, ingredients, and dishes, and the operation of the user is extremely simplified. Moreover, if AR is used for displaying information, the user can use the system as if he / she is playing, and it can be expected that many users will use the system.

The embodiment disclosed herein is illustrative in all respects and does not serve as a basis for limited interpretation. Therefore, the technical scope of the present invention is not interpreted only by the above-described embodiments, but is defined based on the description of the scope of claims. Moreover, all the changes within the meaning and range equivalent to a claim are included.

DESCRIPTION OF SYMBOLS 10 Mobile terminal 11 Image pick-up part 12 Display part 13 Input part 14 Communication part 15 Control part 16 OCR process part (Image analysis means)
17 Food recognition unit (image analysis means)
18 position storage unit 19 setting storage unit 20 system server (information providing apparatus)
21 Control unit 22 Database 23 Communication unit 24 Data extraction unit 30 Internet 41 Cumulative data storage unit 42 Determination unit 43 Image analysis unit (image analysis means)
50 Disease name display area 51 Information display area

Claims

An information providing system that provides information using a portable terminal including a photographing unit and a display unit,
Image analysis means for analyzing the captured image captured by the imaging unit and extracting parameters;
A parameter that is extracted by the image analysis means as an input parameter, and a database that outputs information corresponding to the input parameter;
Information provided from the database is displayed on the display unit.
The information providing system according to claim 1,
The mobile terminal includes an input unit, and it is possible to input a disease name from the input unit,
The image analysis means includes
An OCR processing unit that performs OCR processing on the captured image and converts character portions in the captured image into text data;
An ingredient recognition unit that extracts a part corresponding to the ingredient name from the text data,
In the database, the ingredient name and the ingredient information indicating whether or not the ingredient is a restricted ingredient that should be restricted for each disease are stored in association with each other,
In the database, when the name of the ingredient extracted by the ingredient recognition unit and the name of the disease input from the input unit are input as input parameters, the input food is a restricted ingredient for the input disease. An information providing system that outputs information on whether or not there is.
The information providing system according to claim 1,
The image analysis means includes
It includes an image analysis unit that performs image analysis processing on the captured image and recognizes / extracts foodstuffs reflected in the captured image,
In the database, a dish name and used ingredients used for the dish are stored in association with each other,
The information providing system is characterized in that, when the name of the ingredient extracted by the image analysis unit is input as an input parameter, the database outputs information on a dish using the input ingredient as a used ingredient.
The information providing system according to claim 1,
The image analysis means includes
Including an image analysis unit that performs image analysis processing on the captured image and recognizes / extracts a dish or food that is reflected in the captured image;
In the database, dishes or ingredients and calories in the dishes or ingredients are stored in association with each other,
When the dish or food extracted by the image analysis unit is input as an input parameter, the database outputs calories corresponding to the input dish or food.
An information providing method for providing information using a portable terminal provided with an imaging unit and a display unit,
An image analysis step of analyzing the captured image captured by the imaging unit and extracting parameters;
An information output step of inputting the parameters extracted in the image analysis step to the database as input parameters, and outputting information corresponding to the input parameters from the database;
The information provision method characterized by including the display process which displays the information output at the said information output process on the said display part.
An information providing program for providing information using a mobile terminal provided with an imaging unit and a display unit,
In the control unit of the portable terminal,
An image analysis step of analyzing the captured image captured by the imaging unit and extracting parameters;
An information output step of inputting the parameters extracted in the image analysis step to the database as input parameters, and outputting information corresponding to the input parameters from the database;
And a display step of displaying the information output in the information output step on the display unit.