WO2022009539A1 - Program, terminal device, and service providing system - Google Patents

Program, terminal device, and service providing system Download PDF

Info

Publication number
WO2022009539A1
WO2022009539A1 PCT/JP2021/019424 JP2021019424W WO2022009539A1 WO 2022009539 A1 WO2022009539 A1 WO 2022009539A1 JP 2021019424 W JP2021019424 W JP 2021019424W WO 2022009539 A1 WO2022009539 A1 WO 2022009539A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
control unit
unit
server device
display
Prior art date
Application number
PCT/JP2021/019424
Other languages
French (fr)
Japanese (ja)
Inventor
雅祥 平本
和宏 村上
直紀 杉村
勇次 國武
光洋 阿曽
広武 平
大樹 寺田
Original Assignee
パナソニックＩｐマネジメント株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by パナソニックＩｐマネジメント株式会社 filed Critical パナソニックＩｐマネジメント株式会社
Publication of WO2022009539A1 publication Critical patent/WO2022009539A1/en

Links

Images

Classifications

    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F24—HEATING; RANGES; VENTILATING
    • F24C—DOMESTIC STOVES OR RANGES ; DETAILS OF DOMESTIC STOVES OR RANGES, OF GENERAL APPLICATION
    • F24C15/00—Details
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F24—HEATING; RANGES; VENTILATING
    • F24C—DOMESTIC STOVES OR RANGES ; DETAILS OF DOMESTIC STOVES OR RANGES, OF GENERAL APPLICATION
    • F24C7/00—Stoves or ranges heated by electric energy
    • F24C7/02—Stoves or ranges heated by electric energy using microwaves
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce

Definitions

  • This disclosure relates to programs, terminal devices, and service provision systems.
  • Patent Document 1 is a technique of acquiring food information from code information given to food, acquiring heating information of a microwave oven corresponding to the acquired food information from a server device, and transmitting the acquired heating information to a microwave oven. Is disclosed.
  • the purpose of this disclosure is to improve the convenience of the user based on the code information given to the food.
  • the program in the present disclosure includes a detection procedure for detecting code information included in a captured image captured by a photographing unit on a computer, and a first determination procedure for determining the use of the code information based on an operation accepted by the operating unit.
  • the transmission procedure of transmitting the information acquisition request corresponding to the determined use of the code information to the server device and the display procedure of displaying the information received from the server device are executed.
  • the terminal device in the present disclosure includes a display unit, a communication unit, a photographing unit, a code detecting unit that detects code information included in an image captured by the photographing unit, an operation unit that accepts an operation, and the operation unit.
  • the control unit includes a control unit that determines the use of the code information based on the received operation and transmits an information acquisition request corresponding to the determined use to the server device via the communication unit. The information received from the server device is displayed on the display unit.
  • the service providing system in the present disclosure includes a display unit, a communication unit, a photographing unit, a code detection unit that detects code information included in an image captured by the photographing unit, an operation unit that accepts operations, and the operation unit.
  • a terminal device including a control unit that determines the use of the code information based on the operation accepted by the user and transmits an information acquisition request corresponding to the determined use to the server device via the communication unit, and the above.
  • the server device includes the server device that transmits information corresponding to the acquisition request to the terminal device, and the control unit causes the display unit to display the information received from the server device.
  • the program, terminal device, and service providing system in the present disclosure determine the use of the code information based on the received operation, and can acquire and display the information corresponding to the determined use from the server device. Therefore, even when the same code information is read, the information corresponding to the operation can be displayed on the display unit, and the convenience of the user can be improved.
  • FIG. 1 is a system configuration diagram of a service providing system.
  • FIG. 2 is a block configuration diagram showing the configurations of a terminal device, a cooker, and a server device.
  • FIG. 3 is a diagram showing the configuration of the user database.
  • FIG. 4 is a diagram showing an example of a home screen.
  • FIG. 5 is a diagram showing an example of a process selection screen.
  • FIG. 6 is a diagram showing an example of a scan screen.
  • FIG. 7 is a diagram showing an example of a selection screen.
  • FIG. 8 is a diagram showing an example of a scan screen.
  • FIG. 9 is a diagram showing an example of a confirmation screen.
  • FIG. 10 is a diagram showing an example of a scan screen.
  • FIG. 11 is a diagram showing an example of a scan screen.
  • FIG. 12 is a diagram showing an example of a selection screen.
  • FIG. 13 is a diagram showing an example of a confirmation screen.
  • FIG. 14 is a diagram showing an example of a scan screen.
  • FIG. 15 is a diagram showing an example of a search result screen.
  • FIG. 16 is a flowchart showing the operation of the terminal device.
  • FIG. 17 is a flowchart showing the operation of the M terminal device.
  • FIG. 18 is a flowchart showing the operation of the terminal device.
  • FIG. 19 is a flowchart showing the operation of the server device.
  • FIG. 1 is a system configuration of the service providing system 1.
  • the service providing system 1 has a configuration in which a terminal device 100, a cooker 200, and a server device 300 are connected via a network 10.
  • the network 10 may be a wired network, or may include a wireless network in part.
  • the network 10 may be configured to include, for example, an Internet, a mobile communication network, a wide area network such as a WAN (Wide Area Network), or a private network such as an intranet or a LAN (Local Area Network). ..
  • the connection form of the terminal device 100 and the cooker 200 to the network 10 may be wired or wireless such as Wi-Fi (registered trademark).
  • the terminal device 100 is a portable information processing device, and is a terminal owned by a user who receives a service.
  • a smartphone for example, a smartphone, a tablet PC (Personal Computer), a notebook PC, a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like can be adopted.
  • a case where the terminal device 100 is a smartphone will be described as an example.
  • a service application program (hereinafter, simply referred to as a service application 143) is installed in the terminal device 100.
  • the service application 143 may be downloaded from the server device 300 by the user operating the terminal device 100, or may be downloaded from a server device other than the server device 300.
  • the terminal device 100 executes the service application 143 to provide a service to the user.
  • the service provided by executing the service application 143 includes, for example, when food such as frozen food is cooked by the cooker 200, heating information such as rated high frequency output and heating time is transmitted to the cooker 200. It includes services such as setting, managing the inventory of foods and ingredients purchased by the user, and searching for recipes containing the ingredients selected by the user.
  • the cooker 200 is a cooking device that performs cooking. In the present embodiment, the case where the cooker 200 is an oven range will be described, but the cooker 200 may be, for example, a rice cooker or an electric pressure cooker.
  • the cooker 200 is connected to the server device 300 via the network 10.
  • the cooker 200 sets cooking for cooking based on the data received from the server device 300. When the user presses the cooking start button while the cooking setting is made, the cooker 200 starts cooking according to the set cooking setting.
  • the terminal device 100 and the cooker 200 do not directly communicate with each other. However, for example, the terminal device 100 and the cooker 200 can be communicated with each other by short-range wireless communication such as Bluetooth (registered trademark). May be configured to directly communicate and send / receive data.
  • short-range wireless communication such as Bluetooth (registered trademark). May be configured to directly communicate and send / receive data.
  • the server device 300 provides information to the terminal device 100 owned by the registered user.
  • the information provided by the server device 300 includes, for example, information on heating information of the cooker 200, a food expiration date, a food storage method, a best-by date, a recipe, and the like. Further, when the server device 300 receives the heating information information from the terminal device 100, the server device 300 transmits the received heating information to the cooker 200 associated with the terminal device 100 that has received the information. That is, the server device 300 transmits the heating information selected by the user of the terminal device 100 to the cooker 200 used by the user, and sets the cooking of the cooker 200.
  • FIG. 2 is a block configuration diagram showing the configurations of the terminal device 100, the cooker 200, and the server device 300.
  • the terminal device 100 includes a terminal communication I / F 111, a camera 113, a display unit 115, and a terminal control unit 130.
  • the terminal communication I / F 111 is a communication interface provided with a communication module, and performs data communication with the server device 300 via the network 10.
  • a wireless communication method such as 4G (Generation), 5G, Wi-Fi, etc. is adopted.
  • the terminal communication I / F 111 corresponds to an example of the communication unit.
  • the camera 113 includes a lens and an image sensor.
  • the image sensor is, for example, an image sensor such as a CCD (Charged Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).
  • the camera 113 generates a captured image by shooting, and outputs the generated captured image to the terminal control unit 130.
  • the camera 113 corresponds to an example of a photographing unit.
  • the display unit 115 includes a touch panel 117 including a display panel such as a liquid crystal display or an organic EL (Electronic Luminescent) and a touch sensor for detecting a touch operation.
  • the touch panel 117 corresponds to an example of an operation unit.
  • the touch sensor outputs coordinate information indicating the position of the display panel to which the user's finger or an indicator such as a pen is in contact with the terminal control unit 130. This coordinate information is the coordinates in the preset coordinate system of the display panel.
  • the terminal control unit 130 is a computer including a terminal storage unit 140 and a terminal processor 150.
  • the terminal storage unit 140 includes, for example, a non-volatile storage device such as a flash memory or an SD card, and a volatile storage device such as a RAM (Random access memory).
  • the terminal storage unit 140 stores a control program 141 such as an OS (Operating System), a service application 143, a user ID (Identifier) 145, and various other setting data.
  • the user ID 145 is information set by the user of the terminal device 100, and is identification information for identifying the user of the service providing system 1.
  • the terminal storage unit 140 stores inventory management data of foods and foodstuffs purchased by the user.
  • the inventory management data includes, for example, data such as the type (name) of food or foodstuff, the number of foods, the expiration date, the expiration date, and the storage method.
  • the terminal processor 150 is composed of a CPU (Central Processing Unit), a microcomputer equipped with a CPU (MicroControllerUnit), an MPU (MicroProcessorUnit), and other microcomputers. Further, the terminal processor 150 may be configured as a part of a SoC (System-on-a-Chip) including a terminal communication I / F 111, a camera, various sensors, and a memory.
  • a CPU Central Processing Unit
  • MPU MicroProcessorUnit
  • SoC System-on-a-Chip
  • the terminal control unit 130 includes a code detection unit 151, an image recognition unit 153, a data acquisition unit 155, and a display control unit 157 as functional configurations. These are functions realized by the terminal processor 150 executing the control program 141 stored in the terminal storage unit 140 and the service application 143.
  • the code detection unit 151 detects the code information from the captured image of the camera 113.
  • the code information acquired by the code detection unit 151 includes, for example, a one-dimensional code such as a JAN code and a two-dimensional code such as a QR code (registered trademark).
  • the code detection unit 151 detects code information such as a one-dimensional code and a two-dimensional code from the captured image, and acquires identification data for identifying foods contained in the detected code information.
  • the operation mode of the terminal control unit 130 when the code detection unit 151 operates is called a code recognition mode.
  • the image recognition unit 153 identifies an object captured in the captured image of the camera 113. For example, the image recognition unit 153 extracts feature points such as shape, color, and size included in the captured image, compares the extracted feature points with the feature points of the foodstuff stored in advance, and types of the foodstuff. Is determined.
  • the operation mode of the terminal control unit 130 when the image recognition unit 153 operates is called an image recognition mode.
  • the data acquisition unit 155 controls the terminal communication I / F 111, transmits a data acquisition request to the server device 300, and acquires data from the server device 300.
  • the data acquisition unit 155 is a case where the code detection unit 151 detects the code information including the identification data for identifying the food from the image taken by the camera 113, and the operation detected by the touch panel 117 is the heating information to the cooker 200. When the operation is to select the process of transmitting the code information, it is determined that the use of the code information is the acquisition of the heating information. In this case, the data acquisition unit 155 sends a request for acquisition of heating information including the food identification data and the user ID 145 to the server device 300, and acquires the data including the heating information.
  • the data acquisition unit 155 is a case where the code detection unit 151 detects code information including food identification data from the captured image of the camera 113, and the operation detected by the touch panel 117 is an operation of selecting inventory management. If, the use of the code information is determined to be inventory management. In this case, the data acquisition unit 155 transmits to the server device 300 a request for acquisition of information necessary for inventory management, including food identification data and user ID 145. Information necessary for inventory control includes the expiration date of food, storage method, and the like. Hereinafter, the information necessary for food inventory management is referred to as food management information.
  • the data acquisition unit 155 determines the type of the food material.
  • Information necessary for inventory management includes the expiration date of ingredients, storage method, and the like.
  • the information necessary for inventory management of ingredients is referred to as foodstuff management information.
  • the data acquisition unit 155 determines the type of the food material.
  • a request for acquiring recipe data including the type information and the user ID 145 is transmitted to the server device 300.
  • the display control unit 157 controls the display unit 115 to display the display screen on the touch panel 117.
  • the display control unit 157 generates display data, and outputs the generated display data to the display unit 115.
  • the display unit 115 causes the touch panel 117 to display the display screen based on the input display data. For example, the display control unit 157 displays a food package image received from the server device 300, heating information, a food expiration date, a storage method, a recipe, and the like on the touch panel 117.
  • the cooker 200 includes a cooker communication I / F 201, an operation unit 203, a display unit 205, a cooking control unit 210, and a cooking unit 220.
  • the cooker communication I / F 201 is a communication interface including a communication module.
  • the cooker communication I / F 201 performs data communication with the server device 300 by wireless communication such as Wi-Fi.
  • the operation unit 203 includes, for example, a cooking start button, a setting button for setting a heating time, a rated high frequency output, and the like.
  • the operation unit 203 receives the operation of the cooking start button or the setting button, the operation unit 203 outputs an operation signal corresponding to the button for which the operation is received to the cooking control unit 210.
  • the display unit 205 includes a display panel such as a liquid crystal panel, and displays characters and images on the display panel under the control of the cooking control unit 210. For example, an image of food to be cooked and heating information such as heating time and rated high frequency output are displayed on the display panel.
  • the cooking control unit 210 controls the cooking unit 220 according to the operation received by the operation unit 203 and the heating information received from the server device 300, and executes cooking. Further, the cooking control unit 210 causes the display unit 205 to display the operation content received by the operation unit 203 and the heating information received from the server device 300.
  • the cooking unit 220 heats and cooks the food to be cooked in the heating chamber (not shown) according to the cooking method set by the cooking control unit 210 and the cooking time.
  • Cooking methods include microwave cooking, oven cooking, steam cooking, grill cooking and the like.
  • the server device 300 includes a server communication I / F 301 and a server control unit 310.
  • the server communication I / F 301 includes, for example, a network adapter such as a NIC (Network Interface Card), and performs data communication with the terminal device 100 and the cooker 200 via the network 10.
  • NIC Network Interface Card
  • the server control unit 310 includes a server storage unit 320 and a server processor 330.
  • the server storage unit 320 is composed of, for example, a non-volatile storage device such as a ROM or a flash memory, and an auxiliary storage device such as a hard disk drive or an SSD (Solid State Drive).
  • the server storage unit 320 stores the control program 321 and the database 323.
  • the data registered in the database 323 includes heating information of the registered food.
  • the foods registered in the database 323 are foods provided by the companies participating in the service provision system 1.
  • the heating information includes the rated high frequency output and the heating time information.
  • the rated high frequency output and the information on the heating time are set for each number of foods to be cooked at the same time, for example, when the heating time differs depending on the number of foods to be cooked at the same time such as grilled rice balls and fried chicken.
  • the data registered in the database 323 includes data such as a package image of the registered food, an expiration date, and a storage method.
  • the data registered in the database 323 includes data such as the expiration date for each ingredient such as vegetables, meat, and fish, and the storage method.
  • the data registered in the database 323 includes cooking recipe data.
  • FIG. 3 is a diagram showing the configuration of the user database 323a included in the database 323.
  • the user database 323a is a database in which information of users registered in the service providing system 1 is registered.
  • the type and model information of the cooker 200 used by the user and the communication address of the cooker 200 are registered in association with the user ID and password which are the identification information of the user.
  • the server processor 330 is composed of, for example, a CPU, an MCU, and an MPU.
  • the server control unit 310 executes various processes by the server processor 330 executing the control program 321 stored in the server storage unit 320.
  • the server control unit 310 when the server control unit 310 receives a heating information acquisition request from the terminal device 100, the server control unit 310 reads and reads the food package image corresponding to the identification data included in the received acquisition request and the heating information from the database 323. The data is transmitted to the terminal device 100. Further, when the server control unit 310 receives the acquisition request of the food management information including the food identification data and the user ID 145 from the terminal device 100, the server control unit 310 may receive a food package image corresponding to the identification data included in the received acquisition request. , Data such as the expiration date of the food and the storage method are read from the database 323, and the read data is transmitted to the terminal device 100.
  • the server control unit 310 when the server control unit 310 receives the acquisition request of the food material management information including the information indicating the type of the food material and the user ID 145 from the terminal device 100, the server control unit 310 stores data such as the expiration date of the corresponding food material and the storage method in the database. It is read from 323 and the read data is transmitted to the terminal device 100. Further, when the server control unit 310 receives the recipe data acquisition request including the information indicating the type of the food material and the user ID 145 from the terminal device 100, the server control unit 310 reads the recipe data including the food material indicated by the received information from the database 323. The read data is transmitted to the terminal device 100.
  • the server control unit 310 when the server control unit 310 receives the heating information acquisition request from the terminal device 100, the server control unit 310 acquires the communication address of the cooker 200 associated with the user ID 145 of the user who owns the terminal device 100, and acquires the communication. Send heating information to the address.
  • FIG. 4 is a diagram showing an example of the home screen 50.
  • the terminal control unit 130 activates the service application 143 and displays the home screen 50 shown in FIG.
  • the home screen 50 displays a menu icon 51, a scan icon 53, notification screens 55, 57, and the like.
  • the menu icon 51 is an icon that displays a list of menus when the menu icon 51 is selected.
  • the scan icon 53 is an icon that scans the code information attached to the food by the image taken by the camera 113 and the food and executes the process based on the scan result.
  • scanning Information such as notifications and recommended recipes is displayed on the notification screens 55 and 57.
  • FIG. 4 shows an example in which two notification screens 55 and 57 are displayed, but the number of notification screens 55 and 57 displayed on the home screen 50 is arbitrary.
  • FIG. 5 is a diagram showing an example of a process selection screen 510 displayed on the touch panel 117 when the scan icon 53 is selected.
  • the process selection screen 510 is a screen displaying a list of processes executed by using scanning.
  • the first process display 511, the second process display 513, and the third process display 515 are displayed in a selectable manner.
  • the first process display 511 is selected by the user touching the area of the touch panel 117 on which the first process display 511 is displayed. The same applies to the second process display 513 and the third process display 515.
  • the first process is displayed on the first process display 511.
  • the first process is a process of transmitting heating information to the cooker 200.
  • the second process is displayed on the second process display 513.
  • the second process is a process for managing the inventory of scanned foods and foodstuffs.
  • the third process is displayed on the third process display 515.
  • the third process is a process of searching for a recipe containing the scanned ingredients.
  • FIG. 6 is a diagram showing an example of a scan screen 520 displayed on the touch panel 117 when the first processing display 511 is selected.
  • a switching display 521 On the scan screen 520 shown in FIG. 6, a switching display 521, a guidance display 523, and a range image 525 are displayed.
  • the switching display 521 sets the operation mode of the terminal control unit 130 to a code recognition mode for recognizing code information such as a one-dimensional code or a two-dimensional code, and extracts feature points from captured images, and uses the extracted feature points as ingredients. It is a display to switch to the image recognition mode for recognizing.
  • Code recognition 521A and image recognition 521B are displayed on the switching display 521.
  • FIG. 6 shows a state in which the code recognition 521A indicated by the hatching of the diagonal line is selected.
  • the display of the scan screen 520 displayed when the code recognition 521A is selected is referred to as a code recognition display.
  • a display indicating which of the code recognition 521A included in the switching display 521 and the image recognition 521B is selected is displayed.
  • a guidance such as "The one-dimensional code and the two-dimensional code are automatically recognized.”
  • the image recognition 521B for example, a guidance such as "Analyzing the image and recognizing the foodstuff" is displayed.
  • the range image 525 displays an image showing the recognition range of the code recognition 521A and the image recognition 521B. That is, among the images captured by the camera 113, code recognition and image recognition are performed on the images within the range indicated by the range image 525.
  • FIG. 7 is a diagram showing an example of a selection screen 530 displayed on the touch panel 117 when the code information attached to the food is recognized on the scan screen 520 shown in FIG.
  • the terminal control unit 130 determines that the use of the code information is the acquisition of heating information.
  • the terminal control unit 130 transmits the acquired data to the server device 300, and acquires the food heating information corresponding to the code information from the server device 300.
  • a display field 531 of the food package, a list display field 533 displaying a list of heating information that can be set in the cooker 200, and a transmission button 535 are displayed.
  • the food package displayed in the food image display field 331 is an image acquired by the terminal device 100 from the server device 300.
  • the terminal control unit 130 reads the code information
  • the terminal control unit 130 transmits the read code information to the server device 300.
  • the server device 300 transmits the food package image corresponding to the code information received from the terminal device 100 and the heating information to the corresponding terminal device 100.
  • the rated high frequency output 5331 and the heating time 5333 are displayed as heating information.
  • the heating time 5333 when the food is a food whose cooking time is changed depending on the number of foods, the heating time for each number of foods is displayed.
  • the transmission button 535 is a button for transmitting information of the rated high frequency output 5331 selected by the user and the heating time 5333 to the cooker 200.
  • the terminal control unit 130 transmits information about the rated high frequency output 5331 and the heating time 5333 to the server device 300.
  • the server device 300 transmits the information of the rated high frequency output 5331 received from the terminal device 100 and the heating time 5333 to the corresponding cooking appliance.
  • FIG. 8 is a diagram showing an example of the scan screen 540 displayed on the touch panel 117 when the second process display 513 shown in FIG. 5 is selected, and shows the scan screen 540 of the code recognition display. Since the scan screen 540 shown in FIG. 8 is the same as the scan screen 520 shown in FIG. 6, a detailed description of the scan screen 540 will be omitted.
  • FIG. 9 is a diagram showing an example of a confirmation screen 550 displayed on the touch panel 117 when the code information attached to the food is recognized on the scan screen 540 shown in FIG.
  • the terminal control unit 130 determines that the use of the code information is inventory management of the scanned food.
  • the terminal control unit 130 transmits a food management information acquisition request including the food identification data and the user ID 145 to the server device 300, and acquires the food food management information of the food from the server device 300.
  • a food package display field 551, a food information display field 553, and a registration button 555 are displayed.
  • the food package displayed in the food image display field 331 is an image acquired by the terminal device 100 from the server device 300.
  • the registration button 555 is a button for registering in inventory management.
  • the terminal control unit 130 registers the food displayed in the display field 551 in inventory management. That is, the number data for managing the number of foods displayed in the display field 551 is added by 1.
  • FIG. 10 is a diagram showing an example of a scan screen 540 of the code recognition display displayed after the registration button 555 is pressed on the confirmation screen 550 shown in FIG. 9.
  • the terminal control unit 130 displays the scan screen 540 shown in FIG. 10 again so that food can be continuously scanned.
  • the scan screen 540 shown in FIG. 10 indicates that the food has been registered in inventory control and that the food can be continuously scanned as the guidance display 543. Further, on the scan screen 540 shown in FIG. 10, an end button 547 for ending the food scan is displayed.
  • FIG. 11 is a diagram showing an example of a scan screen 540 displayed on the touch panel 117 when the second processing display 513 shown in FIG. 5 is selected, and is a diagram showing a case where the image recognition 541B is selected.
  • FIG. 11 shows a state in which the image recognition 521B shown by hatching of diagonal lines is selected.
  • the display of the scan screen 520 displayed when the image recognition 521B is selected is referred to as an image recognition display.
  • the guidance display 543 displayed on the scan screen 540 displays a guidance such as "Foodstuff is detected from the image.” Further, when the image recognition 541B is selected, the scan button 549 for starting the scan is displayed.
  • the user moves the position of the terminal device 100 so that the image of the food material to be recognized among the images taken by the camera 113 displayed on the touch panel 117 is within the range indicated by the range image 545.
  • the user presses the scan button 549.
  • the terminal control unit 130 extracts feature points from the captured image of the range indicated by the range image 545, and compares the extracted feature points with the feature points of the food material stored in advance. Determine the type of ingredients.
  • FIG. 12 is a diagram showing an example of a selection screen 560 displayed on the touch panel 117 when image recognition is executed on the scan screen 540 shown in FIG. 11.
  • the food selection field 561 and the selection field 563 are displayed.
  • the food material selection field 561 the food material name of the candidate food material detected by the image recognition and the check box are displayed.
  • Check boxes are displayed in association with the displayed food name.
  • the user checks the corresponding check box.
  • buttons 563 are displayed in the selection field 563.
  • the first is a button 563A to be pressed when registering the foodstuff selected by the check box in inventory management.
  • the second is the button 563B that is pressed when there is no corresponding ingredient.
  • the third is the button 563C that is pressed when the image recognition is performed again.
  • the terminal control unit 130 displays, for example, a major classification of vegetables, meat, fish, fruits, etc. and a minor classification of tomatoes, lettuce, Japanese mustard spinach, etc. on the touch panel 117. The user checks the corresponding items in the major classification and the minor classification.
  • FIG. 13 shows a confirmation screen 570 displayed when the check box of the tomato displayed in the food selection field 561 is checked in the selection screen 560 shown in FIG. 12 and the process of registering in inventory management is selected. It is a figure which shows an example.
  • the terminal control unit 130 determines that the use of the code information is inventory management of the scanned foodstuff.
  • the terminal control unit 130 transmits an acquisition request for food material management information including information indicating the type of food material and the user ID 145 to the server device 300, and acquires the food material management information for the food material from the server device 300.
  • a display field 571 for the food material name to be registered in inventory management a display field 571 for food material information, and a registration button 573 are displayed.
  • the food information display field 571 the expiration date 571A and the storage method 571B are displayed.
  • the registration button 573 is a button for registering in inventory management.
  • the terminal control unit 130 registers the foodstuff displayed in the display field 551 in inventory management. That is, the number data for managing the number of ingredients displayed in the display field 551 is added by 1.
  • FIG. 14 is a diagram showing an example of a scan screen 580 displayed on the touch panel 117 when the third process display 515 is selected.
  • the scan screen 580 of the image recognition display is displayed on the touch panel 117.
  • a switching display 581, a guidance display 583, a range image 585, and a scan button 587 are displayed on the scan screen 580.
  • the scan screen 580 is changed to the image recognition display.
  • the scan screen 580 is the same screen as the scan screen 540 shown in FIG.
  • FIG. 15 is a diagram showing an example of the search result screen 590.
  • the terminal control unit 130 displays the search result screen 590 shown in FIG.
  • the search result screen 590 shown in FIG. 15 shows a search result screen 590 when an onion is detected by image recognition.
  • the terminal control unit 130 transmits a recipe acquisition request using the detected onion to the server device 300.
  • the server device 300 searches the database 323, reads out the recipe data of the recipe using the onion, and transmits the read recipe data to the corresponding terminal device 100.
  • [1-2. motion] 16 to 18 are flowcharts showing the operation of the terminal device 100. The operation of the terminal device 100 will be described with reference to the flowcharts of FIGS. 16 to 18.
  • the terminal control unit 130 executes the service application 143 and causes the TE and the touch panel 117 to display the home screen 50. After that, the terminal control unit 130 determines whether or not the scan icon 53 has been operated (step S1). When the scan icon 53 is not operated (step S1 / NO), the terminal control unit 130 waits for the start of processing until the scan icon 53 is operated.
  • the terminal control unit 130 displays the process selection screen 510 shown in FIG. 5 on the touch panel 117 (step S2). Next, the terminal control unit 130 determines whether or not transmission of heating information, which is the first process, is selected (step S3). The terminal control unit 130 determines whether or not the first process display 511 is selected, and determines whether or not the first process is selected.
  • the terminal control unit 130 changes the display mode of the scan screen 520 shown in FIG. 6 to a code recognition display (step S4), and changes the operation mode to the code recognition mode. change. Then, the terminal control unit 130 determines whether or not the one-dimensional code or the two-dimensional code is detected from the captured image of the camera 113 (step S5). When the one-dimensional code or the two-dimensional code cannot be detected (step S5 / NO), the terminal control unit 130 repeats the determination in step S5 until the one-dimensional code or the two-dimensional code is detected. Step S5 corresponds to the detection procedure.
  • the terminal control unit 130 acquires the detected one-dimensional code or the data included in the two-dimensional code.
  • the terminal control unit 130 determines that the use of the code information is the acquisition of heating information.
  • the terminal control unit 130 transmits the acquired data to the server device 300, and acquires the food heating information indicated by the identification data from the server device 300 (step S6).
  • the terminal control unit 130 displays the acquired heating information on the selection screen 530 shown in FIG. 7 and displays a list of the heating information that can be set in the cooker 200 (step S7).
  • Steps S5 and S6 correspond to the first determination procedure and the transmission procedure. Further, step S7 corresponds to a display procedure.
  • the terminal control unit 130 determines whether or not one of the heating information has been selected (step S8).
  • the terminal control unit 130 does not accept the operation of selecting the heating information (step S8 / NO)
  • the terminal control unit 130 waits for the start of the process until the heating information is selected.
  • the terminal control unit 130 determines whether or not the transmission button 535 shown in FIG. 7 is pressed (step S9). When the transmission button 535 is not pressed (step S9 / NO), the terminal control unit 130 waits until the transmission button 535 is pressed. When the transmission button 535 is pressed (step S9 / YES), the terminal device 100 transmits the selected heating information to the server device 300 (step S10).
  • the terminal control unit 130 determines whether or not a second process, that is, a process for managing the inventory of scanned foods and foodstuffs, has been selected (step S11).
  • the terminal control unit 130 determines whether or not the second process display 513 is selected, and determines whether or not the second process is selected.
  • the terminal control unit 130 displays the scan screen 540 shown in FIG.
  • the terminal control unit 130 displays the scan screen 540 of the code recognition display set by default (step S12).
  • the terminal control unit 130 determines whether or not the operation of changing the display of the scan screen 540 is accepted by the operation of the switching display 541 (step S13).
  • the terminal control unit 130 determines whether or not the one-dimensional code or the two-dimensional code is detected from the captured image of the camera 113 (step S13 / NO).
  • Step S14 corresponds to the detection procedure.
  • step S14 When the one-dimensional code or the two-dimensional code is not detected (step S14 / NO), the terminal control unit 130 repeats the determination in step S14 until the one-dimensional code or the two-dimensional code is detected.
  • the terminal control unit 130 acquires the identification data included in the detected one-dimensional code or the two-dimensional code.
  • the terminal control unit 130 determines that the use of the code information is inventory management of the scanned food or foodstuff.
  • the terminal control unit 130 transmits a food management information acquisition request including the food identification data and the user ID 145 to the server device 300, and acquires the food food management information of the food corresponding to the identification data from the server device 300 (step). S15).
  • step S16 When the terminal control unit 130 acquires the food management information from the server device 300, the terminal control unit 130 displays the acquired food management information on the confirmation screen 550 shown in FIG. 9 (step S16). On the confirmation screen 550, information such as the expiration date and the storage method included in the food management information acquired from the server device 300 is displayed. Steps S14 and S15 correspond to the first determination procedure and the transmission procedure. Further, step S16 corresponds to a display procedure.
  • the terminal control unit 130 determines whether or not the registration button 555 displayed on the confirmation screen 550 has been pressed (step S17).
  • the terminal control unit 130 stores food management information as inventory management data stored by the terminal storage unit 140 (step S18).
  • the terminal control unit 130 causes the touch panel 117 to display the home screen 50, and returns to the determination in step S1.
  • step S13 when the terminal control unit 130 accepts the operation of changing the display of the scan screen 540 by the operation of the switching display 541 in step S13 (step S13 / YES), the display mode of the scan screen 520 is shown in FIG. Change to the image recognition display shown and change the operation mode to the image recognition mode.
  • the terminal control unit 130 determines whether or not the scan button 549 is pressed (step S19). When the scan button 549 is not pressed (step S20 / NO), the terminal control unit 130 waits for the start of processing until the scan button 549 is pressed. Further, when the scan button 549 is pressed (step S19 / YES), the terminal control unit 130 extracts feature points from the captured image of the camera 113, and compares the extracted feature points with the feature points of the food material stored in advance. To determine the ingredients (step S20). Step S19 and step S20 correspond to the second determination procedure.
  • the terminal control unit 130 displays the candidate ingredients determined by image recognition on the touch panel 117 (step S21).
  • the terminal control unit 130 displays the selection screen 560 shown in FIG. On the selection screen 560, a food material selection field 561 and a selection field 563 are displayed.
  • the terminal control unit 130 determines whether or not the food has been selected (step S22).
  • the terminal control unit 130 transmits a request for acquiring the foodstuff management information of the corresponding foodstuff to the server device 300, and acquires the foodstuff management information from the server device 300 (step S25). ).
  • the terminal control unit 130 displays the acquired foodstuff management information on the confirmation screen 570 shown in FIG. 13 (step S26).
  • the confirmation screen 570 also displays information such as the expiration date and the storage method included in the food management information acquired from the server device 300.
  • the terminal control unit 130 determines whether or not the registration button 573 displayed on the confirmation screen 570 is pressed (step S27).
  • the terminal control unit 130 stores food material management information as inventory management data stored by the terminal storage unit 140 (step S28). After that, the terminal control unit 130 causes the touch panel 117 to display the home screen 50, and returns to the determination in step S1.
  • step S22 determines whether or not the operation of selecting the rescan is accepted without selecting the food (step S23).
  • the terminal control unit 130 returns to the process of step S12 and displays the scan screen 540.
  • the terminal control unit 130 when the terminal control unit 130 does not accept the operation of selecting the rescan (step S23 / NO), the terminal control unit 130 accepts the user's input (step S24). For example, the terminal control unit 130 displays a major classification of vegetables, meat, fish, fruits, etc. and a minor classification of tomatoes, lettuce, Japanese mustard spinach, etc. on the touch panel 117, and allows the user to display the corresponding items of the major classification and the minor classification. Is entered in. After that, the terminal control unit 130 shifts to the process of step S23 and acquires the food material management information from the server device 300.
  • the terminal control unit 130 determines whether or not the process of searching for the recipe including the scanned foodstuff, which is the third process, is selected (step S29).
  • the terminal control unit 130 determines whether or not the third process display 515 is selected, and determines whether or not the third process is selected.
  • step S29 / NO the terminal control unit 130 returns to the determination in step S3 shown in FIG. Further, when the third process is selected (step S29 / YES), the terminal control unit 130 shifts the operation mode to the image confirmation mode and displays the scan screen 540 of the image recognition display shown in FIG.
  • steps S30 to S36 shown in FIG. 18 are the same as the operations of steps S19 to S24 shown in FIG. 17, the description of the operations of steps S30 to S36 will be omitted.
  • the terminal control unit 130 specifies the foodstuff
  • the terminal control unit 130 sends a recipe acquisition request including the specified foodstuff to the server device 300, and acquires the recipe data from the server device 300 (step S37).
  • the terminal control unit 130 displays the search result screen 590 shown in FIG. 15 based on the acquired recipe data.
  • the acquired recipe data is displayed in a list on the search result screen 590 (step S38).
  • the terminal control unit 130 determines whether or not the operation of selecting the recipe has been accepted (step S39).
  • the terminal control unit 130 accepts the operation of selecting a recipe (step S39 / YES)
  • the terminal control unit 130 displays the selected recipe on the touch panel 117 (step S40).
  • the terminal control unit 130 determines whether or not the end operation is accepted (step S41). For example, the terminal control unit 130 determines whether or not the close button displayed on the display screen has been pressed, and determines whether or not the end operation has been accepted.
  • step S41 / YES When the terminal control unit 130 accepts the end operation (step S41 / YES), the terminal control unit 130 displays the home screen 50 on the touch panel 117 (step S42), and returns to the determination in step S1. Further, when the terminal control unit 130 does not accept the end operation (step S41 / NO), the terminal control unit 130 returns to the determination in step S39.
  • FIG. 19 is a flowchart showing the operation of the server device 300.
  • the server control unit 310 determines whether or not data has been received from the terminal device 100 (step T1).
  • the server control unit 310 waits until the data is received.
  • the server control unit 310 determines whether or not the received data includes a request for acquisition of heating information (step T2).
  • the server control unit 310 reads the heating information of the corresponding food from the database 323 (step T3).
  • the server control unit 310 transmits the read heating information to the terminal device 100 that has received the data (step T10).
  • the server control unit 310 determines whether or not the received data includes the food management information acquisition request (step). T4).
  • the server control unit 310 reads the food management information of the corresponding food from the database 323 (step T5).
  • the server control unit 310 transmits the read food management information to the terminal device 100 that has received the data (step T10).
  • the server control unit 310 determines whether or not the received data includes the food management information acquisition request (step T4 / NO). Step T6).
  • the server control unit 310 reads the foodstuff management information of the corresponding foodstuff from the database 323 (step T7).
  • the server control unit 310 transmits the read food material management information to the terminal device 100 that has received the data (step T10).
  • the server control unit 310 determines whether or not the received data includes the request for acquiring the recipe data (step). T8).
  • the server control unit 310 reads the recipe data including the designated ingredients included in the data from the database 323 (step T9).
  • the server control unit 310 transmits the read recipe data to the terminal device 100 that has received the data (step T10).
  • the server control unit 310 executes the process corresponding to the received data (step T11), and ends the process flow.
  • the service application 143 causes the terminal control unit 130 to execute the detection procedure, the first determination procedure, the transmission procedure, and the display procedure.
  • the detection procedure is a procedure for detecting the code information included in the captured image captured by the camera 113.
  • the first determination procedure is a procedure for determining the use of the code information based on the operation received by the touch operation on the touch panel 117.
  • the transmission procedure is a procedure for transmitting an information acquisition request corresponding to the determined use of the code information to the server device 300.
  • the display procedure is a procedure for displaying the information received from the server device 300.
  • the use of the code information is determined based on the received operation, and the information corresponding to the determined use can be acquired from the server device 300 and displayed. Therefore, even when the same code information is read, the information corresponding to the operation can be displayed on the display unit, and the convenience of the user can be improved.
  • the transmission procedure when the use of the code information determined by the first determination procedure is cooking for heating food, a request for acquisition of heating information regarding cooking is transmitted to the server device 300.
  • the transmission procedure when the use of the code information determined by the first determination procedure is food inventory management, a request for acquiring information indicating the expiration date of the food indicated by the code information is transmitted to the server device 300.
  • the use of the code information is food inventory management, it is possible to acquire information indicating the expiration date of the food and display it on the terminal device 100.
  • the service application 143 causes the terminal control unit 130 to execute the second determination procedure for determining the food material captured in the captured image.
  • the transmission procedure transmits a request for acquisition of information indicating the expiration date of the food material determined by the second determination procedure to the server device 300.
  • the terminal control unit 130 When the terminal control unit 130 acquires the code information on the scan screens 520 and 540 of the code recognition display, the terminal control unit 130 displays the process selection screen on which the first process display 511 and the second process display 513 are displayed, and causes the user to perform any process. Lets you choose whether to execute. Further, when the terminal control unit 130 determines the foodstuff on the scan screen 540 and 580 of the image recognition display, the terminal control unit 130 causes the user to display the process selection screen on which the second process display 513 and the third process display 515 are displayed. Lets you choose whether to execute the process.
  • the function of the terminal control unit 130 may be realized by a plurality of processors or semiconductor chips.
  • the image recognition unit 153 may specify the food material by image recognition by AI. For example, a machine learning process is performed to learn features such as the color and shape of foodstuffs from a photographed image of a foodstuff for a teacher, and a trained model is constructed. This trained model is stored in the terminal storage unit 140. When the captured image of the camera 113 is input, the image recognition unit 153 refers to the trained model and determines the food material contained in the captured image. Further, the above-mentioned trained model may be stored in the server storage unit 320, and the server device 300 may be configured to determine the foodstuff. In this case, when the captured image of the camera 113 is input, the image recognition unit 153 transmits the input captured image to the server device 300, and acquires the determination result of the foodstuff from the server device 300.
  • AI image recognition by AI. For example, a machine learning process is performed to learn features such as the color and shape of foodstuffs from a photographed image of a foodstuff for a teacher, and a trained model
  • FIG. 2 Each part shown in FIG. 2 is an example, and the specific mounting form is not particularly limited. That is, it is not always necessary to implement the hardware corresponding to each part individually, and it is of course possible to realize the function of each part by executing the program by one processor. Further, a part of the functions realized by the software in the above-described embodiment may be realized by the hardware, or a part of the functions realized by the hardware may be realized by the software.
  • the operation unit shown in the flowchart of FIGS. 16 to 18 is divided according to the main processing contents in order to facilitate understanding of the operation of the terminal device 100, and the processing unit is divided.
  • the present invention is not limited by the method or the name. It may be divided into more step units depending on the processing content. Further, one step unit may be divided so as to include more processes. Further, the order of the steps may be appropriately changed as long as it does not interfere with the gist of the present invention. The same applies to the flowchart showing the operation of the server device 300 shown in FIG.
  • the program and the terminal device of the present invention can be used for providing information related to cooking to the user.
  • Service provision system 10 Network 100 Terminal equipment 111 Terminal communication I / F 113 Camera 115 Display unit 117 Touch panel 130 Terminal control unit 140 Terminal storage unit 141 Control program 143 Service application 150 Terminal processor 151 Code detection unit 153 Image recognition unit 155 Data acquisition unit 200 Cooker 201 Cooker communication I / F 203 Operation unit 210 Cooking control unit 220 Cooking unit 300 Server device 301 Server communication I / F 310 Server control unit 320 Server storage unit 323 Database 323a User database

Abstract

Provided is a program which enhances the convenience of a user on the basis of code information imparted to food.　A service application 143 causes a terminal control unit 130 to execute: a detection step for detecting code information included in a captured image captured with a camera 113; a first determination step for determining the use of the code information on the basis of an operation received through a touch panel; a transmission step for transmitting, to a server device 300, an acquisition request for information corresponding to the determined use of the code information; and a display step for displaying the information received from the server device 300.

Description

プログラム、端末装置、及びサービス提供システムPrograms, terminals, and service provision systems
　本開示は、プログラム、端末装置、及びサービス提供システムに関する。 This disclosure relates to programs, terminal devices, and service provision systems.
　特許文献１は、食品に付与されたコード情報から食品情報を取得し、取得した食品情報に対応する電子レンジの加熱情報をサーバー装置から取得して、取得した加熱情報を電子レンジに送信する技術を開示している。 Patent Document 1 is a technique of acquiring food information from code information given to food, acquiring heating information of a microwave oven corresponding to the acquired food information from a server device, and transmitting the acquired heating information to a microwave oven. Is disclosed.
特開２００６－２７５４８９号公報Japanese Unexamined Patent Publication No. 2006-275489
　本開示は、食品に付与されているコード情報に基づいて、ユーザーの利便性を向上させることを目的とする。 The purpose of this disclosure is to improve the convenience of the user based on the code information given to the food.
　本開示におけるプログラムは、コンピューターに、撮影部により撮影された撮影画像に含まれるコード情報を検出する検出手順と、操作部が受け付けた操作に基づいて前記コード情報の用途を判定する第１判定手順と、判定した前記コード情報の用途に対応した情報の取得要求を、サーバー装置に送信する送信手順と、前記サーバー装置から受信した情報を表示させる表示手順と、を実行させる。 The program in the present disclosure includes a detection procedure for detecting code information included in a captured image captured by a photographing unit on a computer, and a first determination procedure for determining the use of the code information based on an operation accepted by the operating unit. The transmission procedure of transmitting the information acquisition request corresponding to the determined use of the code information to the server device and the display procedure of displaying the information received from the server device are executed.
　本開示における端末装置は、表示部と、通信部と、撮影部と、前記撮影部が撮影した画像に含まれるコード情報を検出するコード検出部と、操作を受け付ける操作部と、前記操作部が受け付けた操作に基づいて前記コード情報の用途を判定し、判定した用途に対応した情報の取得要求を、前記通信部を介してサーバー装置に送信する制御部と、を備え、前記制御部は、前記サーバー装置から受信した情報を前記表示部に表示させる。 The terminal device in the present disclosure includes a display unit, a communication unit, a photographing unit, a code detecting unit that detects code information included in an image captured by the photographing unit, an operation unit that accepts an operation, and the operation unit. The control unit includes a control unit that determines the use of the code information based on the received operation and transmits an information acquisition request corresponding to the determined use to the server device via the communication unit. The information received from the server device is displayed on the display unit.
　本開示におけるサービス提供システムは、表示部と、通信部と、撮影部と、前記撮影部が撮影した画像に含まれるコード情報を検出するコード検出部と、操作を受け付ける操作部と、前記操作部が受け付けた操作に基づいて前記コード情報の用途を判定し、判定した用途に対応した情報の取得要求を、前記通信部を介してサーバー装置に送信する制御部と、を備える端末装置と、前記取得要求に対応した情報を前記端末装置に送信する前記サーバー装置と、を備え、前記制御部は、前記サーバー装置から受信した情報を前記表示部に表示させる。
　なお、この明細書には、２０２０年７月６日に出願された日本国特許出願・特願２０２０－１１６６８５号の全ての内容が含まれるものとする。 The service providing system in the present disclosure includes a display unit, a communication unit, a photographing unit, a code detection unit that detects code information included in an image captured by the photographing unit, an operation unit that accepts operations, and the operation unit. A terminal device including a control unit that determines the use of the code information based on the operation accepted by the user and transmits an information acquisition request corresponding to the determined use to the server device via the communication unit, and the above. The server device includes the server device that transmits information corresponding to the acquisition request to the terminal device, and the control unit causes the display unit to display the information received from the server device.
It should be noted that this specification shall include all the contents of the Japanese patent application / Japanese Patent Application No. 2020-116685 filed on July 6, 2020.
　本開示におけるプログラム、端末装置、及びサービス提供システムは、受け付けた操作に基づいてコード情報の用途が判定され、判定された用途に対応した情報をサーバー装置から取得して表示させることができる。このため、同一のコード情報を読み取った場合であっても、操作に対応した情報を表示部に表示させることができ、ユーザーの利便性を向上させることができる。 The program, terminal device, and service providing system in the present disclosure determine the use of the code information based on the received operation, and can acquire and display the information corresponding to the determined use from the server device. Therefore, even when the same code information is read, the information corresponding to the operation can be displayed on the display unit, and the convenience of the user can be improved.
図１は、サービス提供システムのシステム構成図FIG. 1 is a system configuration diagram of a service providing system. 図２は、端末装置、調理器及びサーバー装置の構成を示すブロック構成図FIG. 2 is a block configuration diagram showing the configurations of a terminal device, a cooker, and a server device. 図３は、ユーザーデータベースの構成を示す図FIG. 3 is a diagram showing the configuration of the user database. 図４は、ホーム画面の一例を示す図FIG. 4 is a diagram showing an example of a home screen. 図５は、処理選択画面の一例を示す図FIG. 5 is a diagram showing an example of a process selection screen. 図６は、スキャン画面の一例を示す図FIG. 6 is a diagram showing an example of a scan screen. 図７は、選択画面の一例を示す図FIG. 7 is a diagram showing an example of a selection screen. 図８は、スキャン画面の一例を示す図FIG. 8 is a diagram showing an example of a scan screen. 図９は、確認画面の一例を示す図FIG. 9 is a diagram showing an example of a confirmation screen. 図１０は、スキャン画面の一例を示す図FIG. 10 is a diagram showing an example of a scan screen. 図１１は、スキャン画面の一例を示す図FIG. 11 is a diagram showing an example of a scan screen. 図１２は、選択画面の一例を示す図FIG. 12 is a diagram showing an example of a selection screen. 図１３は、確認画面の一例を示す図FIG. 13 is a diagram showing an example of a confirmation screen. 図１４は、スキャン画面の一例を示す図FIG. 14 is a diagram showing an example of a scan screen. 図１５は、検索結果画面の一例を示す図FIG. 15 is a diagram showing an example of a search result screen. 図１６は、端末装置の動作を示すフローチャートFIG. 16 is a flowchart showing the operation of the terminal device. 図１７は、Ｍ端末装置の動作を示すフローチャートFIG. 17 is a flowchart showing the operation of the M terminal device. 図１８は、端末装置の動作を示すフローチャートFIG. 18 is a flowchart showing the operation of the terminal device. 図１９は、サーバー装置の動作を示すフローチャートFIG. 19 is a flowchart showing the operation of the server device.
　以下、図面を参照しながら実施の形態を詳細に説明する。但し、必要以上に詳細な説明を省略する場合がある。例えば、既によく知られた事項の詳細説明、または、実質的に同一の構成に対する重複説明を省略する場合がある。
　なお、添付図面および以下の説明は、当業者が本開示を十分に理解するために提供されるのであって、これらにより特許請求の範囲に記載の主題を限定することを意図していない。 Hereinafter, embodiments will be described in detail with reference to the drawings. However, more detailed explanations may be omitted than necessary. For example, detailed explanations of already well-known matters or duplicate explanations for substantially the same configuration may be omitted.
It should be noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.
　（実施の形態１）
　以下、図１－図１９を用いて、実施の形態１を説明する。
　［１－１．構成］
　図１は、サービス提供システム１のシステム構成である。
　サービス提供システム１は、端末装置１００と、調理器２００と、サーバー装置３００とがネットワーク１０を介して接続された構成を有する。ネットワーク１０は、有線ネットワークであってもよく、一部に無線ネットワークが含まれていてもよい。ネットワーク１０は、例えば、インターネットや、移動体通信網、ＷＡＮ（Wide Area Network）等の広域ネットワークを含む構成としてもよいし、イントラネットやＬＡＮ（Local Area Network）等のプライベートネットワークにより構成してもよい。また、端末装置１００及び調理器２００のネットワーク１０への接続形態は、有線であってもよいし、Ｗｉ－Ｆｉ（登録商標）等の無線であってもよい。 (Embodiment 1)
Hereinafter, the first embodiment will be described with reference to FIGS. 1 to 19.
[1-1. composition]
FIG. 1 is a system configuration of the service providing system 1.
The service providing system 1 has a configuration in which a terminal device 100, a cooker 200, and a server device 300 are connected via a network 10. The network 10 may be a wired network, or may include a wireless network in part. The network 10 may be configured to include, for example, an Internet, a mobile communication network, a wide area network such as a WAN (Wide Area Network), or a private network such as an intranet or a LAN (Local Area Network). .. Further, the connection form of the terminal device 100 and the cooker 200 to the network 10 may be wired or wireless such as Wi-Fi (registered trademark).
　端末装置１００は、可搬型の情報処理装置であり、サービスの提供を受けるユーザーが所持する端末である。端末装置１００には、例えば、スマートフォンや、タブレット型ＰＣ（Personal Computer）、ノート型ＰＣ、デスクトップＰＣ、携帯電話機、ＰＤＡ（Personal Digital  Assistant）等を採用可能である。本実施の形態では、端末装置１００がスマートフォンである場合を例に挙げて説明する。 The terminal device 100 is a portable information processing device, and is a terminal owned by a user who receives a service. For the terminal device 100, for example, a smartphone, a tablet PC (Personal Computer), a notebook PC, a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like can be adopted. In the present embodiment, a case where the terminal device 100 is a smartphone will be described as an example.
　端末装置１００には、サービスアプリケーションプログラム（以下、簡単にサービスアプリ１４３という）がインストールされている。このサービスアプリ１４３は、ユーザーが端末装置１００を操作してサーバー装置３００からダウンロードしてもよいし、サーバー装置３００以外の他のサーバー装置からダウンロードしてもよい。 A service application program (hereinafter, simply referred to as a service application 143) is installed in the terminal device 100. The service application 143 may be downloaded from the server device 300 by the user operating the terminal device 100, or may be downloaded from a server device other than the server device 300.
　端末装置１００は、サービスアプリ１４３を実行して、ユーザーにサービスを提供する。サービスアプリ１４３が実行されることにより提供されるサービスには、例えば、冷凍食品等の食品を調理器２００により加熱調理する場合に、定格高周波出力や、加熱時間等の加熱情報を調理器２００に設定したり、ユーザーが購入した食品や食材の在庫管理を行ったり、ユーザーにより選択された食材を含むレシピを検索したりするサービスが含まれる。 The terminal device 100 executes the service application 143 to provide a service to the user. The service provided by executing the service application 143 includes, for example, when food such as frozen food is cooked by the cooker 200, heating information such as rated high frequency output and heating time is transmitted to the cooker 200. It includes services such as setting, managing the inventory of foods and ingredients purchased by the user, and searching for recipes containing the ingredients selected by the user.
　調理器２００は、加熱調理を行う調理機器である。本実施の形態では、調理器２００がオーブンレンジである場合について説明するが、調理器２００は、例えば、炊飯器であってもよいし、電気圧力鍋であってもよい。
　調理器２００は、ネットワーク１０を介してサーバー装置３００に接続される。調理器２００は、サーバー装置３００から受信したデータに基づき、加熱調理の調理設定を行う。調理器２００は、調理設定が行われた状態で、ユーザーにより調理の開始ボタンが押下されると、設定された調理設定に従って、加熱調理を開始する。
　なお、本実施の形態では、端末装置１００と、調理器２００とが直接通信を行うことはないが、例えば、Ｂｌｕｅｔｏｏｔｈ（登録商標）等の近距離無線通信により端末装置１００と、調理器２００とが直接通信を行い、データを送受信する構成であってもよい。 The cooker 200 is a cooking device that performs cooking. In the present embodiment, the case where the cooker 200 is an oven range will be described, but the cooker 200 may be, for example, a rice cooker or an electric pressure cooker.
The cooker 200 is connected to the server device 300 via the network 10. The cooker 200 sets cooking for cooking based on the data received from the server device 300. When the user presses the cooking start button while the cooking setting is made, the cooker 200 starts cooking according to the set cooking setting.
In the present embodiment, the terminal device 100 and the cooker 200 do not directly communicate with each other. However, for example, the terminal device 100 and the cooker 200 can be communicated with each other by short-range wireless communication such as Bluetooth (registered trademark). May be configured to directly communicate and send / receive data.
　サーバー装置３００は、登録されたユーザーの所持する端末装置１００に、情報を提供する。サーバー装置３００が提供する情報には、例えば、調理器２００の加熱情報に関する情報や、食品の利用期限、食材の保存方法や賞味期限、レシピ等が含まれる。
　また、サーバー装置３００は、端末装置１００から加熱情報の情報を受信すると、情報を受信した端末装置１００に対応づけられた調理器２００に、受信した加熱情報を送信する。すなわち、サーバー装置３００は、端末装置１００のユーザーにより選択された加熱情報を、ユーザーが利用する調理器２００に送信し、調理器２００の調理設定を行う。 The server device 300 provides information to the terminal device 100 owned by the registered user. The information provided by the server device 300 includes, for example, information on heating information of the cooker 200, a food expiration date, a food storage method, a best-by date, a recipe, and the like.
Further, when the server device 300 receives the heating information information from the terminal device 100, the server device 300 transmits the received heating information to the cooker 200 associated with the terminal device 100 that has received the information. That is, the server device 300 transmits the heating information selected by the user of the terminal device 100 to the cooker 200 used by the user, and sets the cooking of the cooker 200.
　図２は、端末装置１００、調理器２００及びサーバー装置３００の構成を示すブロック構成図である。まず、端末装置１００について説明する。
　端末装置１００は、端末通信Ｉ／Ｆ１１１、カメラ１１３、表示部１１５及び端末制御部１３０を備える。 FIG. 2 is a block configuration diagram showing the configurations of the terminal device 100, the cooker 200, and the server device 300. First, the terminal device 100 will be described.
The terminal device 100 includes a terminal communication I / F 111, a camera 113, a display unit 115, and a terminal control unit 130.
　端末通信Ｉ／Ｆ１１１は、通信モジュールを備える通信インターフェイスであり、ネットワーク１０を介してサーバー装置３００と相互にデータ通信を行う。端末通信Ｉ／Ｆ１１１の通信方式には、例えば、４Ｇ（Generation）、５Ｇ、Ｗｉ－Ｆｉ等の無線通信方式が採用される。端末通信Ｉ／Ｆ１１１は、通信部の一例に相当する。 The terminal communication I / F 111 is a communication interface provided with a communication module, and performs data communication with the server device 300 via the network 10. As the communication method of the terminal communication I / F111, for example, a wireless communication method such as 4G (Generation), 5G, Wi-Fi, etc. is adopted. The terminal communication I / F 111 corresponds to an example of the communication unit.
　カメラ１１３は、レンズ及び撮像素子を備える。撮像素子は、例えば、ＣＣＤ（Charged Coupled Device）又はＣＭＯＳ（Complementary Metal Oxide Semiconductor）等のイメージセンサーである。カメラ１１３は撮影によって撮像画像を生成し、生成した撮影画像を端末制御部１３０に出力する。カメラ１１３は、撮影部の一例に相当する。 The camera 113 includes a lens and an image sensor. The image sensor is, for example, an image sensor such as a CCD (Charged Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 113 generates a captured image by shooting, and outputs the generated captured image to the terminal control unit 130. The camera 113 corresponds to an example of a photographing unit.
　表示部１１５は、液晶や有機ＥＬ（Electronic Luminescent）等の表示パネルと、タッチ操作を検出するタッチセンサーとを備えるタッチパネル１１７を備える。タッチパネル１１７は、操作部の一例に相当する。タッチセンサーは、ユーザーの指や、ペン等の指示体が接触した表示パネルの位置を示す座標情報を端末制御部１３０に出力する。この座標情報は、表示パネルの予め設定された座標系での座標である。 The display unit 115 includes a touch panel 117 including a display panel such as a liquid crystal display or an organic EL (Electronic Luminescent) and a touch sensor for detecting a touch operation. The touch panel 117 corresponds to an example of an operation unit. The touch sensor outputs coordinate information indicating the position of the display panel to which the user's finger or an indicator such as a pen is in contact with the terminal control unit 130. This coordinate information is the coordinates in the preset coordinate system of the display panel.
　端末制御部１３０は、端末記憶部１４０及び端末プロセッサー１５０を備えるコンピューターである。 The terminal control unit 130 is a computer including a terminal storage unit 140 and a terminal processor 150.
　端末記憶部１４０は、例えば、フラッシュメモリーや、ＳＤカード等の不揮発性の記憶装置と、ＲＡＭ（Random access memory）等の揮発性の記憶装置とを備える。
　端末記憶部１４０は、ＯＳ（Operating System）等の制御プログラム１４１や、サービスアプリ１４３、ユーザーＩＤ（Identifier）１４５、その他各種の設定データを記憶する。ユーザーＩＤ１４５は、端末装置１００のユーザーにより設定された情報であり、サービス提供システム１の利用者を識別する識別情報である。また、端末記憶部１４０には、ユーザーが購入した食品や食材の在庫管理データが記憶される。在庫管理データには、例えば、食品や食材の種類（名称）、個数、利用期限、賞味期限、保存方法等のデータが含まれる。 The terminal storage unit 140 includes, for example, a non-volatile storage device such as a flash memory or an SD card, and a volatile storage device such as a RAM (Random access memory).
The terminal storage unit 140 stores a control program 141 such as an OS (Operating System), a service application 143, a user ID (Identifier) 145, and various other setting data. The user ID 145 is information set by the user of the terminal device 100, and is identification information for identifying the user of the service providing system 1. Further, the terminal storage unit 140 stores inventory management data of foods and foodstuffs purchased by the user. The inventory management data includes, for example, data such as the type (name) of food or foodstuff, the number of foods, the expiration date, the expiration date, and the storage method.
　端末プロセッサー１５０は、ＣＰＵ（Central Processing Unit）や、ＣＰＵを搭載したＭＣＵ（Micro Controller Unit）や、ＭＰＵ（Micro Processor Unit）等のマイコンにより構成される。また、端末プロセッサー１５０を、端末通信Ｉ／Ｆ１１１やカメラ、各種センサー、メモリーを備えるＳｏＣ（System-on-a-Chip）の一部として構成してもよい。 The terminal processor 150 is composed of a CPU (Central Processing Unit), a microcomputer equipped with a CPU (MicroControllerUnit), an MPU (MicroProcessorUnit), and other microcomputers. Further, the terminal processor 150 may be configured as a part of a SoC (System-on-a-Chip) including a terminal communication I / F 111, a camera, various sensors, and a memory.
　端末制御部１３０は、機能的構成として、コード検出部１５１、画像認識部１５３、データ取得部１５５及び表示制御部１５７を備える。これらは、端末プロセッサー１５０が、端末記憶部１４０に記憶された制御プログラム１４１や、サービスアプリ１４３を実行することに実現される機能である。 The terminal control unit 130 includes a code detection unit 151, an image recognition unit 153, a data acquisition unit 155, and a display control unit 157 as functional configurations. These are functions realized by the terminal processor 150 executing the control program 141 stored in the terminal storage unit 140 and the service application 143.
　コード検出部１５１は、カメラ１１３の撮影画像からコード情報を検出する。コード検出部１５１が取得するコード情報には、例えば、ＪＡＮコード等の一次元コードや、ＱＲコード(登録商標)等の二次元コードが含まれる。コード検出部１５１は、撮影画像から一次元コードや二次元コード等のコード情報を検出し、検出したコード情報に含まれる食品を識別する識別データを取得する。コード検出部１５１が動作する場合の端末制御部１３０の動作モードを、コード認識モードという。 The code detection unit 151 detects the code information from the captured image of the camera 113. The code information acquired by the code detection unit 151 includes, for example, a one-dimensional code such as a JAN code and a two-dimensional code such as a QR code (registered trademark). The code detection unit 151 detects code information such as a one-dimensional code and a two-dimensional code from the captured image, and acquires identification data for identifying foods contained in the detected code information. The operation mode of the terminal control unit 130 when the code detection unit 151 operates is called a code recognition mode.
　画像認識部１５３は、カメラ１１３の撮影画像に撮影された物体を特定する。例えば、画像認識部１５３は、撮影画像に含まれる形状や、色、大きさ等の特徴点を抽出し、抽出した特徴点と、予め記憶した食材の特徴点とを比較して、食材の種類を判定する。画像認識部１５３が動作する場合の端末制御部１３０の動作モードを、画像認識モードという。 The image recognition unit 153 identifies an object captured in the captured image of the camera 113. For example, the image recognition unit 153 extracts feature points such as shape, color, and size included in the captured image, compares the extracted feature points with the feature points of the foodstuff stored in advance, and types of the foodstuff. Is determined. The operation mode of the terminal control unit 130 when the image recognition unit 153 operates is called an image recognition mode.
　データ取得部１５５は、端末通信Ｉ／Ｆ１１１を制御して、サーバー装置３００にデータの取得要求を送信し、サーバー装置３００からデータを取得する。
　データ取得部１５５は、カメラ１１３の撮影画像から食品を識別する識別データを含むコード情報をコード検出部１５１が検出した場合であって、タッチパネル１１７により検出された操作が、調理器２００に加熱情報を送信する処理を選択する操作である場合、コード情報の用途を、加熱情報の取得と判定する。この場合、データ取得部１５５は、食品の識別データと、ユーザーＩＤ１４５とを含む加熱情報の取得要求を、サーバー装置３００に送信して、加熱情報を含むデータを取得する。 The data acquisition unit 155 controls the terminal communication I / F 111, transmits a data acquisition request to the server device 300, and acquires data from the server device 300.
The data acquisition unit 155 is a case where the code detection unit 151 detects the code information including the identification data for identifying the food from the image taken by the camera 113, and the operation detected by the touch panel 117 is the heating information to the cooker 200. When the operation is to select the process of transmitting the code information, it is determined that the use of the code information is the acquisition of the heating information. In this case, the data acquisition unit 155 sends a request for acquisition of heating information including the food identification data and the user ID 145 to the server device 300, and acquires the data including the heating information.
　また、データ取得部１５５は、コード検出部１５１がカメラ１１３の撮影画像から食品の識別データを含むコード情報を検出した場合であって、タッチパネル１１７により検出された操作が、在庫管理を選択する操作である場合、コード情報の用途を、在庫管理と判定する。この場合、データ取得部１５５は、食品の識別データと、ユーザーＩＤ１４５とを含む在庫管理に必要な情報の取得要求を、サーバー装置３００に送信する。在庫管理に必要な情報には、食品の利用期限や、保存方法等が含まれる。以下、食品の在庫管理に必要な情報を、食品管理情報という。 Further, the data acquisition unit 155 is a case where the code detection unit 151 detects code information including food identification data from the captured image of the camera 113, and the operation detected by the touch panel 117 is an operation of selecting inventory management. If, the use of the code information is determined to be inventory management. In this case, the data acquisition unit 155 transmits to the server device 300 a request for acquisition of information necessary for inventory management, including food identification data and user ID 145. Information necessary for inventory control includes the expiration date of food, storage method, and the like. Hereinafter, the information necessary for food inventory management is referred to as food management information.
　また、データ取得部１５５は、画像認識部１５３が、食材の種類を判定した場合であって、タッチパネル１１７により検出された操作が、在庫管理を選択する操作であった場合、判定された食材の種類を示す情報と、ユーザーＩＤ１４５とを含む在庫管理に必要なデータの取得要求を、サーバー装置３００に送信する。在庫管理に必要な情報には、食材の利用期限や、保存方法等が含まれる。以下、食材の在庫管理に必要な情報を、食材管理情報という。 Further, when the image recognition unit 153 determines the type of the food material and the operation detected by the touch panel 117 is the operation of selecting the inventory management, the data acquisition unit 155 determines the type of the food material. A request for acquiring data necessary for inventory management, including information indicating the type and the user ID 145, is transmitted to the server device 300. Information necessary for inventory management includes the expiration date of ingredients, storage method, and the like. Hereinafter, the information necessary for inventory management of ingredients is referred to as foodstuff management information.
　また、データ取得部１５５は、画像認識部１５３が、食材の種類を判定した場合であって、タッチパネル１１７により検出された操作が、レシピ検索を選択する操作であった場合、判定された食材の種類を示す情報と、ユーザーＩＤ１４５とを含むレシピデータの取得要求を、サーバー装置３００に送信する。 Further, when the image recognition unit 153 determines the type of the food material and the operation detected by the touch panel 117 is the operation of selecting the recipe search, the data acquisition unit 155 determines the type of the food material. A request for acquiring recipe data including the type information and the user ID 145 is transmitted to the server device 300.
　表示制御部１５７は、表示部１１５を制御してタッチパネル１１７に表示画面を表示させる。表示制御部１５７は、表示データを生成し、生成した表示データを表示部１１５に出力する。表示部１１５は、入力された表示データに基づいて、表示画面をタッチパネル１１７に表示させる。例えば、表示制御部１５７は、サーバー装置３００から受信した食品のパッケージ画像や、加熱情報、食品の利用期限や、保存方法、レシピ等をタッチパネル１１７に表示させる。 The display control unit 157 controls the display unit 115 to display the display screen on the touch panel 117. The display control unit 157 generates display data, and outputs the generated display data to the display unit 115. The display unit 115 causes the touch panel 117 to display the display screen based on the input display data. For example, the display control unit 157 displays a food package image received from the server device 300, heating information, a food expiration date, a storage method, a recipe, and the like on the touch panel 117.
　調理器２００は、調理器通信Ｉ／Ｆ２０１、操作部２０３、表示部２０５、調理制御部２１０及び加熱調理部２２０を備える。調理器通信Ｉ／Ｆ２０１は、通信モジュールを備える通信インターフェイスである。調理器通信Ｉ／Ｆ２０１は、例えば、Ｗｉ－Ｆｉ等の無線通信によりサーバー装置３００とデータ通信を行う。 The cooker 200 includes a cooker communication I / F 201, an operation unit 203, a display unit 205, a cooking control unit 210, and a cooking unit 220. The cooker communication I / F 201 is a communication interface including a communication module. The cooker communication I / F 201 performs data communication with the server device 300 by wireless communication such as Wi-Fi.
　操作部２０３は、例えば、調理開始ボタンや、加熱時間、定格高周波出力等を設定する設定ボタン等を備える。操作部２０３は、調理開始ボタンや設定ボタンの操作を受け付けると、操作を受け付けたボタンに対応した操作信号を調理制御部２１０に出力する。 The operation unit 203 includes, for example, a cooking start button, a setting button for setting a heating time, a rated high frequency output, and the like. When the operation unit 203 receives the operation of the cooking start button or the setting button, the operation unit 203 outputs an operation signal corresponding to the button for which the operation is received to the cooking control unit 210.
　表示部２０５は、液晶パネル等の表示パネルを備え、調理制御部２１０の制御に従い、文字や画像を表示パネルに表示させる。例えば、表示パネルには、加熱調理する食品の画像や、加熱時間、定格高周波出力等の加熱情報が表示される。 The display unit 205 includes a display panel such as a liquid crystal panel, and displays characters and images on the display panel under the control of the cooking control unit 210. For example, an image of food to be cooked and heating information such as heating time and rated high frequency output are displayed on the display panel.
　調理制御部２１０は、操作部２０３で受け付けた操作や、サーバー装置３００から受信した加熱情報に従い加熱調理部２２０を制御し、加熱調理を実行する。
　また、調理制御部２１０は、操作部２０３で受け付けた操作内容や、サーバー装置３００から受信した加熱情報を、表示部２０５に表示させる。 The cooking control unit 210 controls the cooking unit 220 according to the operation received by the operation unit 203 and the heating information received from the server device 300, and executes cooking.
Further, the cooking control unit 210 causes the display unit 205 to display the operation content received by the operation unit 203 and the heating information received from the server device 300.
　加熱調理部２２０は、調理制御部２１０により設定された調理方法や、調理時間に従い、不図示の加熱室内に収容された被調理物を加熱調理する。調理方法には、レンジ調理、オーブン調理、スチーム調理、グリル調理等が含まれる。 The cooking unit 220 heats and cooks the food to be cooked in the heating chamber (not shown) according to the cooking method set by the cooking control unit 210 and the cooking time. Cooking methods include microwave cooking, oven cooking, steam cooking, grill cooking and the like.
　次に、サーバー装置３００の構成について説明する。
　サーバー装置３００は、サーバー通信Ｉ／Ｆ３０１及びサーバー制御部３１０を備える。
　サーバー通信Ｉ／Ｆ３０１は、例えば、ＮＩＣ（Network Interface Card）等のネットワークアダプタを備え、ネットワーク１０を介して端末装置１００や、調理器２００とデータ通信を行う。 Next, the configuration of the server device 300 will be described.
The server device 300 includes a server communication I / F 301 and a server control unit 310.
The server communication I / F 301 includes, for example, a network adapter such as a NIC (Network Interface Card), and performs data communication with the terminal device 100 and the cooker 200 via the network 10.
　サーバー制御部３１０は、サーバー記憶部３２０及びサーバープロセッサー３３０を備える。
　サーバー記憶部３２０は、例えば、ＲＯＭや、フラッシュメモリー等の不揮発性の記憶装置と、ハードディスクドライブや、ＳＳＤ（Solid State Drive）等の補助記憶装置により構成される。サーバー記憶部３２０は、制御プログラム３２１や、データベース３２３を記憶する。 The server control unit 310 includes a server storage unit 320 and a server processor 330.
The server storage unit 320 is composed of, for example, a non-volatile storage device such as a ROM or a flash memory, and an auxiliary storage device such as a hard disk drive or an SSD (Solid State Drive). The server storage unit 320 stores the control program 321 and the database 323.
　データベース３２３に登録されるデータには、登録された食品の加熱情報が含まれる。データベース３２３に登録される食品は、サービス提供システム１に参加した企業が提供する食品である。加熱情報には、定格高周波出力や、加熱時間の情報が含まれる。定格高周波出力や、加熱時間の情報は、例えば、食品が焼きおにぎりやから揚げ等の同時に調理する個数によって加熱時間が異なる場合、同時調理する個数ごとに設定される。
　また、データベース３２３に登録されるデータには、登録された食品のパッケージ画像や、利用期限、保存方法等のデータが含まれる。
　また、データベース３２３に登録されるデータには、野菜や肉、魚等の食材ごとの利用期限や、保存方法等のデータが含まれる。
　また、データベース３２３に登録されるデータには、料理のレシピデータが含まれる。 The data registered in the database 323 includes heating information of the registered food. The foods registered in the database 323 are foods provided by the companies participating in the service provision system 1. The heating information includes the rated high frequency output and the heating time information. The rated high frequency output and the information on the heating time are set for each number of foods to be cooked at the same time, for example, when the heating time differs depending on the number of foods to be cooked at the same time such as grilled rice balls and fried chicken.
Further, the data registered in the database 323 includes data such as a package image of the registered food, an expiration date, and a storage method.
In addition, the data registered in the database 323 includes data such as the expiration date for each ingredient such as vegetables, meat, and fish, and the storage method.
Further, the data registered in the database 323 includes cooking recipe data.
　図３は、データベース３２３に含まれるユーザーデータベース３２３ａの構成を示す図である。ユーザーデータベース３２３ａは、サービス提供システム１に登録されたユーザーの情報を登録したデータベースである。ユーザーデータベース３２３ａには、ユーザーの使用する調理器２００の種別や機種情報と、調理器２００の通信アドレスとが、ユーザーの識別情報であるユーザーＩＤ及びパスワードに対応づけて登録されている。 FIG. 3 is a diagram showing the configuration of the user database 323a included in the database 323. The user database 323a is a database in which information of users registered in the service providing system 1 is registered. In the user database 323a, the type and model information of the cooker 200 used by the user and the communication address of the cooker 200 are registered in association with the user ID and password which are the identification information of the user.
　サーバープロセッサー３３０は、例えば、ＣＰＵやＭＣＵ、ＭＰＵにより構成される。サーバー制御部３１０は、サーバープロセッサー３３０が、サーバー記憶部３２０に記憶された制御プログラム３２１を実行することにより、各種処理を実行する。 The server processor 330 is composed of, for example, a CPU, an MCU, and an MPU. The server control unit 310 executes various processes by the server processor 330 executing the control program 321 stored in the server storage unit 320.
　サーバー制御部３１０は、例えば、端末装置１００から加熱情報の取得要求を受信した場合、受信した取得要求に含まれる識別データに対応した食品のパッケージ画像や、加熱情報をデータベース３２３から読み出し、読み出したデータを端末装置１００に送信する。また、サーバー制御部３１０は、端末装置１００から食品の識別データと、ユーザーＩＤ１４５とを含む食品管理情報の取得要求を受信すると、受信した取得要求に含まれる識別データに対応した食品のパッケージ画像や、食品の利用期限や、保存方法等のデータをデータベース３２３から読み出し、読み出したデータを端末装置１００に送信する。 For example, when the server control unit 310 receives a heating information acquisition request from the terminal device 100, the server control unit 310 reads and reads the food package image corresponding to the identification data included in the received acquisition request and the heating information from the database 323. The data is transmitted to the terminal device 100. Further, when the server control unit 310 receives the acquisition request of the food management information including the food identification data and the user ID 145 from the terminal device 100, the server control unit 310 may receive a food package image corresponding to the identification data included in the received acquisition request. , Data such as the expiration date of the food and the storage method are read from the database 323, and the read data is transmitted to the terminal device 100.
　また、サーバー制御部３１０は、端末装置１００から食材の種類を示す情報と、ユーザーＩＤ１４５とを含む食材管理情報の取得要求を受信すると、該当する食材の利用期限や、保存方法等のデータをデータベース３２３から読み出し、読み出したデータを端末装置１００に送信する。また、サーバー制御部３１０は、端末装置１００から食材の種類を示す情報と、ユーザーＩＤ１４５とを含むレシピデータの取得要求を受信すると、受信した情報が示す食材を含むレシピデータをデータベース３２３から読み出し、読み出したデータを端末装置１００に送信する。 Further, when the server control unit 310 receives the acquisition request of the food material management information including the information indicating the type of the food material and the user ID 145 from the terminal device 100, the server control unit 310 stores data such as the expiration date of the corresponding food material and the storage method in the database. It is read from 323 and the read data is transmitted to the terminal device 100. Further, when the server control unit 310 receives the recipe data acquisition request including the information indicating the type of the food material and the user ID 145 from the terminal device 100, the server control unit 310 reads the recipe data including the food material indicated by the received information from the database 323. The read data is transmitted to the terminal device 100.
　また、サーバー制御部３１０は、端末装置１００から加熱情報の取得要求を受信した場合、端末装置１００を所持するユーザーのユーザーＩＤ１４５に対応づけられた調理器２００の通信アドレスを取得し、取得した通信アドレス宛に、加熱情報を送信する。 Further, when the server control unit 310 receives the heating information acquisition request from the terminal device 100, the server control unit 310 acquires the communication address of the cooker 200 associated with the user ID 145 of the user who owns the terminal device 100, and acquires the communication. Send heating information to the address.
　図４は、ホーム画面５０の一例を示す図である。
　端末制御部１３０は、ユーザーのタッチ操作によりサービスアプリ１４３が選択されると、サービスアプリ１４３を起動させて図４に示すホーム画面５０を表示させる。
　ホーム画面５０には、メニューアイコン５１、スキャンアイコン５３、通知画面５５，５７等が表示される。
　メニューアイコン５１は、このメニューアイコン５１が選択された場合に、メニューの一覧が表示されるアイコンである。
　スキャンアイコン５３は、カメラ１１３の撮影画像により食品に付されたコード情報や、食材をスキャンし、スキャン結果に基づく処理を実行するアイコンである。以下、カメラ１１３の撮影画像により食品に付されたコード情報や、食材を認識する処理をスキャンという。
　通知画面５５，５７には、お知らせや、オススメのレシピ等の情報が表示される。図４には、２つの通知画面５５，５７が表示された例を示すが、ホーム画面５０に表示される通知画面５５，５７の数は任意である。 FIG. 4 is a diagram showing an example of the home screen 50.
When the service application 143 is selected by the user's touch operation, the terminal control unit 130 activates the service application 143 and displays the home screen 50 shown in FIG.
The home screen 50 displays a menu icon 51, a scan icon 53, notification screens 55, 57, and the like.
The menu icon 51 is an icon that displays a list of menus when the menu icon 51 is selected.
The scan icon 53 is an icon that scans the code information attached to the food by the image taken by the camera 113 and the food and executes the process based on the scan result. Hereinafter, the process of recognizing the code information attached to the food and the food by the image taken by the camera 113 is referred to as scanning.
Information such as notifications and recommended recipes is displayed on the notification screens 55 and 57. FIG. 4 shows an example in which two notification screens 55 and 57 are displayed, but the number of notification screens 55 and 57 displayed on the home screen 50 is arbitrary.
　図５は、スキャンアイコン５３が選択された場合に、タッチパネル１１７に表示される処理選択画面５１０の一例を示す図である。
　処理選択画面５１０は、スキャンを利用して実行される処理の一覧を表示した画面である。処理選択画面５１０には、第１処理表示５１１、第２処理表示５１３及び第３処理表示５１５が選択可能に表示される。第１処理表示５１１は、第１処理表示５１１が表示されたタッチパネル１１７の領域を、ユーザーがタッチ操作することにより選択される。第２処理表示５１３及び第３処理表示５１５についても同様である。 FIG. 5 is a diagram showing an example of a process selection screen 510 displayed on the touch panel 117 when the scan icon 53 is selected.
The process selection screen 510 is a screen displaying a list of processes executed by using scanning. On the process selection screen 510, the first process display 511, the second process display 513, and the third process display 515 are displayed in a selectable manner. The first process display 511 is selected by the user touching the area of the touch panel 117 on which the first process display 511 is displayed. The same applies to the second process display 513 and the third process display 515.
　第１処理表示５１１には、第１処理が表示される。第１処理は、加熱情報を、調理器２００に送信する処理である。
　第２処理表示５１３には、第２処理が表示される。第２処理は、スキャンした食品や食材の在庫管理を行う処理である。
　第３処理表示５１５には、第３処理が表示される。第３処理は、スキャンした食材を含むレシピを検索する処理である。 The first process is displayed on the first process display 511. The first process is a process of transmitting heating information to the cooker 200.
The second process is displayed on the second process display 513. The second process is a process for managing the inventory of scanned foods and foodstuffs.
The third process is displayed on the third process display 515. The third process is a process of searching for a recipe containing the scanned ingredients.
　図６は、第１処理表示５１１が選択された場合に、タッチパネル１１７に表示されるスキャン画面５２０の一例を示す図である。
　図６に示すスキャン画面５２０には、切替表示５２１と、案内表示５２３と、範囲画像５２５とが表示される。
　切替表示５２１は、端末制御部１３０の動作モードを、一次元コードや二次元コード等のコード情報を認識するコード認識モードと、撮影画像から特徴点を抽出し、抽出した特徴点に基づいて食材を認識する画像認識モードとに切り替える表示である。切替表示５２１には、コード認識５２１Ａと、画像認識５２１Ｂとが表示される。端末制御部１３０は、ユーザーの操作によりコード認識５２１Ａが選択されると、動作モードをコード認識モードに変更する。また、端末制御部１３０は、ユーザーの操作により画像認識５２１Ｂが選択されると、動作モードを画像認識モードに変更する。図６には、斜め線のハッチングにより示すコード認識５２１Ａが選択された状態を示す。コード認識５２１Ａが選択された場合に表示されるスキャン画面５２０の表示を、コード認識表示という。 FIG. 6 is a diagram showing an example of a scan screen 520 displayed on the touch panel 117 when the first processing display 511 is selected.
On the scan screen 520 shown in FIG. 6, a switching display 521, a guidance display 523, and a range image 525 are displayed.
The switching display 521 sets the operation mode of the terminal control unit 130 to a code recognition mode for recognizing code information such as a one-dimensional code or a two-dimensional code, and extracts feature points from captured images, and uses the extracted feature points as ingredients. It is a display to switch to the image recognition mode for recognizing. Code recognition 521A and image recognition 521B are displayed on the switching display 521. When the code recognition 521A is selected by the user's operation, the terminal control unit 130 changes the operation mode to the code recognition mode. Further, when the image recognition 521B is selected by the user's operation, the terminal control unit 130 changes the operation mode to the image recognition mode. FIG. 6 shows a state in which the code recognition 521A indicated by the hatching of the diagonal line is selected. The display of the scan screen 520 displayed when the code recognition 521A is selected is referred to as a code recognition display.
　案内表示５２３には、切替表示５２１に含まれるコード認識５２１Ａと、画像認識５２１Ｂとのいずれが選択されているかを案内する表示が表示される。例えば、コード認識３２１Ａが選択された場合、例えば、「一次元コード、二次元コードを自動で認識します。」といった案内が表示される。また、画像認識５２１Ｂが選択された場合、例えば、「画像を解析して食材を認識します。」といった案内が表示される。 On the guidance display 523, a display indicating which of the code recognition 521A included in the switching display 521 and the image recognition 521B is selected is displayed. For example, when the code recognition 321A is selected, for example, a guidance such as "The one-dimensional code and the two-dimensional code are automatically recognized." Is displayed. Further, when the image recognition 521B is selected, for example, a guidance such as "Analyzing the image and recognizing the foodstuff" is displayed.
　範囲画像５２５には、コード認識５２１Ａや、画像認識５２１Ｂの認識範囲を示す画像が表示される。すなわち、カメラ１１３の撮影画像のうち、範囲画像５２５により示される範囲内の画像を対象としてコード認識や、画像認識が行われる。 The range image 525 displays an image showing the recognition range of the code recognition 521A and the image recognition 521B. That is, among the images captured by the camera 113, code recognition and image recognition are performed on the images within the range indicated by the range image 525.
　図７は、図６に示すスキャン画面５２０において、食品に付されたコード情報が認識された場合に、タッチパネル１１７に表示される選択画面５３０の一例を示す図である。
　端末制御部１３０は、第１処理表示５１１が選択された場合、コード情報の用途を、加熱情報の取得と判定する。端末制御部１３０は、取得したデータをサーバー装置３００に送信して、コード情報に対応した食品の加熱情報をサーバー装置３００から取得する。
　選択画面５３０には、食品パッケージの表示欄５３１と、調理器２００に設定可能な加熱情報の一覧が表示された一覧表示欄５３３と、送信ボタン５３５とが表示される。
　食品画像の表示欄３３１に表示される食品パッケージは、端末装置１００が、サーバー装置３００から取得した画像である。端末制御部１３０は、コード情報を読み取ると、読み取ったコード情報をサーバー装置３００に送信する。サーバー装置３００は、端末装置１００から受信したコード情報に対応する食品のパッケージ画像と、加熱情報とを該当の端末装置１００に送信する。 FIG. 7 is a diagram showing an example of a selection screen 530 displayed on the touch panel 117 when the code information attached to the food is recognized on the scan screen 520 shown in FIG.
When the first processing display 511 is selected, the terminal control unit 130 determines that the use of the code information is the acquisition of heating information. The terminal control unit 130 transmits the acquired data to the server device 300, and acquires the food heating information corresponding to the code information from the server device 300.
On the selection screen 530, a display field 531 of the food package, a list display field 533 displaying a list of heating information that can be set in the cooker 200, and a transmission button 535 are displayed.
The food package displayed in the food image display field 331 is an image acquired by the terminal device 100 from the server device 300. When the terminal control unit 130 reads the code information, the terminal control unit 130 transmits the read code information to the server device 300. The server device 300 transmits the food package image corresponding to the code information received from the terminal device 100 and the heating information to the corresponding terminal device 100.
　一覧表示欄５３３には、加熱情報として、定格高周波出力５３３１と、加熱時間５３３３とが表示される。加熱時間５３３３は、食品が、個数によって調理時間が変更される食品である場合、食品の個数ごとの加熱時間が表示される。 In the list display column 533, the rated high frequency output 5331 and the heating time 5333 are displayed as heating information. In the heating time 5333, when the food is a food whose cooking time is changed depending on the number of foods, the heating time for each number of foods is displayed.
　送信ボタン５３５は、ユーザーにより選択された定格高周波出力５３３１と、加熱時間５３３３との情報を、調理器２００に送信するボタンである。端末制御部１３０は、送信ボタン５３５が押下されると、定格高周波出力５３３１と、加熱時間５３３３との情報をサーバー装置３００に送信する。サーバー装置３００は、端末装置１００から受信した定格高周波出力５３３１と、加熱時間５３３３との情報を、該当の調理家電に送信する。 The transmission button 535 is a button for transmitting information of the rated high frequency output 5331 selected by the user and the heating time 5333 to the cooker 200. When the transmission button 535 is pressed, the terminal control unit 130 transmits information about the rated high frequency output 5331 and the heating time 5333 to the server device 300. The server device 300 transmits the information of the rated high frequency output 5331 received from the terminal device 100 and the heating time 5333 to the corresponding cooking appliance.
　図８は、図５に示す第２処理表示５１３が選択された場合に、タッチパネル１１７に表示されるスキャン画面５４０の一例を示す図であって、コード認識表示のスキャン画面５４０を示す。図８に示すスキャン画面５４０は、図６に示すスキャン画面５２０と同一であるため、スキャン画面５４０の詳細な説明は省略する。 FIG. 8 is a diagram showing an example of the scan screen 540 displayed on the touch panel 117 when the second process display 513 shown in FIG. 5 is selected, and shows the scan screen 540 of the code recognition display. Since the scan screen 540 shown in FIG. 8 is the same as the scan screen 520 shown in FIG. 6, a detailed description of the scan screen 540 will be omitted.
　図９は、図８に示すスキャン画面５４０において、食品に付されたコード情報が認識された場合に、タッチパネル１１７に表示される確認画面５５０の一例を示す図である。
　端末制御部１３０は、第２処理表示５１３が選択された場合、コード情報の用途を、スキャンした食品の在庫管理と判定する。端末制御部１３０は、食品の識別データと、ユーザーＩＤ１４５とを含む食品管理情報の取得要求をサーバー装置３００に送信し、食品の食品管理情報をサーバー装置３００から取得する。
　確認画面５５０には、食品パッケージの表示欄５５１と、食品情報の表示欄５５３と、登録ボタン５５５とが表示される。食品画像の表示欄３３１に表示される食品パッケージは、端末装置１００が、サーバー装置３００から取得した画像である。
　食品情報の表示欄５５３には、利用期限５５３Ａと、保存方法５５３Ｂとが表示される。
　登録ボタン５５５は、在庫管理に登録するボタンである。端末制御部１３０は、登録ボタン５５５が押下されると、表示欄５５１に表示された食品を在庫管理に登録する。すなわち、表示欄５５１に表示された食品の個数を管理する個数データを１加算する。 FIG. 9 is a diagram showing an example of a confirmation screen 550 displayed on the touch panel 117 when the code information attached to the food is recognized on the scan screen 540 shown in FIG.
When the second processing display 513 is selected, the terminal control unit 130 determines that the use of the code information is inventory management of the scanned food. The terminal control unit 130 transmits a food management information acquisition request including the food identification data and the user ID 145 to the server device 300, and acquires the food food management information of the food from the server device 300.
On the confirmation screen 550, a food package display field 551, a food information display field 553, and a registration button 555 are displayed. The food package displayed in the food image display field 331 is an image acquired by the terminal device 100 from the server device 300.
In the food information display field 553, the expiration date 553A and the storage method 553B are displayed.
The registration button 555 is a button for registering in inventory management. When the registration button 555 is pressed, the terminal control unit 130 registers the food displayed in the display field 551 in inventory management. That is, the number data for managing the number of foods displayed in the display field 551 is added by 1.
　図１０は、図９に示す確認画面５５０において、登録ボタン５５５が押下された後に表示されるコード認識表示のスキャン画面５４０の一例を示す図である。
　端末制御部１３０は、登録ボタン５５５が押下されると、食品を続けてスキャンできるようにするため、図１０に示すスキャン画面５４０を再度、表示させる。図１０に示すスキャン画面５４０は、案内表示５４３として、食品が在庫管理に登録されたこと、食品のスキャンを続けて行うことができることが表示される。
　また、図１０に示すスキャン画面５４０には、食品のスキャンを終了させる終了ボタン５４７が表示される。 FIG. 10 is a diagram showing an example of a scan screen 540 of the code recognition display displayed after the registration button 555 is pressed on the confirmation screen 550 shown in FIG. 9.
When the registration button 555 is pressed, the terminal control unit 130 displays the scan screen 540 shown in FIG. 10 again so that food can be continuously scanned. The scan screen 540 shown in FIG. 10 indicates that the food has been registered in inventory control and that the food can be continuously scanned as the guidance display 543.
Further, on the scan screen 540 shown in FIG. 10, an end button 547 for ending the food scan is displayed.
　図１１は、図５に示す第２処理表示５１３が選択された場合に、タッチパネル１１７に表示されるスキャン画面５４０の一例を示す図であって、画像認識５４１Ｂが選択された場合を示す図である。図１１には、斜め線のハッチングにより示す画像認識５２１Ｂが選択された状態を示す。画像認識５２１Ｂが選択された場合に表示されるスキャン画面５２０の表示を、画像認識表示という。
　画像認識５４１Ｂが選択された場合に、スキャン画面５４０に表示される案内表示５４３には、「画像から食材を検出します。」といった案内が表示される。
　また、画像認識５４１Ｂが選択された場合、スキャンを開始させるスキャンボタン５４９が表示される。ユーザーは、タッチパネル１１７に表示されるカメラ１１３の撮影画像のうち、認識させたい食材の画像が、範囲画像５４５により示される範囲内に納まるように端末装置１００の位置を移動させる。認識させたい食材の画像が範囲内に表示されると、ユーザーは、スキャンボタン５４９を押下する。 FIG. 11 is a diagram showing an example of a scan screen 540 displayed on the touch panel 117 when the second processing display 513 shown in FIG. 5 is selected, and is a diagram showing a case where the image recognition 541B is selected. be. FIG. 11 shows a state in which the image recognition 521B shown by hatching of diagonal lines is selected. The display of the scan screen 520 displayed when the image recognition 521B is selected is referred to as an image recognition display.
When the image recognition 541B is selected, the guidance display 543 displayed on the scan screen 540 displays a guidance such as "Foodstuff is detected from the image."
Further, when the image recognition 541B is selected, the scan button 549 for starting the scan is displayed. The user moves the position of the terminal device 100 so that the image of the food material to be recognized among the images taken by the camera 113 displayed on the touch panel 117 is within the range indicated by the range image 545. When the image of the foodstuff to be recognized is displayed within the range, the user presses the scan button 549.
　端末制御部１３０は、スキャンボタン５４９が押下されると、範囲画像５４５により示される範囲の撮影画像から特徴点を抽出し、抽出した特徴点を、予め記憶した食材の特徴点とを比較して食材の種類を判定する。 When the scan button 549 is pressed, the terminal control unit 130 extracts feature points from the captured image of the range indicated by the range image 545, and compares the extracted feature points with the feature points of the food material stored in advance. Determine the type of ingredients.
　図１２は、図１１に示すスキャン画面５４０において、画像認識が実行された場合に、タッチパネル１１７に表示される選択画面５６０の一例を示す図である。
　選択画面５６０には、食材選択欄５６１と、選択欄５６３とが表示される。食材選択欄５６１には、画像認識により検出された候補となる食材の食材名と、チェックボックスとが表示される。チェックボックスは、表示された食材名にそれぞれ対応づけて表示される。ユーザーは、食材選択欄５６１に、画像認識させた食材の食材名が表示された場合、対応するチェックボックスにチェックを入れる。 FIG. 12 is a diagram showing an example of a selection screen 560 displayed on the touch panel 117 when image recognition is executed on the scan screen 540 shown in FIG. 11.
On the selection screen 560, the food selection field 561 and the selection field 563 are displayed. In the food material selection field 561, the food material name of the candidate food material detected by the image recognition and the check box are displayed. Check boxes are displayed in association with the displayed food name. When the food material name of the food material recognized in the image is displayed in the food material selection field 561, the user checks the corresponding check box.
　選択欄５６３には、３つのボタン５６３が表示される。
　１つ目は、チェックボックスにより選択された食材を在庫管理に登録する場合に押下するボタン５６３Ａである。２つ目は、該当する食材がない場合に押下するボタン５６３Ｂである。３つ目は、画像認識を再度行う場合に押下するボタン５６３Ｃである。端末制御部１３０は、ボタン５６３Ｂが押下された場合、例えば、野菜、肉、魚、果物等の大分類と、トマト、レタス、小松菜等の小分類とをタッチパネル１１７に表示させる。ユーザーは、大分類及び小分類の該当する項目にチェックを入れる。 Three buttons 563 are displayed in the selection field 563.
The first is a button 563A to be pressed when registering the foodstuff selected by the check box in inventory management. The second is the button 563B that is pressed when there is no corresponding ingredient. The third is the button 563C that is pressed when the image recognition is performed again. When the button 563B is pressed, the terminal control unit 130 displays, for example, a major classification of vegetables, meat, fish, fruits, etc. and a minor classification of tomatoes, lettuce, Japanese mustard spinach, etc. on the touch panel 117. The user checks the corresponding items in the major classification and the minor classification.
　図１３は、図１２に示す選択画面５６０において、食材選択欄５６１に表示されたトマトのチェックボックスにチェックが入れられ、在庫管理に登録する処理が選択された場合に表示される確認画面５７０の一例を示す図である。
　端末制御部１３０は、第２処理表示５１３が選択された場合、コード情報の用途を、スキャンした食材の在庫管理と判定する。端末制御部１３０は、食材の種類を示す情報と、ユーザーＩＤ１４５とを含む食材管理情報の取得要求をサーバー装置３００に送信し、食材の食材管理情報をサーバー装置３００から取得する。
　確認画面５７０には、在庫管理に登録する食材名の表示欄５７１と、食材情報の表示欄５７１と、登録ボタン５７３とが表示される。食材情報の表示欄５７１には、利用期限５７１Ａと、保存方法５７１Ｂとが表示される。
　登録ボタン５７３は、在庫管理に登録するボタンである。端末制御部１３０は、登録ボタン５７３が押下されると、表示欄５５１に表示された食材を在庫管理に登録する。すなわち、表示欄５５１に表示された食材の個数を管理する個数データを１加算する。 FIG. 13 shows a confirmation screen 570 displayed when the check box of the tomato displayed in the food selection field 561 is checked in the selection screen 560 shown in FIG. 12 and the process of registering in inventory management is selected. It is a figure which shows an example.
When the second processing display 513 is selected, the terminal control unit 130 determines that the use of the code information is inventory management of the scanned foodstuff. The terminal control unit 130 transmits an acquisition request for food material management information including information indicating the type of food material and the user ID 145 to the server device 300, and acquires the food material management information for the food material from the server device 300.
On the confirmation screen 570, a display field 571 for the food material name to be registered in inventory management, a display field 571 for food material information, and a registration button 573 are displayed. In the food information display field 571, the expiration date 571A and the storage method 571B are displayed.
The registration button 573 is a button for registering in inventory management. When the registration button 573 is pressed, the terminal control unit 130 registers the foodstuff displayed in the display field 551 in inventory management. That is, the number data for managing the number of ingredients displayed in the display field 551 is added by 1.
　図１４は、第３処理表示５１５が選択された場合に、タッチパネル１１７に表示されるスキャン画面５８０の一例を示す図である。
　第３処理表示５１５が選択された場合、タッチパネル１１７には、画像認識表示のスキャン画面５８０が表示される。
　スキャン画面５８０には、切替表示５８１と、案内表示５８３と、範囲画像５８５と、スキャンボタン５８７とが表示される。第３処理表示５１５が選択された場合、スキャン画面５８０は、画像認識表示に変更される。スキャン画面５８０は、図１１に示すスキャン画面５４０と同一の画面である。 FIG. 14 is a diagram showing an example of a scan screen 580 displayed on the touch panel 117 when the third process display 515 is selected.
When the third processing display 515 is selected, the scan screen 580 of the image recognition display is displayed on the touch panel 117.
On the scan screen 580, a switching display 581, a guidance display 583, a range image 585, and a scan button 587 are displayed. When the third process display 515 is selected, the scan screen 580 is changed to the image recognition display. The scan screen 580 is the same screen as the scan screen 540 shown in FIG.
　図１５は、検索結果画面５９０の一例を示す図である。
　端末制御部１３０は、図１４に示すスキャン画面５８０において、画像認識により食材が検出された場合、図１５に示す検索結果画面５９０を表示させる。図１５に示す検索結果画面５９０は、画像認識により玉ねぎが検出された場合の検索結果画面５９０を示す。端末制御部１３０は、画像認識により玉ねぎが検出されると、検出された玉ねぎを使用したレシピの取得要求をサーバー装置３００に送信する。サーバー装置３００は、データベース３２３を検索して、玉ねぎを使用したレシピのレシピデータを読み出し、読み出したレシピデータを該当の端末装置１００に送信する。 FIG. 15 is a diagram showing an example of the search result screen 590.
When the foodstuff is detected by image recognition on the scan screen 580 shown in FIG. 14, the terminal control unit 130 displays the search result screen 590 shown in FIG. The search result screen 590 shown in FIG. 15 shows a search result screen 590 when an onion is detected by image recognition. When the onion is detected by the image recognition, the terminal control unit 130 transmits a recipe acquisition request using the detected onion to the server device 300. The server device 300 searches the database 323, reads out the recipe data of the recipe using the onion, and transmits the read recipe data to the corresponding terminal device 100.
　［１－２．動作］
　図１６～図１８は、端末装置１００の動作を示すフローチャートである。
　図１６～図１８のフローチャートを参照しながら端末装置１００の動作を説明する。
　ユーザーの操作によりサービスアプリ１４３が選択されると、端末制御部１３０は、サービスアプリ１４３を実行しＴＥ、タッチパネル１１７にホーム画面５０を表示させる。その後、端末制御部１３０は、スキャンアイコン５３が操作されたか否かを判定する（ステップＳ１）。端末制御部１３０は、スキャンアイコン５３が操作されていない場合（ステップＳ１／ＮＯ）、スキャンアイコン５３が操作されるまで処理の開始を待機する。 [1-2. motion]
16 to 18 are flowcharts showing the operation of the terminal device 100.
The operation of the terminal device 100 will be described with reference to the flowcharts of FIGS. 16 to 18.
When the service application 143 is selected by the user's operation, the terminal control unit 130 executes the service application 143 and causes the TE and the touch panel 117 to display the home screen 50. After that, the terminal control unit 130 determines whether or not the scan icon 53 has been operated (step S1). When the scan icon 53 is not operated (step S1 / NO), the terminal control unit 130 waits for the start of processing until the scan icon 53 is operated.
　端末制御部１３０は、スキャンアイコン５３が操作された場合（ステップＳ１／ＹＥＳ）、図５に示す処理選択画面５１０をタッチパネル１１７に表示させる（ステップＳ２）。次に、端末制御部１３０は、第１処理である加熱情報の送信が選択されたか否かを判定する（ステップＳ３）。端末制御部１３０は、第１処理表示５１１が選択されたか否かを判定して、第１処理が選択されたか否かを判定する。 When the scan icon 53 is operated (step S1 / YES), the terminal control unit 130 displays the process selection screen 510 shown in FIG. 5 on the touch panel 117 (step S2). Next, the terminal control unit 130 determines whether or not transmission of heating information, which is the first process, is selected (step S3). The terminal control unit 130 determines whether or not the first process display 511 is selected, and determines whether or not the first process is selected.
　端末制御部１３０は、第１処理が選択された場合（ステップＳ３／ＹＥＳ）、図６に示すスキャン画面５２０の表示態様をコード認識表示に変更し（ステップＳ４）、動作モードをコード認識モードに変更する。そして、端末制御部１３０は、カメラ１１３の撮影画像から一次元コード又は二次元コードを検出したか否かを判定する（ステップＳ５）。端末制御部１３０は、一次元コード又は二次元コードが検出できなかった場合（ステップＳ５／ＮＯ）、一次元コード又は二次元コードが検出されるまで、ステップＳ５の判定を繰り返す。ステップＳ５は、検出手順に相当する。 When the first process is selected (step S3 / YES), the terminal control unit 130 changes the display mode of the scan screen 520 shown in FIG. 6 to a code recognition display (step S4), and changes the operation mode to the code recognition mode. change. Then, the terminal control unit 130 determines whether or not the one-dimensional code or the two-dimensional code is detected from the captured image of the camera 113 (step S5). When the one-dimensional code or the two-dimensional code cannot be detected (step S5 / NO), the terminal control unit 130 repeats the determination in step S5 until the one-dimensional code or the two-dimensional code is detected. Step S5 corresponds to the detection procedure.
　また、端末制御部１３０は、一次元コード又は二次元コードが検出されると（ステップＳ５／ＹＥＳ）、検出した一次元コード又は二次元コードに含まれるデータを取得する。端末制御部１３０は、ステップＳ３の判定で、第１処理表示５１１が選択された場合、コード情報の用途を、加熱情報の取得と判定する。端末制御部１３０は、取得したデータをサーバー装置３００に送信し、識別データが示す食品の加熱情報をサーバー装置３００から取得する（ステップＳ６）。端末制御部１３０は、サーバー装置３００から加熱情報を取得すると、図７に示す選択画面５３０に取得した加熱情報を表示させ、調理器２００に設定可能な加熱情報を一覧表示させる（ステップＳ７）。ステップＳ５及びＳ６は、第１判定手順及び送信手順に相当する。また、ステップＳ７は、表示手順に相当する。 Further, when the one-dimensional code or the two-dimensional code is detected (step S5 / YES), the terminal control unit 130 acquires the detected one-dimensional code or the data included in the two-dimensional code. When the first process display 511 is selected in the determination in step S3, the terminal control unit 130 determines that the use of the code information is the acquisition of heating information. The terminal control unit 130 transmits the acquired data to the server device 300, and acquires the food heating information indicated by the identification data from the server device 300 (step S6). When the terminal control unit 130 acquires the heating information from the server device 300, the terminal control unit 130 displays the acquired heating information on the selection screen 530 shown in FIG. 7 and displays a list of the heating information that can be set in the cooker 200 (step S7). Steps S5 and S6 correspond to the first determination procedure and the transmission procedure. Further, step S7 corresponds to a display procedure.
　次に、端末制御部１３０は、加熱情報の１つが選択されたか否かを判定する（ステップＳ８）。端末制御部１３０は、加熱情報を選択する操作を受け付けていない場合（ステップＳ８／ＮＯ）、加熱情報が選択されるまで処理の開始を待機する。 Next, the terminal control unit 130 determines whether or not one of the heating information has been selected (step S8). When the terminal control unit 130 does not accept the operation of selecting the heating information (step S8 / NO), the terminal control unit 130 waits for the start of the process until the heating information is selected.
　また、端末制御部１３０は、加熱情報を選択する操作を受け付けた場合（ステップＳ８／ＹＥＳ）、図７に示す送信ボタン５３５が押下されたか否かを判定する（ステップＳ９）。端末制御部１３０は、送信ボタン５３５が押下されていない場合（ステップＳ９／ＮＯ）、送信ボタン５３５が押下されるまで待機する。端末装置１００は、送信ボタン５３５が押下されると（ステップＳ９／ＹＥＳ）、選択された加熱情報をサーバー装置３００に送信する（ステップＳ１０）。 Further, when the terminal control unit 130 receives the operation of selecting the heating information (step S8 / YES), the terminal control unit 130 determines whether or not the transmission button 535 shown in FIG. 7 is pressed (step S9). When the transmission button 535 is not pressed (step S9 / NO), the terminal control unit 130 waits until the transmission button 535 is pressed. When the transmission button 535 is pressed (step S9 / YES), the terminal device 100 transmits the selected heating information to the server device 300 (step S10).
　次に、図１６に示すステップＳ３の判定が否定判定であった場合の端末制御部１３０の動作を図１７に示すフローチャートを参照しながら説明する。
　端末制御部１３０は、まず、第２処理であるスキャンした食品や食材の在庫管理を行う処理が選択されたか否かを判定する（ステップＳ１１）。端末制御部１３０は、第２処理表示５１３が選択されたか否かを判定して、第２処理が選択されたか否かを判定する。 Next, the operation of the terminal control unit 130 when the determination in step S3 shown in FIG. 16 is a negative determination will be described with reference to the flowchart shown in FIG.
First, the terminal control unit 130 determines whether or not a second process, that is, a process for managing the inventory of scanned foods and foodstuffs, has been selected (step S11). The terminal control unit 130 determines whether or not the second process display 513 is selected, and determines whether or not the second process is selected.
　端末制御部１３０は、第２処理が選択された場合（ステップＳ１１／ＹＥＳ）、図８に示すスキャン画面５４０を表示させる。例えば、端末制御部１３０は、デフォルトで設定されたコード認識表示のスキャン画面５４０を表示させる（ステップＳ１２）。 When the second process is selected (step S11 / YES), the terminal control unit 130 displays the scan screen 540 shown in FIG. For example, the terminal control unit 130 displays the scan screen 540 of the code recognition display set by default (step S12).
　次に、端末制御部１３０は、切替表示５４１の操作により、スキャン画面５４０の表示を変更する操作を受け付けたか否かを判定する（ステップＳ１３）。端末制御部１３０は、スキャン画面５４０の表示を変更する操作を受け付けていない場合（ステップＳ１３／ＮＯ）、カメラ１１３の撮影画像から一次元コード又は二次元コードを検出したか否かを判定する（ステップＳ１４）。ステップＳ１４は、検出手順に相当する。 Next, the terminal control unit 130 determines whether or not the operation of changing the display of the scan screen 540 is accepted by the operation of the switching display 541 (step S13). When the terminal control unit 130 does not accept the operation of changing the display of the scan screen 540 (step S13 / NO), the terminal control unit 130 determines whether or not the one-dimensional code or the two-dimensional code is detected from the captured image of the camera 113 (step S13 / NO). Step S14). Step S14 corresponds to the detection procedure.
　端末制御部１３０は、一次元コード又は二次元コードが検出されなかった場合（ステップＳ１４／ＮＯ）、一次元コード又は二次元コードが検出されるまで、ステップＳ１４の判定を繰り返す。 When the one-dimensional code or the two-dimensional code is not detected (step S14 / NO), the terminal control unit 130 repeats the determination in step S14 until the one-dimensional code or the two-dimensional code is detected.
　また、端末制御部１３０は、一次元コード又は二次元コードが検出されると（ステップＳ１４／ＹＥＳ）、検出した一次元コード又は二次元コードに含まれる識別データを取得する。端末制御部１３０は、ステップＳ１１の判定で、第２処理表示５１３が選択された場合、コード情報の用途を、スキャンした食品や食材の在庫管理と判定する。端末制御部１３０は、食品の識別データと、ユーザーＩＤ１４５とを含む食品管理情報の取得要求をサーバー装置３００に送信し、識別データに対応した食品の食品管理情報をサーバー装置３００から取得する（ステップＳ１５）。端末制御部１３０は、サーバー装置３００から食品管理情報を取得すると、取得した食品管理情報を、図９に示す確認画面５５０に表示させる（ステップＳ１６）。この確認画面５５０には、サーバー装置３００から取得した食品管理情報に含まれる利用期限や、保存方法等の情報が表示される。ステップＳ１４及びＳ１５は、第１判定手順及び送信手順に相当する。また、ステップＳ１６は、表示手順に相当する。 Further, when the one-dimensional code or the two-dimensional code is detected (step S14 / YES), the terminal control unit 130 acquires the identification data included in the detected one-dimensional code or the two-dimensional code. When the second process display 513 is selected in the determination in step S11, the terminal control unit 130 determines that the use of the code information is inventory management of the scanned food or foodstuff. The terminal control unit 130 transmits a food management information acquisition request including the food identification data and the user ID 145 to the server device 300, and acquires the food food management information of the food corresponding to the identification data from the server device 300 (step). S15). When the terminal control unit 130 acquires the food management information from the server device 300, the terminal control unit 130 displays the acquired food management information on the confirmation screen 550 shown in FIG. 9 (step S16). On the confirmation screen 550, information such as the expiration date and the storage method included in the food management information acquired from the server device 300 is displayed. Steps S14 and S15 correspond to the first determination procedure and the transmission procedure. Further, step S16 corresponds to a display procedure.
　次に、端末制御部１３０は、確認画面５５０に表示された登録ボタン５５５が押下されたか否かを判定する（ステップＳ１７）。端末制御部１３０は、登録ボタン５５５が押下されると（ステップＳ１７／ＹＥＳ）、端末記憶部１４０が記憶する在庫管理データとして食品管理情報を記憶させる（ステップＳ１８）。その後、端末制御部１３０は、タッチパネル１１７にホーム画面５０を表示させ、ステップＳ１の判定に戻る。 Next, the terminal control unit 130 determines whether or not the registration button 555 displayed on the confirmation screen 550 has been pressed (step S17). When the registration button 555 is pressed (step S17 / YES), the terminal control unit 130 stores food management information as inventory management data stored by the terminal storage unit 140 (step S18). After that, the terminal control unit 130 causes the touch panel 117 to display the home screen 50, and returns to the determination in step S1.
　また、端末制御部１３０は、ステップＳ１３において、切替表示５４１の操作により、スキャン画面５４０の表示を変更する操作を受け付けた場合（ステップＳ１３／ＹＥＳ）、スキャン画面５２０の表示態様を、図１１に示す画像認識表示に変更し、動作モードを画像認識モードに変更する。 Further, when the terminal control unit 130 accepts the operation of changing the display of the scan screen 540 by the operation of the switching display 541 in step S13 (step S13 / YES), the display mode of the scan screen 520 is shown in FIG. Change to the image recognition display shown and change the operation mode to the image recognition mode.
　次に、端末制御部１３０は、スキャンボタン５４９が押下されたか否かを判定する（ステップＳ１９）。端末制御部１３０は、スキャンボタン５４９が押下されていない場合（ステップＳ２０／ＮＯ）、スキャンボタン５４９が押下されるまで処理の開始を待機する。また、端末制御部１３０は、スキャンボタン５４９が押下されると（ステップＳ１９／ＹＥＳ）、カメラ１１３の撮影画像から特徴点を抽出し、抽出した特徴点を、予め記憶した食材の特徴点と比較して食材を判定する（ステップＳ２０）。ステップＳ１９及びステップＳ２０は、第２判定手順に相当する。 Next, the terminal control unit 130 determines whether or not the scan button 549 is pressed (step S19). When the scan button 549 is not pressed (step S20 / NO), the terminal control unit 130 waits for the start of processing until the scan button 549 is pressed. Further, when the scan button 549 is pressed (step S19 / YES), the terminal control unit 130 extracts feature points from the captured image of the camera 113, and compares the extracted feature points with the feature points of the food material stored in advance. To determine the ingredients (step S20). Step S19 and step S20 correspond to the second determination procedure.
　次に、端末制御部１３０は、画像認識により判定した食材の候補をタッチパネル１１７に表示させる（ステップＳ２１）。端末制御部１３０は、図１２に示す選択画面５６０を表示させる。この選択画面５６０には、食材選択欄５６１と、選択欄５６３とが表示される。端末制御部１３０は、選択画面５６０を表示させると、食材が選択されたか否かを判定する（ステップＳ２２）。端末制御部１３０は、食材が選択された場合（ステップＳ２２／ＹＥＳ）、該当する食材の食材管理情報の取得要求をサーバー装置３００に送信し、サーバー装置３００から食材管理情報を取得する（ステップＳ２５）。 Next, the terminal control unit 130 displays the candidate ingredients determined by image recognition on the touch panel 117 (step S21). The terminal control unit 130 displays the selection screen 560 shown in FIG. On the selection screen 560, a food material selection field 561 and a selection field 563 are displayed. When the selection screen 560 is displayed, the terminal control unit 130 determines whether or not the food has been selected (step S22). When the foodstuff is selected (step S22 / YES), the terminal control unit 130 transmits a request for acquiring the foodstuff management information of the corresponding foodstuff to the server device 300, and acquires the foodstuff management information from the server device 300 (step S25). ).
　次に、端末制御部１３０は、取得した食材管理情報を、図１３に示す確認画面５７０に表示させる（ステップＳ２６）。この確認画面５７０にも、サーバー装置３００から取得した食品管理情報に含まれる利用期限や、保存方法等の情報が表示される。 Next, the terminal control unit 130 displays the acquired foodstuff management information on the confirmation screen 570 shown in FIG. 13 (step S26). The confirmation screen 570 also displays information such as the expiration date and the storage method included in the food management information acquired from the server device 300.
　次に、端末制御部１３０は、確認画面５７０に表示された登録ボタン５７３が押下されたか否かを判定する（ステップＳ２７）。端末制御部１３０は、登録ボタン５７３が押下されると（ステップＳ２７／ＹＥＳ）、端末記憶部１４０が記憶する在庫管理データとして食材管理情報を記憶させる（ステップＳ２８）。その後、端末制御部１３０は、タッチパネル１１７にホーム画面５０を表示させ、ステップＳ１の判定に戻る。 Next, the terminal control unit 130 determines whether or not the registration button 573 displayed on the confirmation screen 570 is pressed (step S27). When the registration button 573 is pressed (step S27 / YES), the terminal control unit 130 stores food material management information as inventory management data stored by the terminal storage unit 140 (step S28). After that, the terminal control unit 130 causes the touch panel 117 to display the home screen 50, and returns to the determination in step S1.
　また、端末制御部１３０は、ステップＳ２２の判定が否定判定の場合（ステップＳ２２／ＮＯ）、食材が選択されずに、再スキャンを選択する操作を受け付けたか否かを判定する（ステップＳ２３）。端末制御部１３０は、再スキャンが選択された場合（ステップＳ２３／ＹＥＳ）、ステップＳ１２の処理に戻り、スキャン画面５４０を表示させる。 Further, when the determination in step S22 is a negative determination (step S22 / NO), the terminal control unit 130 determines whether or not the operation of selecting the rescan is accepted without selecting the food (step S23). When rescan is selected (step S23 / YES), the terminal control unit 130 returns to the process of step S12 and displays the scan screen 540.
　また、端末制御部１３０は、再スキャンを選択する操作を受け付けなかった場合（ステップＳ２３／ＮＯ）、ユーザーの入力を受け付ける（ステップＳ２４）。例えば、端末制御部１３０は、野菜、肉、魚、果物等の大分類と、トマト、レタス、小松菜等の小分類とをタッチパネル１１７に表示させ、ユーザーに、大分類及び小分類の該当する項目に入力される。その後、端末制御部１３０は、ステップＳ２３の処理に移行し、サーバー装置３００から食材管理情報を取得する。 Further, when the terminal control unit 130 does not accept the operation of selecting the rescan (step S23 / NO), the terminal control unit 130 accepts the user's input (step S24). For example, the terminal control unit 130 displays a major classification of vegetables, meat, fish, fruits, etc. and a minor classification of tomatoes, lettuce, Japanese mustard spinach, etc. on the touch panel 117, and allows the user to display the corresponding items of the major classification and the minor classification. Is entered in. After that, the terminal control unit 130 shifts to the process of step S23 and acquires the food material management information from the server device 300.
　次に、図１７に示すステップＳ２９の判定が否定判定であった場合の端末制御部１３０の動作を図１８に示すフローチャートを参照しながら説明する。
　端末制御部１３０は、まず、第３処理であるスキャンした食材を含むレシピを検索する処理が選択されたか否かを判定する（ステップＳ２９）。端末制御部１３０は、第３処理表示５１５が選択されたか否かを判定して、第３処理が選択されたか否かを判定する。 Next, the operation of the terminal control unit 130 when the determination in step S29 shown in FIG. 17 is a negative determination will be described with reference to the flowchart shown in FIG.
First, the terminal control unit 130 determines whether or not the process of searching for the recipe including the scanned foodstuff, which is the third process, is selected (step S29). The terminal control unit 130 determines whether or not the third process display 515 is selected, and determines whether or not the third process is selected.
　端末制御部１３０は、第３処理が選択されなかった場合（ステップＳ２９／ＮＯ）、図１６に示すステップＳ３の判定に戻る。また、端末制御部１３０は、第３処理が選択された場合（ステップＳ２９／ＹＥＳ）、動作モードを画像確認モードに移行させ、図１４に示す画像認識表示のスキャン画面５４０を表示させる。 When the third process is not selected (step S29 / NO), the terminal control unit 130 returns to the determination in step S3 shown in FIG. Further, when the third process is selected (step S29 / YES), the terminal control unit 130 shifts the operation mode to the image confirmation mode and displays the scan screen 540 of the image recognition display shown in FIG.
　図１８に示すステップＳ３０～Ｓ３６までの動作は、図１７に示すステップＳ１９～Ｓ２４の動作と同一であるため、ステップＳ３０～Ｓ３６の動作の説明は省略する。
　次に、端末制御部１３０は、食材を特定すると、特定した食材を含むレシピの取得要求をサーバー装置３００に送信して、サーバー装置３００からレシピデータを取得する（ステップＳ３７）。 Since the operations of steps S30 to S36 shown in FIG. 18 are the same as the operations of steps S19 to S24 shown in FIG. 17, the description of the operations of steps S30 to S36 will be omitted.
Next, when the terminal control unit 130 specifies the foodstuff, the terminal control unit 130 sends a recipe acquisition request including the specified foodstuff to the server device 300, and acquires the recipe data from the server device 300 (step S37).
　次に、端末制御部１３０は、取得したレシピデータに基づき、図１５に示す検索結果画面５９０を表示させる。検索結果画面５９０には、取得したレシピデータが一覧表示される（ステップＳ３８）。次に、端末制御部１３０は、レシピを選択する操作を受け付けたか否かを判定する（ステップＳ３９）。端末制御部１３０は、レシピを選択する操作を受け付けた場合（ステップＳ３９／ＹＥＳ）、選択されたレシピをタッチパネル１１７に表示させる（ステップＳ４０）。また、端末制御部１３０は、レシピを選択する操作を受け付けなかった場合（ステップＳ３９／ＮＯ）、終了操作を受け付けたか否かを判定する（ステップＳ４１）。例えば、端末制御部１３０は、表示画面に表示された閉じるボタンが押下されたか否かを判定して、終了操作を受け付けたか否かを判定する。 Next, the terminal control unit 130 displays the search result screen 590 shown in FIG. 15 based on the acquired recipe data. The acquired recipe data is displayed in a list on the search result screen 590 (step S38). Next, the terminal control unit 130 determines whether or not the operation of selecting the recipe has been accepted (step S39). When the terminal control unit 130 accepts the operation of selecting a recipe (step S39 / YES), the terminal control unit 130 displays the selected recipe on the touch panel 117 (step S40). Further, when the terminal control unit 130 does not accept the operation of selecting the recipe (step S39 / NO), the terminal control unit 130 determines whether or not the end operation is accepted (step S41). For example, the terminal control unit 130 determines whether or not the close button displayed on the display screen has been pressed, and determines whether or not the end operation has been accepted.
　端末制御部１３０は、終了操作を受け付けた場合（ステップＳ４１／ＹＥＳ）、タッチパネル１１７にホーム画面５０を表示させ（ステップＳ４２）、ステップＳ１の判定に戻る。また、端末制御部１３０は、終了操作を受け付けていない場合（ステップＳ４１／ＮＯ）、ステップＳ３９の判定に戻る。 When the terminal control unit 130 accepts the end operation (step S41 / YES), the terminal control unit 130 displays the home screen 50 on the touch panel 117 (step S42), and returns to the determination in step S1. Further, when the terminal control unit 130 does not accept the end operation (step S41 / NO), the terminal control unit 130 returns to the determination in step S39.
　図１９は、サーバー装置３００の動作を示すフローチャートである。
　図１９に示すフローチャートを参照しながらサーバー装置３００の動作を説明する。
　まず、サーバー制御部３１０は、端末装置１００からデータを受信したか否かを判定する（ステップＴ１）。サーバー制御部３１０は、端末装置１００からデータを受信していない場合（ステップＴ１／ＮＯ）、データを受信するまで待機する。 FIG. 19 is a flowchart showing the operation of the server device 300.
The operation of the server device 300 will be described with reference to the flowchart shown in FIG.
First, the server control unit 310 determines whether or not data has been received from the terminal device 100 (step T1). When the server control unit 310 has not received data from the terminal device 100 (step T1 / NO), the server control unit 310 waits until the data is received.
　サーバー制御部３１０は、端末装置１００からデータを受信した場合（ステップＴ１／ＹＥＳ）、受信したデータに加熱情報の取得要求が含まれるか否かを判定する（ステップＴ２）。サーバー制御部３１０は、受信したデータに加熱情報の取得要求が含まれる場合（ステップＴ２／ＹＥＳ）、該当する食品の加熱情報をデータベース３２３から読み出す（ステップＴ３）。サーバー制御部３１０は、読み出した加熱情報を、データを受信した端末装置１００に送信する（ステップＴ１０）。 When the server control unit 310 receives data from the terminal device 100 (step T1 / YES), the server control unit 310 determines whether or not the received data includes a request for acquisition of heating information (step T2). When the received data includes a request for acquiring heating information (step T2 / YES), the server control unit 310 reads the heating information of the corresponding food from the database 323 (step T3). The server control unit 310 transmits the read heating information to the terminal device 100 that has received the data (step T10).
　また、サーバー制御部３１０は、受信したデータに加熱情報の取得要求が含まれない場合（ステップＴ２／ＮＯ）、受信したデータに食品管理情報の取得要求が含まれるか否かを判定する（ステップＴ４）。サーバー制御部３１０は、受信したデータに食品管理情報の取得要求が含まれる場合（ステップＴ４／ＹＥＳ）、該当する食品の食品管理情報をデータベース３２３から読み出す（ステップＴ５）。サーバー制御部３１０は、読み出した食品管理情報を、データを受信した端末装置１００に送信する（ステップＴ１０）。 Further, when the received data does not include the heating information acquisition request (step T2 / NO), the server control unit 310 determines whether or not the received data includes the food management information acquisition request (step). T4). When the received data includes a request for acquiring food management information (step T4 / YES), the server control unit 310 reads the food management information of the corresponding food from the database 323 (step T5). The server control unit 310 transmits the read food management information to the terminal device 100 that has received the data (step T10).
　また、サーバー制御部３１０は、受信したデータに食品管理情報の取得要求が含まれない場合（ステップＴ４／ＮＯ）、受信したデータに食材管理情報の取得要求が含まれるか否かを判定する（ステップＴ６）。サーバー制御部３１０は、受信したデータに食材管理情報の取得要求が含まれる場合（ステップＴ６／ＹＥＳ）、該当する食材の食材管理情報をデータベース３２３から読み出す（ステップＴ７）。サーバー制御部３１０は、読み出した食材管理情報を、データを受信した端末装置１００に送信する（ステップＴ１０）。 Further, when the received data does not include the food management information acquisition request (step T4 / NO), the server control unit 310 determines whether or not the received data includes the food management information acquisition request (step T4 / NO). Step T6). When the received data includes a request for acquisition of foodstuff management information (step T6 / YES), the server control unit 310 reads the foodstuff management information of the corresponding foodstuff from the database 323 (step T7). The server control unit 310 transmits the read food material management information to the terminal device 100 that has received the data (step T10).
　また、サーバー制御部３１０は、受信したデータに食材管理情報の取得要求が含まれない場合（ステップＴ６／ＮＯ）、受信したデータにレシピデータの取得要求が含まれるか否かを判定する（ステップＴ８）。サーバー制御部３１０は、受信したデータにレシピデータの取得要求が含まれる場合（ステップＴ８／ＹＥＳ）、データに含まれる指定食材を含むレシピデータをデータベース３２３から読み出す（ステップＴ９）。サーバー制御部３１０は、読み出したレシピデータを、データを受信した端末装置１００に送信する（ステップＴ１０）。 Further, when the received data does not include the request for acquiring the food material management information (step T6 / NO), the server control unit 310 determines whether or not the received data includes the request for acquiring the recipe data (step). T8). When the received data includes a recipe data acquisition request (step T8 / YES), the server control unit 310 reads the recipe data including the designated ingredients included in the data from the database 323 (step T9). The server control unit 310 transmits the read recipe data to the terminal device 100 that has received the data (step T10).
　またサーバー制御部３１０は、受信したデータにレシピデータの取得要求が含まれない場合（ステップＴ８／ＮＯ）、受信したデータに対応した処理を実行して（ステップＴ１１）、処理フローを終了させる。 Further, when the received data does not include the recipe data acquisition request (step T8 / NO), the server control unit 310 executes the process corresponding to the received data (step T11), and ends the process flow.
　［１－３．効果等］ [1-3. Effect, etc.]
　以上、説明したように、サービスアプリ１４３は、端末制御部１３０に、検出手順、第１判定手順、送信手順及び表示手順と、を実行させる。
　検出手順は、カメラ１１３により撮影された撮影画像に含まれるコード情報を検出する手順である。
　第１判定手順は、タッチパネル１１７に対するタッチ操作により受け付けた操作に基づいてコード情報の用途を判定する手順である。
　送信手順は、判定したコード情報の用途に対応した情報の取得要求を、サーバー装置３００に送信する手順である。
　表示手順は、サーバー装置３００から受信した情報を表示させる手順である。 As described above, the service application 143 causes the terminal control unit 130 to execute the detection procedure, the first determination procedure, the transmission procedure, and the display procedure.
The detection procedure is a procedure for detecting the code information included in the captured image captured by the camera 113.
The first determination procedure is a procedure for determining the use of the code information based on the operation received by the touch operation on the touch panel 117.
The transmission procedure is a procedure for transmitting an information acquisition request corresponding to the determined use of the code information to the server device 300.
The display procedure is a procedure for displaying the information received from the server device 300.
　これにより、受け付けた操作に基づいてコード情報の用途が判定され、判定された用途に対応した情報をサーバー装置３００から取得して表示させることができる。従って、同一のコード情報を読み取った場合であっても、操作に対応した情報を表示部に表示させることができ、ユーザーの利便性を向上させることができる。 Thereby, the use of the code information is determined based on the received operation, and the information corresponding to the determined use can be acquired from the server device 300 and displayed. Therefore, even when the same code information is read, the information corresponding to the operation can be displayed on the display unit, and the convenience of the user can be improved.
　また、送信手順は、第１判定手順により判定されたコード情報の用途が、食品を加熱する加熱調理である場合、加熱調理に関する加熱情報の取得要求をサーバー装置３００に送信する。 Further, in the transmission procedure, when the use of the code information determined by the first determination procedure is cooking for heating food, a request for acquisition of heating information regarding cooking is transmitted to the server device 300.
　これにより、コード情報の用途が、食品を加熱する加熱調理である場合、加熱調理に関する加熱情報を取得して端末装置１００に表示させることができる。 Thereby, when the use of the code information is cooking for heating food, it is possible to acquire the heating information regarding cooking and display it on the terminal device 100.
　また、送信手順は、第１判定手順により判定したコード情報の用途が食品の在庫管理である場合、コード情報が示す食品の利用期限を示す情報の取得要求をサーバー装置３００に送信する。 Further, in the transmission procedure, when the use of the code information determined by the first determination procedure is food inventory management, a request for acquiring information indicating the expiration date of the food indicated by the code information is transmitted to the server device 300.
　これにより、コード情報の用途が、食品の在庫管理である場合、食品の利用期限を示す情報を取得して端末装置１００に表示することができる。 As a result, when the use of the code information is food inventory management, it is possible to acquire information indicating the expiration date of the food and display it on the terminal device 100.
　また、サービスアプリ１４３は、端末制御部１３０に、撮影画像に撮影された食材を判定する第２判定手順を実行させる。
　送信手順は、第２判定手順により判定された食材の利用期限を示す情報の取得要求をサーバー装置３００に送信する。 Further, the service application 143 causes the terminal control unit 130 to execute the second determination procedure for determining the food material captured in the captured image.
The transmission procedure transmits a request for acquisition of information indicating the expiration date of the food material determined by the second determination procedure to the server device 300.
　これにより、撮影画像に撮影された食材を判定し、判定した食材の利用期限を示す情報を端末装置１００に表示させることができる。 As a result, it is possible to determine the ingredients taken in the captured image and display the information indicating the expiration date of the determined ingredients on the terminal device 100.
　（実施の形態２）
　実施の形態２を説明する。
　上述した実施の形態１では、スキャンアイコン５３が選択されると、タッチパネル１１７に処理選択画面５１０を表示させ、第１処理、第２処理及び第３処理のいずれかをユーザーに選択させた。実施の形態２では、図４に示すホーム画面５０に表示されたスキャンアイコン５３が選択されると、第１処理、第２処理及び第３処理を選択する処理選択画面５１０を表示させることなく、コード認識表示のスキャン画面５２０、５４０、又は画像認識表示のスキャン画面５４０、５８０を表示させ、コード認識又は画像認識を実行する。 (Embodiment 2)
The second embodiment will be described.
In the first embodiment described above, when the scan icon 53 is selected, the touch panel 117 displays the process selection screen 510, and the user is made to select one of the first process, the second process, and the third process. In the second embodiment, when the scan icon 53 displayed on the home screen 50 shown in FIG. 4 is selected, the process selection screen 510 for selecting the first process, the second process, and the third process is not displayed. The scan screens 520 and 540 of the code recognition display or the scan screens 540 and 580 of the image recognition display are displayed, and code recognition or image recognition is executed.
　端末制御部１３０は、コード認識表示のスキャン画面５２０、５４０によりコード情報を取得した場合、第１処理表示５１１及び第２処理表示５１３が表示された処理選択画面を表示させ、ユーザーにいずれの処理を実行するかを選択させる。
　また、端末制御部１３０は、画像認識表示のスキャン画面５４０、５８０により食材を判定した場合、第２処理表示５１３及び第３処理表示５１５が表示された処理選択画面を表示させ、ユーザーにいずれの処理を実行するかを選択させる。 When the terminal control unit 130 acquires the code information on the scan screens 520 and 540 of the code recognition display, the terminal control unit 130 displays the process selection screen on which the first process display 511 and the second process display 513 are displayed, and causes the user to perform any process. Lets you choose whether to execute.
Further, when the terminal control unit 130 determines the foodstuff on the scan screen 540 and 580 of the image recognition display, the terminal control unit 130 causes the user to display the process selection screen on which the second process display 513 and the third process display 515 are displayed. Lets you choose whether to execute the process.
　このような手順であっても、上述した実施の形態１と同一の効果を得ることができる。 Even with such a procedure, the same effect as that of the first embodiment described above can be obtained.
　上述した実施の形態は、あくまでも本発明の一態様を示すものであり、本発明の範囲内で任意に変形および応用が可能である。 The above-described embodiment shows only one aspect of the present invention, and can be arbitrarily modified and applied within the scope of the present invention.
　例えば、端末制御部１３０の機能は、複数のプロセッサー、又は、半導体チップにより実現してもよい。 For example, the function of the terminal control unit 130 may be realized by a plurality of processors or semiconductor chips.
　また、画像認識部１５３は、ＡＩによる画像認識により、食材を特定してもよい。例えば食材を撮像した教師用の撮影画像から食材の色、形状等の特徴量を学習する機械学習処理を行い、学習済みモデルを構築しておく。この学習済みモデルは、端末記憶部１４０に記憶させておく。画像認識部１５３は、カメラ１１３の撮影画像が入力されると、学習済みモデルを参照して、撮像画像に含まれる食材を判定する。また、前述の学習済みモデルをサーバー記憶部３２０に記憶させておき、サーバー装置３００において、食材を判定する構成であってもよい。この場合、画像認識部１５３は、カメラ１１３の撮影画像が入力されると、入力された撮影画像をサーバー装置３００に送信し、サーバー装置３００から食材の判定結果を取得する。 Further, the image recognition unit 153 may specify the food material by image recognition by AI. For example, a machine learning process is performed to learn features such as the color and shape of foodstuffs from a photographed image of a foodstuff for a teacher, and a trained model is constructed. This trained model is stored in the terminal storage unit 140. When the captured image of the camera 113 is input, the image recognition unit 153 refers to the trained model and determines the food material contained in the captured image. Further, the above-mentioned trained model may be stored in the server storage unit 320, and the server device 300 may be configured to determine the foodstuff. In this case, when the captured image of the camera 113 is input, the image recognition unit 153 transmits the input captured image to the server device 300, and acquires the determination result of the foodstuff from the server device 300.
　図２に示した各部は一例であって、具体的な実装形態は特に限定されない。つまり、必ずしも各部に個別に対応するハードウェアが実装される必要はなく、一つのプロセッサーがプログラムを実行することで各部の機能を実現する構成とすることも勿論可能である。また、上述した実施の形態においてソフトウェアで実現される機能の一部をハードウェアとしてもよく、或いは、ハードウェアで実現される機能の一部をソフトウェアで実現してもよい。 Each part shown in FIG. 2 is an example, and the specific mounting form is not particularly limited. That is, it is not always necessary to implement the hardware corresponding to each part individually, and it is of course possible to realize the function of each part by executing the program by one processor. Further, a part of the functions realized by the software in the above-described embodiment may be realized by the hardware, or a part of the functions realized by the hardware may be realized by the software.
　また、例えば、図１６－図１８のフローチャートに示す動作の単位は、端末装置１００の動作の理解を容易にするために、主な処理内容に応じて分割したものであり、処理単位の分割の仕方や名称によって、本発明が限定されることはない。処理内容に応じて、さらに多くのステップ単位に分割してもよい。また、１つのステップ単位がさらに多くの処理を含むように分割してもよい。また、そのステップの順番は、本発明の趣旨に支障のない範囲で適宜に入れ替えてもよい。また、図１９に示すサーバー装置３００の動作を示すフローチャートについても同様である。 Further, for example, the operation unit shown in the flowchart of FIGS. 16 to 18 is divided according to the main processing contents in order to facilitate understanding of the operation of the terminal device 100, and the processing unit is divided. The present invention is not limited by the method or the name. It may be divided into more step units depending on the processing content. Further, one step unit may be divided so as to include more processes. Further, the order of the steps may be appropriately changed as long as it does not interfere with the gist of the present invention. The same applies to the flowchart showing the operation of the server device 300 shown in FIG.
　なお、上述の実施の形態は、本開示における技術を例示するためのものであるから、特許請求の範囲またはその均等の範囲において種々の変更、置き換え、付加、省略などを行うことができる。 Since the above-described embodiment is for exemplifying the technique in the present disclosure, various changes, replacements, additions, omissions, etc. can be made within the scope of claims or the equivalent thereof.
　以上のように、本発明のプログラム及び端末装置は、料理に係る情報をユーザーに提供する用途に利用可能である。 As described above, the program and the terminal device of the present invention can be used for providing information related to cooking to the user.
　１　サービス提供システム
　１０　ネットワーク
　１００　端末装置
　１１１　端末通信Ｉ／Ｆ
　１１３　カメラ
　１１５　表示部
　１１７　タッチパネル
　１３０　端末制御部
　１４０　端末記憶部
　１４１　制御プログラム
　１４３　サービスアプリ
　１５０　端末プロセッサー
　１５１　コード検出部
　１５３　画像認識部
　１５５　データ取得部
　２００　調理器
　２０１　調理器通信Ｉ／Ｆ
　２０３　操作部
　２１０　調理制御部
　２２０　加熱調理部
　３００　サーバー装置
　３０１　サーバー通信Ｉ／Ｆ
　３１０　サーバー制御部
　３２０　サーバー記憶部
　３２３　データベース
　３２３ａ　ユーザーデータベース 1 Service provision system 10 Network 100 Terminal equipment 111 Terminal communication I / F
113 Camera 115 Display unit 117 Touch panel 130 Terminal control unit 140 Terminal storage unit 141 Control program 143 Service application 150 Terminal processor 151 Code detection unit 153 Image recognition unit 155 Data acquisition unit 200 Cooker 201 Cooker communication I / F
203 Operation unit 210 Cooking control unit 220 Cooking unit 300 Server device 301 Server communication I / F
310 Server control unit 320 Server storage unit 323 Database 323a User database

Claims (6)

  1. 　コンピューターに、
    　撮影部により撮影された撮影画像に含まれるコード情報を検出する検出手順と、
    　操作部が受け付けた操作に基づいて前記コード情報の用途を判定する第１判定手順と、
    　判定した前記コード情報の用途に対応した情報の取得要求を、サーバー装置に送信する送信手順と、
    　前記サーバー装置から受信した情報を表示させる表示手順と、
    　を実行させるプログラム。     On the computer
    A detection procedure that detects the code information contained in the captured image captured by the imaging unit, and
    The first determination procedure for determining the use of the code information based on the operation accepted by the operation unit, and
    A transmission procedure for transmitting an information acquisition request corresponding to the determined use of the code information to the server device, and
    The display procedure for displaying the information received from the server device and
    A program to execute.
  2. 　前記送信手順は、前記第１判定手順により判定された前記コード情報の用途が、食品を加熱する加熱調理である場合、加熱調理に関する加熱情報の取得要求を前記サーバー装置に送信する、請求項１記載のプログラム。 The transmission procedure is claimed 1 in which, when the use of the code information determined by the first determination procedure is cooking for heating food, a request for acquisition of heating information regarding cooking is transmitted to the server device. The program described.
  3. 　前記送信手順は、前記第１判定手順により判定した前記コード情報の用途が食品の在庫管理である場合、前記コード情報が示す食品の利用期限を示す情報の取得要求を前記サーバー装置に送信する、請求項１又は２記載のプログラム。 In the transmission procedure, when the use of the code information determined by the first determination procedure is food inventory management, a request for acquisition of information indicating the expiration date of the food indicated by the code information is transmitted to the server device. The program according to claim 1 or 2.
  4. 　前記撮影画像に撮影された食材を判定する第２判定手順を実行し、
    　前記送信手順は、前記第２判定手順により判定された食材の利用期限を示す情報の取得要求を前記サーバー装置に送信する、請求項１から３のいずれか一項に記載のプログラム。     The second determination procedure for determining the ingredients captured in the captured image is executed.
    The program according to any one of claims 1 to 3, wherein the transmission procedure transmits a request for acquisition of information indicating an expiration date of the food material determined by the second determination procedure to the server device.
  5. 　表示部と、
    　通信部と、
    　撮影部と、
    　前記撮影部が撮影した画像に含まれるコード情報を検出するコード検出部と、
    　操作を受け付ける操作部と、
    　前記操作部が受け付けた操作に基づいて前記コード情報の用途を判定し、判定した用途に対応した情報の取得要求を、前記通信部を介してサーバー装置に送信する制御部と、を備え、
    　前記制御部は、前記サーバー装置から受信した情報を前記表示部に表示させる、端末装置。     Display and
    Communication department and
    With the shooting department
    A code detection unit that detects code information contained in an image captured by the imaging unit, and a code detection unit.
    The operation unit that accepts operations and
    A control unit that determines the use of the code information based on the operation received by the operation unit and transmits an information acquisition request corresponding to the determined use to the server device via the communication unit is provided.
    The control unit is a terminal device that displays information received from the server device on the display unit.
  6. 　表示部と、通信部と、撮影部と、前記撮影部が撮影した画像に含まれるコード情報を検出するコード検出部と、操作を受け付ける操作部と、前記操作部が受け付けた操作に基づいて前記コード情報の用途を判定し、判定した用途に対応した情報の取得要求を、前記通信部を介してサーバー装置に送信する制御部と、を備える端末装置と、
    　前記取得要求に対応した情報を前記端末装置に送信する前記サーバー装置と、を備え、
    　前記制御部は、前記サーバー装置から受信した情報を前記表示部に表示させる、サービス提供システム。     The display unit, the communication unit, the photographing unit, the code detection unit that detects the code information included in the image captured by the photographing unit, the operation unit that accepts the operation, and the operation unit that accepts the operation. A terminal device including a control unit that determines the use of code information and transmits an information acquisition request corresponding to the determined use to the server device via the communication unit.
    The server device that transmits information corresponding to the acquisition request to the terminal device is provided.
    The control unit is a service providing system that displays information received from the server device on the display unit.
PCT/JP2021/019424 2020-07-06 2021-05-21 Program, terminal device, and service providing system WO2022009539A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020116685A JP2022014385A (en) 2020-07-06 2020-07-06 Program, terminal device, and service providing system
JP2020-116685 2020-07-06

Publications (1)

Publication Number Publication Date
WO2022009539A1 true WO2022009539A1 (en) 2022-01-13

Family

ID=79552906

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/019424 WO2022009539A1 (en) 2020-07-06 2021-05-21 Program, terminal device, and service providing system

Country Status (2)

Country Link
JP (1) JP2022014385A (en)
WO (1) WO2022009539A1 (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3076649U (en) * 2000-09-26 2001-04-13 國榮 張 Microwave oven with barcode reading function

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3076649U (en) * 2000-09-26 2001-04-13 國榮 張 Microwave oven with barcode reading function

Also Published As

Publication number Publication date
JP2022014385A (en) 2022-01-19

Similar Documents

Publication Publication Date Title
WO2017004999A1 (en) Cooking system, terminal, server and cooking method
US9818083B1 (en) Association of item identifiers
US20140104385A1 (en) Method and apparatus for determining information associated with a food product
US10987804B2 (en) Robot device and non-transitory computer readable medium
CN105929713A (en) Control method and device of cooking electric appliance, and cooking electric appliance
US11528785B2 (en) Heating cooker, method of controlling heating cooker, and heating cooking system
JP2007024487A (en) Cooker, cooking system, and cooking control method therefor
US20160086513A1 (en) Server apparatus, data integration method and electronic device
JP2015158351A (en) Cooking apparatus, cooking method, cooking control program, and method of providing cooking information
JP5613592B2 (en) Bar code reader
CN113138564A (en) Cooking method and system based on intelligent steaming and baking oven, intelligent steaming and baking oven and medium
JP2019192091A (en) Food information providing system
WO2022009539A1 (en) Program, terminal device, and service providing system
JP2019219766A (en) Analysis device, analysis system, and analysis program
JP2019020981A (en) Cooking support system, cooking support method, and cooking support program
JP2017021650A (en) Cooking recipe creation method and program
US9508323B2 (en) Appliance control system and method for controlling an appliance based on processing done by another appliance
CN110990102B (en) Information code acquisition method and device, cooking utensil and storage medium
US9369515B2 (en) Communication method, communication device, appliance control method, and appliance control device
CN110689945A (en) Method, device and storage medium for recipe collocation
WO2022074955A1 (en) Information acquisition device, information acquisition program, and information acquisition method
CN113325722B (en) Multi-mode implementation method and device for intelligent cooking and intelligent cabinet
WO2017164050A1 (en) Heat-cooking device, heat-cooking device control method, and heat-cooking system
TWI554758B (en) A rapid-test system
CN107991966A (en) A kind of food intelligent monitoring method, device and intelligent health kitchen system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21837841

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21837841

Country of ref document: EP

Kind code of ref document: A1