JP2023169440A - Terminal, server, method for executing task, and method for providing information - Google Patents

Abstract

To provide a technique for processing a task on an item.SOLUTION: A terminal according to an aspect of the present disclosure includes at least one memory and at least one processor. The at least one processor can send identification information of an item to a server, receive item information of the item from the server, and execute a task on the item on the basis of the item information. The task on the item is an operation on the item or deduction of information on the item.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a terminal, a server, a task execution method, and an information provision method.

The use of robots is expanding at manufacturing and logistics sites. For example, in factories, robots are being introduced in various manufacturing processes. In addition, robots are being used in distribution warehouses to transport and remove products.

With the spread of robots, the introduction of robots into offices and homes is being considered. For example, consideration is being given to having robots grasp and move objects in offices and homes.

US Patent Application Publication No. 2020/0014761

An object of the present disclosure is to provide a technique for processing any task that targets an article.

In order to solve the above problems, one aspect of the present disclosure includes at least one memory and at least one processor, and the at least one processor transmits identification information of an article to a server, and the server and executing a task related to the article based on the article information, and the task related to the article includes at least the operation of the article or the operation of the article. Any inference of information regarding a terminal.

1 is a schematic diagram illustrating an article handling system according to one embodiment of the present disclosure. FIG. FIG. 2 is a block diagram showing the functional configuration of a server according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing the functional configuration of a terminal according to an embodiment of the present disclosure. 1 is a diagram illustrating a neural network according to one embodiment of the present disclosure. FIG. 2 is a flowchart illustrating a process of transmitting and receiving article information by a server according to an embodiment of the present disclosure. It is a flowchart which shows article operation control processing by a terminal according to one example of the present disclosure. FIG. 2 is a block diagram showing the hardware configuration of a server and a terminal according to an embodiment of the present disclosure.

Embodiments of the present disclosure will be described below based on the drawings.

In the following embodiments, some task is performed with the object, such as manipulating the object (e.g., grasping, moving, transporting, etc.), inferring information about the object (e.g., recognition, inspection, abnormality detection, etc.), presenting information about the object, etc. (e.g., robots, smart speakers, AR (Augmented Reality)/MR (Mixed Reality) glasses, smartphones, game consoles, mobile objects, etc.) and goods. A server that provides article information to a terminal is disclosed. Note that a plurality of terminals and servers may be provided. Further, one or more terminals among the plurality of terminals may perform the role of a server.

The following embodiments will be described with a focus on a terminal that performs a physical operation on an article as a task targeting an article, but the tasks according to the present disclosure are not limited to this. For example, tasks according to the present disclosure may involve physical interaction with the item, such as grasping, pushing, or pulling the item (e.g., arranging, displaying, etc.), transporting the item (e.g., transporting the item on a belt conveyor, etc.). This includes not only tasks that involve physical contact, but also object recognition (e.g., conveying the object's position and orientation to other devices), object inspection, and anomaly detection (e.g., comparison with a 3D model of the object, damage detection, etc.). Detection, etc.) may include tasks that do not involve physical contact with the item (such as inference of information about the item).
[Article manipulation system]
First, with reference to FIG. 1, an article handling system according to an embodiment of the present disclosure will be described. FIG. 1 is a schematic diagram illustrating an article handling system according to one embodiment of the present disclosure.

As shown in FIG. 1, the article handling system 10 includes a server 100 and a terminal 200. The server 100 is communicatively connected to a user's terminal 200 via a network, for example, and the terminal 200 uses the article information acquired from the server 100 to operate an article corresponding to the acquired article information. The terminal 200 may be, for example, a robot that grasps, moves, etc. objects. However, the terminal according to the present disclosure is not limited to this, and can include smart speakers (for example, outputting audio and images related to handling information and campaign information for articles as products), AR/MR glasses (for example, outputting 3D models of articles), Presentation of explanations, information, etc. used), smartphones (e.g., AR apps), game consoles (use of images captured by the attached camera in the game), devices equipped with cameras, sensors, displays, etc. (e.g., Devices that monitor the moisture status, nutritional status, sunlight status, etc. of houseplants, and supply moisture and nutrients based on the registered appropriate status, and adjust sunlight; projectors that project videos, etc. onto ceilings and walls; and (a control program for the device). Further, the terminal 200 may recognize the article using the 3D model of the article acquired as the article information, or may grasp the article. Furthermore, the article handling system 10 can be implemented as a task processing system that handles tasks related to articles.

For example, the article information may be information about the article, such as the shape, weight, type, etc. of the article. In addition, product information is information that is directly or indirectly necessary when operating the product, such as precautions when operating the product (for example, parts to grip, fragile parts, etc.). There may be.

Further, when the terminal 200 operates the article using a neural network model, the article information may be update information such as parameters and slope information for the neural network model. In this case, the terminal 200 may, for example, replace the parameters of the neural network model for article manipulation stored in advance with the parameters indicated in the acquired article information, or The neural network model may be updated according to the gradient information indicated in the information. Then, the terminal 200 inputs the image of the article obtained by a camera or the like and the operation instruction from the user into the updated neural network model, and operates the article according to the operation procedure output from the neural network model. Note that the information input to the neural network may be information regarding the sound emitted by the article acquired by the microphone.

For example, in the case of gripping, the product information is information that depends on the installed gripping model (it may be additional parameters for the neural network model, or if it is keypoint-based, it may be the keypoints for the product) It may be. In addition, the article information includes parts that must not be touched for grasping on the 3D model, weight, center of gravity, soft parts, hard parts for gripping, placing, etc. described according to any protocol. It may also indicate default values such as load capacity, whether objects can be stacked on top of it, whether it can be placed on top of something, whether it can be pushed away, etc. In addition, the item information may be a default tidying place (for example, the furniture type is registered for furniture, and the tidying place may be specified by a combination of room type (for example, kitchen) and storage type (for example, cupboard). , in the case of placement, the article information may indicate a default posture (for example, do not place a plastic bottle upside down).In the case of a plastic bottle, the article information may include "cap", "body", "label", etc. ”, and functional annotations such as ``the part to hold when pouring or transporting'' or ``the part to twist to open'' may also be shown for each part. In the case of assembled furniture, it may be a CG model (displayed in AR or displayed on a smartphone) that shows how to assemble it using a 3D model.In addition, in the case of a houseplant, the product information may be a CG model that shows how to assemble it. The item information may also include information such as a 3D model for operating the item, and the terminal 200 may indicate how to operate the item based on this item information. In addition, the product information may include information regarding the product, and the terminal 200 may output this information using at least audio or images. Various information including additional information may be added or updated (if necessary other than at the time of initial registration) and may be push-transmitted to the terminal 200.

In step S101, the server 100 acquires article information regarding the article from the article provider 50. The product provider 50 generates in advance product information indicating the shape information, weight, product type, 3D model such as 3D CAD, notes, neural network model update information, etc. of the product to be sold or provided to the user, and sends the product information to the server 100. Register. Note that the product provider 50 may be realized by a computer of a company that manufactures, sells, etc. the product. However, the article information according to the present disclosure is not limited to this, and may be generated by the server 100. Further, the neural network model update information may be generated by the server 100. In this case, the product provider 50 may provide the server 100 with an image set used for training the neural network or an image set (with correct answer annotations) for verifying the performance of the trained neural network. Further, the neural network model update information may be generated by the terminal 200. In this case, the generated update information is transmitted from each terminal 200 to the server 100, and the server 100 generates update information for the neural network model based on the individual update information for each terminal 200 received from each terminal 200, It may also be distributed to each terminal 200. Alternatively, the article provider 50 may send the article itself to the administrator of the server 100 and entrust registration of article information.

In step S102, the server 100 generates article identification information for identifying the article based on the acquired article information, and transmits the generated article identification information to the article provider 50. Further, the server 100 associates and holds the article information acquired from the article provider 50 and the article identification information of the article. However, the article identification information according to the present disclosure is not limited thereto, and may be generated by the article provider 50 and provided to the server 100. Further, information useful for the terminal 200 may be supplemented by a simulator or the like inside the server 100. For example, in addition to training a neural network for vision, (if the terminal 200 is a robot) a simulation of movements such as grasping is performed to obtain update information for the neural network for grasping, information on a stable grasping posture, etc. , may be provided from the server 100.

In step S103, the terminal 200 acquires the article and the article identification information (article identification information) of the article from the article provider 50. For example, when a user purchases an item at a store, the terminal 200 captures the item identification information of the item by capturing an image of the manual attached to the item, the barcode written on the packaging box, etc. of the item. may be obtained. Further, the terminal 200 may store an image of the article in association with the article identification information. Thereby, when the terminal 200 identifies the object to be operated using a camera or the like, it is possible to extract the object identification information corresponding to the object. Moreover, the article identification information may be obtained from the article itself. For example, the article identification information may be written on the article itself, and the terminal 200 may acquire the article identification information by reading the description. Further, image information of the article acquired by the camera of the terminal 200 or the like may be used as the article identification information.

In step S104, the terminal 200 transmits the article identification information of the article to the server 100. For example, the terminal 200 may connect to the network itself and transmit the article identification information to the server 100, or the terminal 200 may transmit the article identification information to the server 100 via another communication device such as a user's smartphone. good.

In step S105, the server 100 transmits article information corresponding to the received article identification information to the terminal 200. The terminal 200 holds the received article information and controls operations on the article based on the article information. Note that the terminal 200 may acquire article information from the server 100 via another communication device such as a user's smartphone.

Here, the terminal 200 may acquire article information of the article from the server 100 when acquiring article identification information, or may acquire article information from the server 100 when operating the article.

Note that in the embodiment described above, the article provider 50 and the server 100 are separate devices, but the present disclosure is not limited to this, and the article provider 50 and the server 100 may be realized as the same device. .
[server]
Next, with reference to FIG. 2, a server 100 according to an embodiment of the present disclosure will be described. FIG. 2 is a block diagram showing the functional configuration of the server 100 according to an embodiment of the present disclosure.

As shown in FIG. 2, the server 100 includes an article information reception section 110, an article identification information allocation section 120, and an article information storage section 130.

The article information reception unit 110 receives article information about articles. Specifically, the article information reception unit 110 receives article information from the article provider 50. For example, the article information may indicate shape information, weight, article type, precautions, etc. of the article. Further, the article information may indicate update information of a neural network model for controlling operations on the article. For example, the update information may include parameters or gradient information for updating a trained neural network model.

The article identification information allocation unit 120 allocates article identification information for identifying the article to the article. Specifically, when article information is received from the article provider 50, the article identification information allocation unit 120 assigns article identification information that uniquely identifies the article of the received article information to the article, and uses the attached article identification information. is provided to the article provider 50. For example, the article identification information may be an alphanumeric character string, or may be convertible into a one-dimensional or two-dimensional barcode.

The article information storage unit 130 stores the received article information. Specifically, the article information storage unit 130 stores article information received from the article provider 50 in a database or the like in association with the article identification information of the article. As will be described later, when article identification information is acquired from the terminal 200 or a communication device such as a smartphone of the user connected to the terminal 200, the article information storage unit 130 stores article information corresponding to the acquired article identification information to the terminal. 200 or a communication device. Note that when image information of an article is acquired as article identification information, recognition processing is performed on this image information to identify the article, and article information corresponding to the identified article is provided to the terminal 200 or the communication device. Good too.

In one embodiment, the article information storage unit 130 may provide the terminal 200 with article information corresponding to the type information of the terminal 200. For example, if the terminal 200 is equipped with a machine learning model such as a neural network model for executing a task related to an article, the article information storage unit 130 may provide the terminal 200 with update information for the machine learning model. Therefore, the terminal 200 may include its own type information in the article information acquisition request to the server 100.

Further, in one embodiment, the terminal 200 transmits to the server 100 information regarding a machine learning model for executing a task related to an article, such as a neural network model owned by the terminal, and the article information storage unit 130 receives the received information. Article information transmitted to the terminal 200 may be controlled based on information regarding the machine learning model.

Further, in one embodiment, the server 100 may specify a task that targets an article to be executed by the terminal 200 based on the type information, and may transmit article information necessary for this task to the terminal 200.

Further, in one embodiment, the article information storage unit 130 may provide the terminal 200 with article information desired by the terminal 200. For example, when the terminal 200 desires the weight of an object to be operated, the article information storage unit 130 may provide the terminal 200 with the weight of the article as article information.

Note that in the embodiment described above, the server 100 provides the article information to the terminal 200, but the article information according to the present disclosure may be provided directly to the terminal 200 from the article provider 50. For example, additional or updated item information may be pushed from item provider 50 to terminal 200 from time to time.
[Terminal]
Next, with reference to FIG. 3, a terminal 200 according to an embodiment of the present disclosure will be described. FIG. 3 is a block diagram showing the functional configuration of the terminal 200 according to an embodiment of the present disclosure.

As shown in FIG. 3, the terminal 200 includes a user interface section 210, an information storage section 220, and a control section 230.

User interface unit 210 provides a user interface. Specifically, when the user obtains an article, the user interface unit 210 acquires the article identification information of the article. For example, the user interface unit 210 may acquire the article identification information by the user capturing an image of a barcode indicating the article identification information, or by inputting an alphanumeric character string indicating the article identification information. . Further, the user interface unit 210 may obtain article identification information from the article provider 50.

Further, upon receiving an operation instruction from the user, the user interface section 210 provides the operation instruction to the control section 230. Specifically, when the terminal 200 is implemented as a robot with an arm, the operation instruction may be to cause the terminal 200 to grasp, move, and transport an article. For example, the operation instruction may be a voice instruction by the user such as "Please move article X located at position A to position B." In this case, the terminal 200 may use any existing voice recognition technology to recognize the voice instruction as an operation instruction to "move article X from position A to position B." Note that the user interface unit 210 may accept a task registered in advance or a task registered by a user, and in this case, the registered object of the task may be added as an operation target.

The information storage unit 220 stores the acquired article identification information. Specifically, the information storage section 220 stores article identification information acquired by the user interface section 210. For example, the information storage unit 220 may store article identification information and an image of the article in association with each other. Furthermore, when article information has been acquired, the information storage unit 220 may store article identification information and article information in association with each other.

The control unit 230 controls operations on articles. Specifically, the control unit 230 extracts the article information of the article to be operated according to an operation instruction from the user, and controls the operation on the article based on the extracted article information. Note that the control unit 230 may add the registered task target article as an operation target for a task registered in advance or a task registered by a user.

For example, if the operation instruction is to instruct the terminal 200, which is implemented as a robot with an arm, to move a book placed on a table to a bookshelf, the control unit 230 may take an image of the book to be operated. , extracts article identification information corresponding to the captured image of the book from the information storage unit 220.

When article information corresponding to the extracted article identification information is stored in the information storage section 220, the control section 230 controls the arm to move the book from the tabletop to the bookshelf according to the article information. For example, when the article information indicates book shape information (e.g., A5 size, A4 size, number of pages, etc.) and article type (e.g., soft cover, hard cover, etc.), the control unit 230 controls the The robot may control the arm to grasp the book and move it to the bookshelf.

On the other hand, if the article information corresponding to the extracted article identification information is not stored in the information storage section 220, the control section 230 transmits an acquisition request for the article information of the article together with the article identification information to the server 100, and the server 100 to obtain the requested article information. Then, the control unit 230 stores the acquired item information in the information storage unit 220, and similarly controls the arm to move the book from the tabletop to the bookshelf according to the item information.

Further, a machine learning model for controlling operations on the article may be provided in advance from the server 100 to the terminal 200, and the control unit 230 uses the machine learning model to control operations on the article according to operation instructions from the user. Operation may be controlled. In this case, the article information may be update information for a machine learning model for appropriately operating the article.

For example, if the machine learning model is a neural network model provided in advance to the terminal 200, the control unit 230 acquires update information of the object to be operated, and updates the neural network model according to the acquired update information. Good too. Further, the control unit 230 may transmit the neural network model before updating to the server 100, receive update information of the transmitted neural network model from the server 100, and update the neural network model according to the received update information. . Specifically, when the update information is a parameter of a neural network model, the control unit 230 may update the neural network model using the acquired parameter and control the operation on the article using the updated neural network model. Further, when the update information is gradient information of a neural network model, the control unit 230 may update the neural network model using the acquired gradient information, and control operations on the article using the updated neural network model. That is, the terminal 200 may acquire a base machine learning model from the server 100 and update the acquired machine learning model based on update information transmitted from the server 100. In this case, the server 100 can manage the machine learning model information held by the terminal 200 in association with the terminal ID. Further, the server 100 can transmit appropriate update information to the terminal 200 based on the object identification information acquired from the terminal 200. Alternatively, the machine learning model held by the terminal 200 may be transmitted to the server 100, and update information for this machine learning model may be generated on the server side. In this way, the update information may include one or both of the parameters of the machine learning model and the gradient information of the machine learning model.

As an example, assume that the neural network model provided to the terminal 200 in advance includes a feature extraction section, an object identification section, and an object identification extension section shown in FIG. Here, the feature extraction unit may be realized as a convolutional neural network, for example. Further, the object identification unit may be a softmax layer, a classifier, or a matching unit of a neural network that matches information (vectors) for each object type with features. The object identification extension adds object information about new objects. In this case, the article information for the object to be operated may indicate an additional part to the object identification section, as shown as an object identification extension section. Upon acquiring such updated information, the control unit 230 adds an additional part to the object identification unit according to the updated information, and controls operations on the article using the feature extraction unit and the object identification extension unit. Specifically, the control unit 230 inputs the image of the article to the feature extraction unit and obtains a feature vector or a feature map. Then, the control unit 230 inputs the acquired feature vector or feature map to the object identification unit 320, identifies the object type according to the acquired feature vector or feature map and the additional object information, and identifies the object according to the identified object type. Control operations on. Here, the object type may be a bounding box or a segmentation mask in which the object is imaged.

In one embodiment, the control unit 230 may transmit type information of the terminal 200 to the server 100 and obtain article information corresponding to the article identification information and type information from the server 100. That is, the article information may be prepared according to the type of terminal 200 (for example, robot, smart speaker, AR (Augmented Reality)/MR (Mixed Reality) glasses, smartphone, game console, mobile object, etc.). The control unit 230 may transmit the type information of the terminal 200 to the server 100 and obtain article information corresponding to the type of the terminal 200 from the server 100.
[Product information transmission/reception processing]
Next, with reference to FIG. 5, article information transmission/reception processing according to an embodiment of the present disclosure will be described. The transmission/reception process is executed by the server 100 described above, and can be realized, for example, by one or more processors executing a program stored in one or more memories of the server 100. FIG. 5 is a flowchart showing a process for transmitting and receiving article information according to an embodiment of the present disclosure.

As shown in FIG. 5, in step S201, the server 100 receives article information from the article provider 50. For example, the product provider 50 may send a registration request for product information about the product to the server 100 before selling the product. Here, the article information may include any one or more of information regarding any of the shape of the article, the weight of the article, the type of article, notes on operating the article, and the 3D model of the article.

In step S202, the server 100 assigns article identification information to the article of the acquired article information, and returns the attached article identification information to the article provider 50. The server 100 associates the acquired article information and article identification information and stores them in a database or the like.

In step S203, the server 100 receives an article information acquisition request including article identification information from the terminal 200. For example, when a user gives an operation instruction for a certain article to the terminal 200, the terminal 200 transmits article identification information of the article to be operated to the server 100.

In step S204, the server 100 returns article information corresponding to the received article identification information to the terminal 200. For example, the article information may indicate shape information, weight, article type, precautions, etc. of the article.

Here, the server 100 may provide article information corresponding to the terminal type of the terminal 200, or may provide the terminal 200 with article information desired by the terminal 200. For example, if the terminal 200 is a terminal type that has a neural network model for controlling operations on articles, the server 100 may provide update information of the neural network model as article information. For example, the update information may include neural network model parameters or gradient information.

Note that in the embodiment described above, the server 100 provides the article information to the terminal 200, but the article information according to the present disclosure may be provided directly to the terminal 200 from the article provider 50. For example, additional or updated item information may be pushed from item provider 50 to terminal 200 from time to time.
[Article operation control processing]
Next, referring to FIG. 6, article manipulation control processing according to an embodiment of the present disclosure will be described. The article manipulation control process is executed by the terminal 200 described above, and can be realized, for example, by one or more processors executing a program stored in one or more memories of the terminal 200. FIG. 6 is a flowchart showing article manipulation control processing according to an embodiment of the present disclosure.

As shown in FIG. 6, in step S301, the terminal 200 acquires article identification information from the article provider 50. For example, when the user purchases an item, the terminal 200 may acquire item identification information provided by the item provider 50. The terminal 200 may associate and hold the acquired article identification information and the image of the article. Note that the terminal 200 may acquire the article identification information not from the article provider 50 but from the purchased article itself or its accessories. Further, the terminal 200 may acquire article information via another communication device such as a user's smartphone.

In step S302, the terminal 200 receives operation instructions for the article from the user. Specifically, the operation instruction may be for the terminal 200 to grasp and move the article. For example, the operation instruction may be a voice instruction by the user such as "Please move article X located at position A to position B." In this case, the terminal 200 may use any existing voice recognition technology to recognize the voice instruction as an operation instruction to "move article X from position A to position B." Note that step S302 may be omitted. Even if no operation instructions are given for the article, the terminal 200 may acquire article information about a newly acquired (purchased) article from the server 100.

In step S303, the terminal 200 transmits the article identification information of the operation target article to the server 100. For example, the terminal 200 images the object to be operated using a camera or the like, and identifies the object identification information from the image corresponding to the object. Then, the terminal 200 transmits an article information acquisition request including the specified article identification information to the server 100. Further, the terminal 200 may include the terminal type and/or the desired article information type (for example, shape information, weight, article type, notes, neural network model update information, etc.) in the article information acquisition request. Note that the terminal 200 may communicate with the server 100 via another communication device such as a user's smartphone.

Note that if the terminal 200 has already acquired the article information of the article, the terminal 200 may omit step S303.

In step S304, the terminal 200 receives article information from the server 100. The terminal 200 may hold the received article information in association with article identification information.

In step S305, the terminal 200 controls operations on the article according to the article information. For example, if the operation instruction is to grip an article and the article information is shape information of the article, the terminal 200 grips the article using an arm or the like based on the shape information. Further, when the operation instruction is to move an article and the article information is update information of a neural network model, the terminal 200 updates the neural network model according to the acquired update information, and uses the updated neural network model to create an image of the article. input, and operate the item according to the output value.

Note that the present disclosure is not limited to the embodiments described above. In one embodiment, the server 100 and the terminal 200 may share machine learning model information. Specifically, the server 100 may hold a global machine learning model and transmit it to the terminal 200. Further, a global machine learning model may be stored in the terminal 200 in advance at an initial stage such as at the time of manufacture or sale. Alternatively, the server 100 may acquire the machine learning model held by the terminal 200 without transmitting the machine learning model.

Further, in one embodiment, when the terminal 200 transmits the identification information of the object to be operated to the server 100, the terminal 200 may encrypt the information. Further, the server 100 may generate or select update information instead of the entire machine learning model for manipulating the article. For example, the server 100 may learn a machine learning model in advance for the article to be operated by the terminal 200, and generate update information for the machine learning model. The update information may be information necessary to update the global machine learning model for the article to be operated by the terminal 200. Further, the update information may be, for example, some parameters of a neural network model, gradient information, or information specific to the article (or its 3D model). Further, the server 100 may newly generate update information for an article that is an operation target of the terminal 200.

Further, in one embodiment, when the server 100 receives a machine learning model from the terminal 200, the server 100 sends the terminal 200 to the terminal 200 as update information necessary for updating the machine learning model for the article to be operated by the terminal 200. You may also send it to

Further, in one embodiment, the server 100 may transmit update information of the machine learning model to the terminal 200. The terminal 200 may update the machine learning model using the received update information and operate the article. The terminal 200 may update the global machine learning model using the received update information. The terminal 200 may update the weights of the neural network model using the gradient information included in the update information. Furthermore, the terminal 200 may update the model by using parameters included in the update information as some parameters of the neural network model. Note that the neural network model is an example of a machine learning model, and other types of models may be used.

According to the embodiment described above, the server 100 does not need to transmit the entire machine learning model, and can generate update information specific to the object to be operated using abundant computational resources. Further, the server 100 can construct a machine learning model specific to the terminal 200 instead of a general-purpose model common to various terminals 200, and can improve the operation performance of the article.

Note that the article information transmitted from the server 100 to the terminal 200 may include all of the article information described in the above embodiment, or may include a combination of some article information. Further, the terminal 200 may perform a task related to the article using part of the received article information.
[Hardware configuration]
Part or all of each device (server 100 or terminal 200) in the embodiments described above may be configured with hardware, or may be executed by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like. It may be configured by information processing of software (program). In the case where the software is configured by software information processing, the software that realizes at least some of the functions of each device in the above-described embodiments may be stored on a flexible disk, CD-ROM (Compact Disc-Read Only Memory), or USB (Universal Software information processing may be executed by storing the information in a non-temporary storage medium (non-temporary computer-readable medium) such as Serial Bus (Serial Bus) memory and reading it into a computer. Further, the software may be downloaded via a communication network. Furthermore, information processing may be performed by hardware by implementing software in a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The type of storage medium that stores software is not limited. The storage medium is not limited to a removable one such as a magnetic disk or an optical disk, but may be a fixed storage medium such as a hard disk or memory. Further, the storage medium may be provided inside the computer or may be provided outside the computer.

FIG. 7 is a block diagram showing an example of the hardware configuration of each device (server 100 or terminal 200) in the embodiment described above. Each device includes, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, which are connected via a bus 76. It may also be realized as a computer 7.

Although the computer 7 in FIG. 7 includes one of each component, it may include a plurality of the same components. Although one computer 7 is shown in FIG. 7, the software may be installed on multiple computers, and each of the multiple computers may execute the same or different part of the software. Good too. In this case, it may be a form of distributed computing in which each computer communicates via the network interface 74 or the like to execute processing. In other words, each device (server 100 or terminal 200) in the embodiment described above is configured as a system that realizes functions by one or more computers executing instructions stored in one or more storage devices. You can. Alternatively, the information transmitted from the terminal may be processed by one or more computers provided on the cloud, and the processing results may be sent to the terminal.

Various calculations of each device (server 100 or terminal 200) in the embodiments described above may be executed in parallel using one or more processors or using multiple computers via a network. . Further, various calculations may be distributed to a plurality of calculation cores within the processor and executed in parallel. Further, a part or all of the processing, means, etc. of the present disclosure may be executed by at least one of a processor and a storage device provided on a cloud that can communicate with the computer 7 via a network. In this way, each device in the embodiments described above may be in the form of parallel computing using one or more computers.

The processor 71 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) including a computer control device and an arithmetic device. Further, the processor 71 may be a semiconductor device or the like including a dedicated processing circuit. The processor 71 is not limited to an electronic circuit using an electronic logic element, but may be realized by an optical circuit using an optical logic element. Further, the processor 71 may include an arithmetic function based on quantum computing.

The processor 71 can perform arithmetic processing based on data and software (programs) input from each device in the internal structure of the computer 7, and can output calculation results and control signals to each device. The processor 71 may control each component constituting the computer 7 by executing the OS (Operating System) of the computer 7, applications, and the like.

Each device (server 100 or terminal 200) in the embodiment described above may be realized by one or more processors 71. Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or one or more electronic circuits arranged on two or more chips or two or more devices. You can also point. When using multiple electronic circuits, each electronic circuit may communicate by wire or wirelessly.

The main storage device 72 is a storage device that stores instructions and various data to be executed by the processor 71, and the information stored in the main storage device 72 is read out by the processor 71. The auxiliary storage device 73 is a storage device other than the main storage device 72. Note that these storage devices are any electronic components capable of storing electronic information, and may be semiconductor memories. Semiconductor memory may be either volatile memory or nonvolatile memory. The storage device for storing various data in each device (server 100 or terminal 200) in the embodiments described above may be realized by the main storage device 72 or the auxiliary storage device 73, and may be realized by the built-in memory built into the processor 71. It may be realized by For example, the storage unit 72 in the embodiment described above may be realized by the main storage device 72 or the auxiliary storage device 73.

A plurality of processors may be connected (combined) to one storage device (memory), or a single processor may be connected to one storage device (memory). A plurality of storage devices (memories) may be connected (combined) to one processor. In the case where each device (server 100 or terminal 200) in the embodiment described above is composed of at least one storage device (memory) and a plurality of processors connected (coupled) to this at least one storage device (memory), The present invention may include a configuration in which at least one processor among the plurality of processors is connected (coupled) to at least one storage device (memory). Further, this configuration may be realized by a storage device (memory) and a processor included in a plurality of computers. Furthermore, a configuration in which a storage device (memory) is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

The network interface 74 is an interface for connecting to the communication network 8 wirelessly or by wire. As the network interface 74, an appropriate interface such as one that complies with existing communication standards may be used. Information may be exchanged with an external device 9A connected via the communication network 8 through the network interface 74. Note that the communication network 8 may be any one of a WAN (Wide Area Network), a LAN (Local Area Network), a PAN (Personal Area Network), etc., or a combination thereof, and can be used to connect the computer 7 and the external device 9A. It is sufficient that information is exchanged between them. Examples of WAN include the Internet, examples of LAN include IEEE802.11 and Ethernet (registered trademark), and examples of PAN include Bluetooth (registered trademark) and NFC (Near Field Communication).

The device interface 75 is an interface such as a USB that is directly connected to the external device 9B.

The external device 9A is a device connected to the computer 7 via a network. The external device 9B is a device directly connected to the computer 7.

The external device 9A or the external device 9B may be an input device, for example. The input device is, for example, a device such as a camera, microphone, motion capture, various sensors, keyboard, mouse, or touch panel, and provides the acquired information to the computer 7. Alternatively, the device may be a device including an input section, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

Further, the external device 9A or the external device 9B may be an output device, for example. The output device may be a display device such as an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), or an organic EL (Electro Luminescence) panel, or output audio or the like. It may also be a speaker or the like. Alternatively, the device may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

Further, the external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage such as an HDD.

Further, the external device 9A or the external device 9B may be a device having some functions of the components of each device (server 100 or terminal 200) in the embodiments described above. That is, the computer 7 may transmit or receive part or all of the processing results of the external device 9A or the external device 9B.

In this specification (including claims), the expression "at least one (one) of a, b, and c" or "at least one (one) of a, b, or c" (including similar expressions) When used, it includes any of a, b, c, a-b, a-c, b-c, or a-b-c. Further, each element may include multiple instances, such as a-a, a-b-b, a-a-b-b-c-c, etc. Furthermore, it also includes adding other elements other than the listed elements (a, b and c), such as having d as in a-b-c-d.

In this specification (including claims), when expressions such as "using data as input/based on data/in accordance with/according to" (including similar expressions) are used, unless otherwise specified, This includes cases where various data itself is used as input, and cases where various data subjected to some processing (for example, noise added, normalized, intermediate representation of various data, etc.) are used as input. In addition, if it is stated that a certain result is obtained "based on/according to/according to data", this includes cases where the result is obtained only based on the data, and other data other than the data, It may also include cases where the results are obtained under the influence of factors, conditions, and/or states. In addition, if it is stated that "data will be output", if there is no special notice, various data itself may be used as output, or data that has been processed in some way (for example, data with added noise, normal This also includes cases in which the output is digitized data, intermediate representations of various data, etc.).

In this specification (including the claims), when the terms "connected" and "coupled" are used, the terms "connected" and "coupled" refer to direct connection/coupling and indirect connection/coupling. , electrically connected/coupled, communicatively connected/coupled, functionally connected/coupled, physically connected/coupled, etc., without limitation. intended as a term. The term should be interpreted as appropriate depending on the context in which the term is used, but forms of connection/coupling that are not intentionally or naturally excluded are not included in the term. Should be construed in a limited manner.

In this specification (including the claims), when the expression "A configured to B" is used, it means that the physical structure of element A is capable of performing operation B. configuration, and includes a permanent or temporary setting/configuration of element A being configured/set to actually perform operation B. good. For example, if element A is a general-purpose processor, the processor has a hardware configuration that can execute operation B, and can perform operation B by setting a permanent or temporary program (instruction). It only needs to be configured to actually execute. In addition, if element A is a dedicated processor or a dedicated arithmetic circuit, the circuit structure of the processor is configured to actually execute operation B, regardless of whether control instructions and data are actually attached. It is sufficient if it has been implemented.

In this specification (including the claims), when terms meaning inclusion or possession (for example, "comprising/including" and "having", etc.) are used, the object of the term It is intended as an open-ended term, including the case of containing or possessing something other than the object indicated. If the object of a term meaning inclusion or possession is an expression that does not specify a quantity or suggests a singular number (an expression with a or an as an article), the expression shall be interpreted as not being limited to a specific number. It should be.

In this specification (including the claims), expressions such as "one or more" or "at least one" are used in some places, and quantities are specified in other places. Even if an expression is used that suggests no or singular (an expression with the article a or an), it is not intended that the latter expression means "one". In general, expressions that do not specify a quantity or imply a singular number (expressions with the article a or an) should be construed as not necessarily being limited to a particular number.

In this specification, if it is stated that a specific effect (advantage/result) can be obtained with respect to a specific configuration of a certain embodiment, unless there is a reason to the contrary, one or more other components having the configuration may be used. It should be understood that the same effect can also be obtained with the embodiment. However, it should be understood that the presence or absence of the said effect generally depends on various factors, conditions, and/or states, and that the said effect is not necessarily obtained by the said configuration. The effect is only obtained by the configuration described in the Examples when various factors, conditions, and/or states, etc. are satisfied, and in the claimed invention that specifies the configuration or a similar configuration. However, this effect is not necessarily obtained.

In this specification (including the claims), when terms such as "maximize" are used, it refers to finding a global maximum value, finding an approximate value of a global maximum value, or finding a local maximum value. and approximating the local maximum value, and should be interpreted as appropriate depending on the context in which the term is used. It also includes finding approximate values of these maximum values probabilistically or heuristically. Similarly, terms such as "minimize" are used to refer to finding a global minimum, finding an approximation to a global minimum, finding a local minimum, and finding a local minimum. It includes approximations of values and should be interpreted as appropriate depending on the context in which the term is used. It also includes finding approximate values of these minimum values probabilistically or heuristically. Similarly, terms such as "optimize" are used to refer to finding a global optimum, finding an approximation to a global optimum, finding a local optimum, and determining a local optimum. It includes approximations of values and should be interpreted as appropriate depending on the context in which the term is used. It also includes finding approximate values of these optimal values probabilistically or heuristically.

In this specification (including claims), when multiple pieces of hardware perform a predetermined process, each piece of hardware may cooperate to perform the predetermined process, or some of the hardware may perform the predetermined process. You may do all of the above. Further, some hardware may perform part of a predetermined process, and another piece of hardware may perform the rest of the predetermined process. In this specification (including claims), when expressions such as "one or more hardware performs the first process, and the one or more hardware performs the second process" are used , the hardware that performs the first processing and the hardware that performs the second processing may be the same or different. In other words, the hardware that performs the first processing and the hardware that performs the second processing may be included in the one or more pieces of hardware. Note that the hardware may include an electronic circuit, a device including an electronic circuit, or the like.

In this specification (including claims), when multiple storage devices (memories) store data, each storage device (memory) among the multiple storage devices (memories) stores only part of the data. It may be stored, or the entire data may be stored.

Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, changes, substitutions, and partial deletions are possible without departing from the conceptual idea and spirit of the present invention derived from the content defined in the claims and equivalents thereof. For example, in all the embodiments described above, when numerical values or formulas are used in the explanation, they are shown as examples, and the invention is not limited to these. Further, the order of each operation in the embodiment is shown as an example, and the order is not limited to this.

10 Article operation system 50 Article provider 100 Server 110 Article information reception unit 120 Article identification information allocation unit 130 Article information storage unit 200 Terminal 210 User interface unit 220 Information storage unit 230 Control unit

Claims (11)

at least one memory;
at least one processor;
The at least one processor includes:
Sending the identification information of the item to the server;
receiving article information regarding the article from the server;
performing a task related to the article based on the article information;
configured to be executable,
The task regarding the article is at least one of manipulating the article or inferring information regarding the article;
terminal. Claim: The article information includes at least information regarding any of the shape of the article, the weight of the article, the type of the article, considerations when operating the article, or a 3D model of the article. The terminal described in 1. The article information includes a 3D model of the article,
The task is at least one of grasping the article using the 3D model or recognizing the article using the 3D model.
The terminal according to claim 1 or 2. 4. The terminal according to claim 1, wherein the article information includes update information of a machine learning model for executing a task related to the article. The at least one processor is further configured to receive the machine learning model from the server;
The update information is update information of the received machine learning model,
The terminal according to claim 4. The at least one processor is further configured to transmit the machine learning model before updating to the server,
The update information is update information of the transmitted machine learning model,
The terminal according to claim 4. 7. The terminal according to claim 4, wherein the update information includes at least one of parameters of the machine learning model or gradient information of the machine learning model. The at least one processor further comprises:
transmitting type information of the terminal to the server;
receiving the article information corresponding to the identification information and the type information from the server;
configured to be executable,
8. A terminal according to any one of claims 1 to 7. at least one memory;
at least one processor;
The at least one processor includes:
receiving identification information of the article from the terminal;
transmitting article information regarding the article to the terminal;
configured to be executable,
The item information is information necessary for the terminal to operate at least the item or infer information regarding the item;
server. at least one processor transmitting identification information of the item to the server;
the at least one processor receiving article information regarding the article from the server;
the at least one processor performing a task regarding the article based on the article information;
has
The task regarding the article is at least one of manipulating the article or inferring information regarding the article;
How to perform the task. at least one processor receiving identification information of the item from the terminal;
the at least one processor transmitting article information regarding the article to the terminal;
has
The item information is information necessary for the terminal to operate at least the item or infer information regarding the item;
Information provision method.

Images