WO2019225875A1

WO2019225875A1 - Method and apparatus for tracking inventory

Info

Publication number: WO2019225875A1
Application number: PCT/KR2019/005187
Authority: WO
Inventors: 프락체크로만; 타카주크야쿠브
Original assignee: 삼성전자 주식회사
Priority date: 2018-05-25
Filing date: 2019-04-30
Publication date: 2019-11-28
Also published as: KR20190140509A

Abstract

An electronic device, according to various embodiments, comprises: a plurality of sensors including a visual sensor and a weight sensor; a memory; and at least one processor, wherein the at least one processor may be configured to: determine whether at least one object has been inserted into the electronic device by using at least one of the plurality of sensors; acquire an image of the at least one object by using the visual sensor, on the basis of the insertion of the at least one object; identify the type of the at least one object corresponding to the image by using a learning model learned through an artificial intelligence algorithm; obtain residual amount information of the at least one object by using the learning model and the weight sensor; and store the type of the object and the residual amount information of the object in the memory. Other embodiments are possible.

Description

Inventory tracking method and device

Various embodiments of the present disclosure relate to a method and an apparatus for tracking inventory related to an object to be inserted into or withdrawn from an electronic device.

Recently, electronic devices using artificial intelligence systems have been used.

Artificial intelligence systems, unlike conventional rule-based smart systems, are machines that learn and judge themselves and become smart. As the artificial intelligence system is used, the recognition rate is improved and the user's taste can be understood more accurately, and the existing rule-based smart system is gradually replaced by the deep learning-based artificial intelligence system. Artificial intelligence technology may be composed of elementary technologies utilizing machine learning (eg, deep learning) and machine learning. Machine learning is an algorithm technology that classifies / learns characteristics of input data by itself. Element technology is a technology that simulates the functions of human brain cognition and judgment by using machine learning algorithms such as deep learning. It may consist of technical fields such as understanding, reasoning / prediction, knowledge representation, motion control, and the like.

The various fields in which artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing and applying / processing a human language / character, and may include natural language processing, machine translation, dialogue system, question and answer, speech recognition / synthesis, and the like. Visual understanding is a technology for recognizing and processing objects as human vision, and may include object recognition, object tracking, image search, person recognition, scene understanding, spatial understanding, image enhancement, and the like. Inference prediction is a technique of determining, logically inferring, and predicting information, and may include knowledge / probability based inference, optimization prediction, preference based planning, and recommendation. Knowledge expression is a technology for automatically processing human experience information into knowledge data, and may include knowledge construction (data generation / classification), knowledge management (data utilization), and the like. Motion control is a technology for controlling autonomous driving of a vehicle and movement of a robot, and may include motion control (navigation, collision, driving), operation control (action control), and the like.

An electronic device according to various embodiments of the present disclosure may include a structure capable of storing an object therein (or an upper portion). The user may want to automatically manage an object that is drawn in or drawn out of the electronic device.

Various embodiments of the present disclosure may provide an electronic device that automatically tracks and manages an inventory regarding an object to be inserted into or withdrawn from an electronic device using an artificial intelligence system.

An electronic device according to various embodiments of the present disclosure may include a plurality of sensors including a visual sensor and a weight sensor; Memory; And at least one processor, wherein the at least one processor determines whether at least one object is inserted into the electronic device using at least one of the plurality of sensors, and pulls in the at least one object. Acquiring an image of the at least one object using the visual sensor, identifying a type of the at least one object corresponding to the image using a learning model trained through an artificial intelligence algorithm, and The residual amount information of the at least one object may be obtained using a learning model and the weight sensor, and the type of the object and the residual amount information of the object may be stored in the memory.

An inventory management method of an electronic device according to various embodiments of the present disclosure may include determining whether at least one object is inserted into the electronic device using at least one of a plurality of sensors including a visual sensor and a weight sensor; Acquiring an image of the at least one object using the visual sensor; Identifying a type of the at least one object based on the acquired image by using a learning model learned through an artificial intelligence algorithm; Obtaining residual amount information of the at least one object by using the learning model learned through the artificial intelligence algorithm and the weight sensor; And storing the type and the remaining amount information of the at least one object in a memory of the electronic device.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments of the present disclosure.

2 is a block diagram illustrating an electronic device 200 according to various embodiments of the present disclosure.

3 is a diagram for describing a method of acquiring arrival information of an object with respect to the electronic device 200 according to various embodiments of the present disclosure.

FIG. 4 is a diagram for describing a method of identifying object type and remaining amount information in an

electronic device

200 or 300 according to various embodiments of the present disclosure.

5 is a diagram for describing an information receiving operation of the electronic device 200 according to an embodiment of the present disclosure.

6 is a diagram illustrating an information output screen in an electronic device according to various embodiments of the present disclosure.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments. Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or the second network 199. The electronic device 104 may communicate with the server 108 through a long range wireless communication network. According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, and a sensor module ( 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197. ) May be included. In some embodiments, at least one of the components (for example, the display device 160 or the camera module 180) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components may be implemented in one integrated circuit. For example, the sensor module 176 (eg, fingerprint sensor, iris sensor, or illuminance sensor) may be implemented embedded in the display device 160 (eg, display).

The processor 120, for example, executes software (eg, the program 140) to execute at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, processor 120 may send instructions or data received from another component (eg, sensor module 176 or communication module 190) to volatile memory 132. Can be loaded into, processed in a command or data stored in volatile memory 132, and stored in the non-volatile memory (134). According to an embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor), and a coprocessor 123 (eg, a graphics processing unit, an image signal processor) that may operate independently or together. , Sensor hub processor, or communication processor). Additionally or alternatively, the coprocessor 123 may be set to use lower power than the main processor 121 or to be specialized for its designated function. The coprocessor 123 may be implemented separately from or as part of the main processor 121.

The coprocessor 123 may, for example, replace the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 may be active (eg, execute an application). At least one of the components of the electronic device 101 (eg, the display device 160, the sensor module 176, or the communication module 190) together with the main processor 121 while in the) state. Control at least some of the functions or states associated with the. According to one embodiment, the coprocessor 123 (eg, an image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 180 or communication module 190). have.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. The data may include, for example, software (eg, the program 140) and input data or output data for a command related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

The program 140 may be stored as software in the memory 130, and may include, for example, an operating system 142, middleware 144, or an application 146.

The input device 150 may receive a command or data to be used for a component (for example, the processor 120) of the electronic device 101 from the outside (for example, a user) of the electronic device 101. The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).

The sound output device 155 may output a sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as multimedia playback or recording playback, and the receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from or as part of a speaker.

The display device 160 may visually provide information to the outside (eg, a user) of the electronic device 101. The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display device 160 may include a touch circuitry configured to sense a touch, or a sensor circuit (eg, a pressure sensor) configured to measure the strength of a force generated by the touch. have.

The audio module 170 may convert sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 may acquire sound through the input device 150, or may output an external electronic device (eg, a sound output device 155, or directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101, or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more designated protocols that may be used for the electronic device 101 to be directly or wirelessly connected to an external electronic device (for example, the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and videos. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101. According to one embodiment, the power management module 388 may be implemented, for example, as at least part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell or a fuel cell.

The communication module 190 may establish a direct (eg wired) communication channel or wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establish and perform communication over established communication channels. The communication module 190 may operate independently of the processor 120 (eg, an application processor) and include one or more communication processors supporting direct (eg, wired) or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a near field communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg It may include a local area network (LAN) communication module, or a power line communication module. The corresponding communication module of these communication modules may be a first network 198 (e.g. a short range communication network such as Bluetooth, WiFi direct or infrared data association (IrDA)) or a second network 199 (e.g. cellular network, the Internet, or Communicate with external electronic devices via a telecommunications network, such as a computer network (eg, LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented by a plurality of components (eg, a plurality of chips) separate from each other. The wireless communication module 192 uses subscriber information (e.g., international mobile subscriber identifier (IMSI)) stored in the subscriber identification module 196 in a communication network such as the first network 198 or the second network 199. The electronic device 101 may be checked and authenticated.

The antenna module 197 may transmit or receive a signal or power to an external (eg, an external electronic device) or from the outside. According to an embodiment, the antenna module may include one antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, a PCB). According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for the communication scheme used in the communication network, such as the first network 198 or the second network 199, is for example by means of the communication module 190 from the plurality of antennas. Can be selected. The signal or power may be transmitted or received between the communication module 190 and the external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, RFICs) may be further formed as part of the antenna module 197.

At least some of the components are connected to each other and connected to each other through a communication method between peripheral devices (eg, a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)). For example, commands or data).

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the

electronic devices

102 and 104 may be a device of the same or different type as the electronic device 101. According to an embodiment of the present disclosure, all or part of operations executed in the electronic device 101 may be executed in one or more external devices among the external

electronic devices

102, 104, or 108. For example, when the electronic device 101 needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service itself. In addition to or in addition, one or more external electronic devices may be requested to perform at least a part of the function or the service. The one or more external electronic devices that receive the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101. The electronic device 101 may process the result as it is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology. This can be used.

The electronic device 200 according to various embodiments may include a plurality of

sensors

212 and 214, a memory 220, a communication module 230, and at least one processor 240.

At least one processor 240 may control the plurality of

sensors

212 and 214, the memory 220, and the communication module 230 as a whole.

According to various embodiments, at least one processor 240 of the electronic device 200 may determine whether a specific object is inserted into the electronic device 200 using the plurality of

sensors

212 and 214. . For example, the at least one processor 240 may determine that the object is inserted into the electronic device 200 when the object is placed on the upper portion of the electronic device 200 or inside the electronic device 200. have.

The plurality of sensors may include a visual sensor 212 and a weight sensor 214. For example, at least one processor 240 of the electronic device 200 may identify whether an object is inserted into the electronic device 200 through an image change acquired using the visual sensor 212, or may include a weight sensor ( Based on the weight change sensed by 214, it is possible to identify whether the object is retracted.

According to various embodiments, in addition to the visual sensor 212 and the weight sensor 214, the electronic device 200 may identify whether an object is drawn through various types of sensors such as a proximity sensor, a temperature sensor, and a pressure sensor.

The visual sensor 212 may include, for example, a camera module (eg, the camera module 180) capable of capturing an environment inside and / or outside the electronic device 200. For example, the electronic device 200 may obtain an image of at least one object introduced into the electronic device 200 using the visual sensor 212. The visual sensor 212 may be disposed in plural locations at various locations in order to acquire internal and / or external images of the electronic device 200.

The weight sensor 214 may measure, for example, the weight of at least one object introduced into the electronic device 200. For example, the electronic device 200 may identify whether the object is inserted or acquire the weight information of the inserted object using the weight sensor 214.

The electronic device 200 may transmit and receive information between the electronic device 200 and the external device 260 or the electronic device 200 and the external server through the communication module 230.

For example, the external server may include a learning model 250 learned through artificial intelligence algorithms. For example, the electronic device 200 may use the learning model 250 learned through the artificial intelligence algorithm or learn the learning model 250 through the communication module 230. According to various embodiments, the artificial intelligence algorithm may include at least one of machine learning, neural networks, genes, deep learning, and classification algorithms.

The electronic device 200 according to various embodiments of the present disclosure may identify a type of a specific object introduced into the electronic device 200 using the learning model 250 learned through an artificial intelligence algorithm. For example, the electronic device 200 may determine whether an object is inserted by using at least one of the plurality of sensors, and image the incoming object by using the visual sensor 212 of the plurality of sensors. Can be obtained. The electronic device 200 may identify the type of object corresponding to the acquired image by using the learning model 250.

According to various embodiments of the present disclosure, the electronic device 200 may obtain residual information of an object inserted into the electronic device 200 by using the learning model 250. For example, the electronic device 200 may obtain the weight information of the inserted object through the weight sensor 214, and may determine the total capacity information of the object corresponding to the image (and the jar of the object). Weight information) can be obtained using the learning model 250. The electronic device 200 may determine the remaining amount information of the object based on the weight information and the total capacity information of the object.

The memory 220 may include information about the type of the object or information about the object acquired using the learning model 250 learned through artificial intelligence algorithms, or information obtained from a plurality of sensors (eg, images and weight information). At least one of the remaining amount information may be stored.

Although not shown, the electronic device 200 may include a plurality of sensors or an output module (eg, the audio output device 155) or a display device (eg, the display device 160, a display) capable of outputting information received from the outside. ), And an input module (eg, a microphone, a touch screen, etc.) capable of receiving a user input.

The electronic device 200 may communicate with the external device 260 using the communication module 230. For example, the external device 260 may include a portable communication terminal including a communication module 262, an input module 264, and an output module 266 for communicating with the electronic device 200.

The electronic device 200 according to various embodiments of the present disclosure may identify a type of an object corresponding to an image related to the inserted object by using the learning model 250 learned through an artificial intelligence algorithm. For example, when the electronic device 200 fails to identify the type of the object using the learning model 250, the electronic device 200 may output an output module (not shown) or an external device 260 of the electronic device 200. Information input regarding the type of the object corresponding to the image may be requested through at least one of the output modules 262 of FIG. Hereinafter, an operation of receiving information input from the outside regarding the type of the object will be described with reference to FIG. 5.

The electronic device 200 recognizes the ingress of the object, obtains an image of the object using the visual sensor 212, and uses the learning model 250 trained through an artificial intelligence algorithm to the image. The type of the corresponding object can be identified. However, as shown in FIG. 5, when the identification of the type of the object corresponding to the image fails using the training model 250, the electronic device 200 may externally through the output module in operation 510. The user may request to input information regarding the type of the object.

For example, when the electronic device 200 identifies the type of the object using the learning model 250, the electronic device 200 may also receive the reliability of the identification result from the learning model 250. If the reliability of the identification result is less than or equal to a specified threshold, the electronic device 200 may determine that the identification of the type of the object has failed.

According to an embodiment of the present disclosure, the electronic device 200 may determine that the inserted object is one of 'tomato sauce' or 'dry tomato' using the learning model 250, but the reliability of the identification result is specified. If it is equal to or less than the threshold value, a message may be output to request to input information on whether the kind of the object displayed in the image is 'tomato sauce' or 'dry tomato'.

The information input request may be output through, for example, a display and / or a sound output device that is at least one of output modules (not shown) of the electronic device 200. For another example, the information input request may be transmitted to an external device (eg, a portable communication terminal) preset in the electronic device 200 and output through an output module of the external device.

In operation 520, the electronic device 200 may receive user input information according to the information input request. For example, the user may input that the image related to the inserted object corresponds to the 'dry tomato' through an input module (not shown) of the electronic device 200 or an input module 262 of the external device 260. have.

The electronic device 200 may determine the type of the object corresponding to the image of the inserted object through the user input information received from the input module of the electronic device 200 or the external device 262.

In operation 530, the electronic device 200 may store, in the memory 220 of the electronic device 200, the type information of the inserted object determined through the user input information. For example, the electronic device 200 may store, in FIG. 5, the inserted object as a 'dry tomato' in the memory 220.

In operation 540, the electronic device 200 may train the learning model 250 to learn the type information of the inserted object determined by the user input information. For example, the electronic device 200 identifies the type of the object corresponding to the image based on user input information received through the input module, and trains the learning model 250 to learn the type of the object corresponding to the image. Can be. For example, the electronic device 200 or another electronic device (not shown) using the learning model 250 may identify the type of the object through an image related to the object after the learning.

Although not illustrated in FIG. 5, the electronic device 200 according to various embodiments of the present disclosure may, in the same manner as learning about the type information of the object in FIG. 5, based on user input information to determine an object corresponding to an image. The learning model 250 may be acquired by acquiring the full capacity information or the weight information of the jar of the object.

The electronic device 200 may use the learning model 250 or train the learning model 250 through the communication module 230.

According to various embodiments of the present disclosure, a plurality of electronic devices may learn a learning model learned through an artificial intelligence algorithm based on user information input through at least one electronic device, and by using the learned learning model, Information about objects introduced into or withdrawn from each of the electronic devices may be obtained to provide inventory information of each electronic device to the user.

The electronic device 200 may be inserted through an output module (not shown) (eg, the audio output device 155) or a display device (eg, the display device 160 or the display) of the electronic device 200. And at least one of a type, an image related to the object, and residual amount information of the object. According to another embodiment, the electronic device 200 transmits at least one of the information to the external device 260 using the communication module 230, and outputs it through the output module 266 of the external device 260. Command can be sent.

Referring to FIG. 3, the electronic device 200 according to various embodiments of the present disclosure may identify whether an object is placed inside an electronic device or on an upper portion of the electronic device. For example, the electronic device 200 may include at least one shelf inside or at the top, and each of the at least one shelf may include at least one weight sensor 310 and a visual sensor 320.

For example, the electronic device 200 may use the at least one weight sensor 310 and the visual sensor 320 based on the order and time at which the at least one object is drawn in or drawn out. , 300b) to obtain information about the withdrawal or withdrawal.

Reference numeral 301 illustrates a case in which the first object 300a is inserted in the electronic device 200. Reference numeral 302 is a diagram illustrating a case where a second object 300b is additionally inserted into the electronic device 200, and reference numeral 303 is an electronic device into which the first object 300a and the second object 300b are inserted. FIG. 200 illustrates a case where the first object 300a is drawn out, and reference numeral 304 illustrates a case where the first object 300a is drawn back into the electronic device 200.

For example, in reference numeral 301, the electronic device 200 may identify an incoming of the first object 300a using at least one of the plurality of sensors. For example, the electronic device 200 may use the weight sensor 310 to identify a change in the weight of the shelf on which the first object 300a is placed, and thereby recognize the entry of the first object 300a. The electronic device 200 may obtain weight information of the first object 300a based on the change in the weight of the shelf. In addition, the electronic device 200 may acquire an image of the first object 300a using the visual sensor 320.

In operation 302, when the second object 300b is inserted into the electronic device 200, the electronic device 200 recognizes that the weight change is caused by the second object 300b based on the weight change of the shelf. The weight information of the second object 300b may be obtained. In addition, the electronic device 200 may acquire an image of the second object 300b using the visual sensor 320.

In reference numeral 303, when any one of the plurality of objects is drawn, the electronic device 200 determines which object is drawn through the visual sensor 320, or determines the plurality of weight sensors 310. This allows you to determine which object was fetched.

As shown in FIG. 3, the electronic device 200 may include a plurality of weight sensors 310 on each shelf of the electronic device 200. For example, the electronic device 200 may identify an object corresponding to a position on the shelf through the plurality of weight sensors 310.

In reference numeral 304, when the first object 300a is retracted, the electronic device 200 recognizes that the weight change due to the retraction of the first object 300a is a change in weight by the first object 300a. In order to do this, the input or withdrawal time information of the electronic device 200 may be used, the visual sensor 320 may be used, or the plurality of weight sensors 310 may be used. For example, when the weight information of the retracted first object 300a is different from the weight information of the first object 300a obtained in reference numeral 301, the electronic device 200 may store the changed weight information in the memory. .

electronic device

200 or 300 according to various embodiments of the present disclosure.

According to various embodiments of the present disclosure, the

electronic devices

200 and 300 may identify an incoming of the object 250 and may acquire the image of the object 250 by using a visual sensor to determine information of the object. For example, when the object 250 is inserted, the electronic device may acquire weight information of the object using a weight sensor, but the type of the object 250 may be obtained through the memory 220 of the

electronic device

200 or 300. May not be identified.

According to various embodiments of the present disclosure, an electronic device may use information about an object (eg, type of object and total weight of an object) by using a learning model 255 learned through an artificial intelligence algorithm through an image acquired using a visual sensor. , Container weight of the object, etc.).

For example, the electronic device may transmit an image of the object 250 to an external server and obtain image identification result information (information of the object) obtained by using the learning model 255.

For example, as shown in FIG. 4, the electronic device is a result of the identification of the image, and the kind of the inserted object 250 is 'salmon flavored milk', and the total weight of the object 250 is 1030 g. The weight of 30g can be recognized using the learning model 255. The electronic device may store the information of the object 250 in the memory 220 of the electronic device, and identify the remaining amount of the object 250 based on the information of the object 250. For example, the remaining amount of the object 250 from the weight excluding 30g corresponding to the container weight among the total weight 1030g of the object 250 and the weight excluding 30g which is the weight of the container from the weight 630g of the inserted object 250, It can be identified that 60%. Through this, the electronic device may recognize that the inserted object 250 corresponds to 60% of remaining milk and 600g of salmon flavor milk.

6 is a diagram illustrating an information output method in an electronic device according to various embodiments of the present disclosure.

According to various embodiments of the present disclosure, the electronic device may identify information about an object introduced into the electronic device or an object drawn out from the electronic device, and output the information to the user through an output module.

For example, the electronic device is an output module of the electronic device, and the object drawn or drawn out through at least one of an audio output device (eg, the audio output device 155) or a display device (eg, the display device 160). You can output information about. For another example, the electronic device may transmit the information to an external device so that the information is output through an output module of the external device.

For example, as shown in reference numeral 610, the electronic device may display information about an object inserted into the electronic device on a display screen.

For example, the electronic device may display an image 612 about an object inserted into the electronic device. In addition, the type, remaining amount information, and weight information of each object corresponding to the image may be displayed as shown in reference numeral 614.

According to various embodiments of the present disclosure, the electronic device may display an item list of at least one object inserted into the electronic device as shown in reference numeral 616. For example, the user may easily grasp the object item inserted into the electronic device through the display screen 616. The item list is information on at least one object to be inserted and may include at least one of a kind of object, remaining amount information, and weight information.

According to another embodiment, the electronic device may acquire analysis information about the at least one inserted object using a learning model learned through an artificial intelligence algorithm and display it in the form of a chart as shown in reference numeral 618. . For example, when at least one object introduced into the electronic device is food, the analysis information may be nutritional information about at least one object introduced.

According to various embodiments of the present disclosure, the electronic device may acquire analysis information about at least one inserted object, using a learning model learned through an artificial intelligence algorithm, and obtain the acquired analysis information to a preset external device. Can transmit

For example, the analysis information may be obtained by using a learning model trained through the artificial intelligence algorithm, based on the type and residual information of the at least one object inserted into the electronic device (eg : Shopping suggestion list 630) or utilization suggestion list (eg, diet suggestion list 630).

For example, as illustrated in reference numeral 620, the electronic device may generate shopping suggestion list information for the object when the remaining amount is less than or equal to a preset value, based on at least one of the type, remaining amount, and nutritional information of the inserted object. The shopping suggestion list information may be transmitted to an external device and displayed on the display module of the external device.

For another example, as shown in FIG. 630, the electronic device may generate a list of dietary suggestions based on at least one of the type of object, remaining amount, and nutritional information, and transmits the information to an external device. It can be displayed through the display module of the external device.

Electronic devices according to various embodiments of the present disclosure may be various types of devices. The electronic device may include, for example, a portable communication device (eg, a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. Electronic devices according to embodiments of the present disclosure are not limited to the above-described devices.

Various embodiments of the present document and terminology used herein are not intended to limit the technical features described in the present specification to specific embodiments, but should be understood to include various changes, equivalents, or substitutes for the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the items, unless the context clearly indicates otherwise. In this document, "A or B", "at least one of A and B", "or at least one of B," "A, B or C," "at least one of A, B and C," and "B, Or phrases such as "at least one of C" may include any one of the items listed together in the corresponding one of the phrases, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used merely to distinguish a component from other corresponding components, and to separate the components from other aspects (e.g. Order). Some (eg, first) component may be referred to as "coupled" or "connected" to another (eg, second) component, with or without the term "functionally" or "communically". When mentioned, it means that any component can be connected directly to the other component (eg, by wire), wirelessly, or via a third component.

As used herein, the term “module” may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. The module may be an integral part or a minimum unit or part of the component, which performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document may include one or more instructions stored on a storage medium (eg, internal memory 136 or external memory 138) that can be read by a machine (eg, electronic device 101). It may be implemented as software (eg, program 140) including the. For example, the processor (eg, the processor 120) of the device (eg, the electronic device 101) may call and execute at least one command among one or more instructions stored from the storage medium. This enables the device to be operated to perform at least one function in accordance with the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means only that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), which is the same as the case where data is stored semi-permanently on the storage medium. It does not distinguish cases where it is temporarily stored.

According to an embodiment, a method according to various embodiments disclosed in the present disclosure may be included in a computer program product. The computer program product may be traded between the seller and the buyer as a product. The computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play StoreTM) or two user devices ( Example: smartphones) can be distributed (eg downloaded or uploaded) directly or online. In the case of on-line distribution, at least a portion of the computer program product may be stored at least temporarily on a device-readable storage medium such as a server of a manufacturer, a server of an application store, or a relay server, or may be temporarily created.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular or plural entity. According to various embodiments, one or more of the aforementioned components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of the component of each of the plurality of components the same as or similar to that performed by the corresponding component of the plurality of components before the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Or one or more other actions may be added.

Claims

In an electronic device,

A plurality of sensors including a visual sensor and a weight sensor;

Memory; And

At least one processor,

The at least one processor,

Determining whether at least one object is inserted into the electronic device using at least one of the plurality of sensors,

Based on the ingress of the at least one object,

Acquires an image of the at least one object using the visual sensor,

Identify a type of the at least one object corresponding to the image by using a learning model trained through an artificial intelligence algorithm,

Obtaining residual amount information of the at least one object using the learning model and the weight sensor,

And store the type of the object and the remaining amount information of the object in the memory.
The method of claim 1,

Output module; And

Further includes an input module,

The at least one processor,

If it fails to identify the type of the object using the learning model, and outputs a request for inputting information about the type of the object through the output module,

Receive information on the type of the object through the input module,

The electronic device is configured to train the learning model using the image of the object and the received information.
The method of claim 2,

Further comprising a communication module,

The at least one processor,

The electronic device is configured to use the learning model learned through the artificial intelligence algorithm or to train the learning model through the communication module.
The method of claim 1,

The at least one processor,

An artificial intelligence algorithm, the electronic device configured to identify the type of the object corresponding to the image using a learning model trained using at least one of machine learning, neural network, gene, deep learning, and classification algorithm.
The method of claim 1,

Further includes a display,

The at least one processor,

And output at least one of the type information of the object, the image of the object, or the remaining amount information of the object by using the display.
The method of claim 1,

Further comprising a communication module,

The at least one processor,

The electronic device is configured to transmit at least one of the type information of the object introduced into the electronic device, the image of the object, or the remaining amount information of the object to the external device using the communication module.
The method of claim 6,

The at least one processor,

If it fails to identify the type of the object using the learning model, by using the communication module, receives information about the type of the object from the external device,

The electronic device is configured to train the learning model using the image of the object and the received information.
The method of claim 6,

The at least one processor,

By using the learning model, based on the type of the object and the remaining amount information of the object, it is determined whether the additional drawing of the object is necessary,

And transmit a user notification to the external device based on the determination result.
The method of claim 6,

The at least one processor,

Acquires nutrition information of at least one object introduced into the electronic device using the learning model,

The electronic device is configured to transmit the obtained nutrition information to the external device.
The method of claim 1,

The at least one processor,

Determine information about an entry time of the object and store the information in the memory;

Using the learning model, based on at least one of the type of the object, information on the remaining amount of the object, and information on the inlet time of the object, it is determined whether the withdrawal of the object or additional induction of the object is necessary;

The electronic device is configured to output a user notification to the outside based on the determination result.
In the inventory tracking method of the electronic device,

Determining whether at least one object is inserted into the electronic device using at least one of a plurality of sensors including a visual sensor and a weight sensor;

Acquiring an image of the at least one object using the visual sensor;

Identifying a type of the at least one object based on the acquired image by using a learning model learned through an artificial intelligence algorithm;

Obtaining residual amount information of the at least one object by using the learning model learned through the artificial intelligence algorithm and the weight sensor; And

And storing the type and the remaining amount information of the at least one object in a memory of the electronic device.
The method of claim 11,

Outputting a request for inputting information regarding the type of the at least one object when the type identification of the at least one object fails using the learning model learned through the artificial intelligence algorithm;

Receiving information input regarding a type of the object; And

And training the learning model by using the received information and the at least one image.
The method of claim 11,

The artificial intelligence algorithm includes at least one of machine learning, neural networks, genes, deep learning, and classification algorithms.
The method of claim 11,

Displaying at least one of information of the object and remaining amount information of the object on a display of the electronic device; And

And transmitting at least one of the image of the object, the information of the object, or the remaining amount information of the object to an external device.
In the inventory tracking method of the electronic device,

Determining whether at least one object is inserted into the electronic device using at least one of a plurality of sensors including a visual sensor and a weight sensor;

Acquiring an image of the at least one object using the visual sensor;

Acquiring weight information of the at least one object using the weight sensor;

Transmitting at least one of the image and weight information to an external server;

Receiving, from the external server, information about the type of the at least one object and the remaining amount of the at least one object, which are identified based on at least one of the image and the weight information, using a learning model learned through an artificial intelligence algorithm. action;

And storing the type and the remaining amount information of the at least one object in a memory of the electronic device.