WO2024262776A1 - Refrigerator and method for controlling same - Google Patents

Abstract

A refrigerator (1) according to one embodiment comprises: a chamber (11); a plurality of trays (20) stored in different tiers in the chamber (11); a camera (100) which is provided at the top of the chamber (11), and which acquires an image of the area in which the plurality of trays (20) move; a control unit (200) for identifying the movement of at least one of the plurality of trays (20) on the basis of the image acquired by the camera (100); and a communication circuit communicating with a user equipment (2), wherein the control unit (200) can acquire, on the basis that one of the plurality of trays (20) is discharged, a first capture image including a discharged tray (20), acquire, on the basis that the object is arranged on the discharged tray (20), a second capture image including an object, detect the position of the object and an identifier of the object on the basis of the difference between the first capture image and the second capture image, and provide, to the user equipment (2), position information about the object and detailed information about the object acquired from the identifier through the communication circuit.

Description

Refrigerator and method of controlling it

The disclosed invention relates to a refrigerator and a method of controlling the same.

Recently, with the development of the Internet of Things and big data, research and development on smart home appliances are being conducted. Smart home appliances refer to home appliances that are directly or indirectly connected to the Internet and can be operated and managed using smartphones or applications.

Since a large number of objects are stored in refrigerators among home appliances, technologies that can effectively manage information about stored objects are emerging. For example, by using a camera located in a refrigerator to identify stored objects and convert information related to the location of the identified objects into data, the stored objects can be managed in their entirety.

However, in order to recognize all objects stored in multiple storage rooms in a refrigerator, a camera is required for each storage room, which increases the cost and requires a lot of power.

The disclosed invention provides a refrigerator and a control method thereof for obtaining location and detailed information of an object stored in the refrigerator using a single camera and transmitting the information to a user device.

A refrigerator (1) according to one embodiment comprises: a chamber (11); a plurality of trays (20) stored in different layers within the chamber (11); a camera (100) provided on an upper portion of the chamber (11) and obtaining an image of an area in which the plurality of trays (20) move; a control unit (200) identifying movement of at least one of the plurality of trays (20) based on the image obtained by the camera (100); And a communication circuit for communicating with a user device (2); wherein the control unit (200) may obtain a first capture image including the taken-out tray (20) based on identifying that one of the plurality of trays (20) is taken out, obtain a second capture image including the object based on identifying that an object is placed on the taken-out tray (20), detect a position of the object and an identifier of the object based on a difference between the first capture image and the second capture image, and provide the position information of the object and detailed information of the object obtained from the identifier to the user device (2) through the communication circuit.

A method for controlling a refrigerator (1) according to one embodiment of the present invention comprises: a chamber (11); a plurality of trays (20) stored in different layers within the chamber (11); a camera (100) provided on an upper portion of the chamber (11) and obtaining an image; and a communication circuit communicating with a user device (2); The method may include obtaining an image of an area in which the plurality of trays (20) move by the camera (100); obtaining a first captured image including the removed tray (20) based on identifying that one of the plurality of trays (20) is being removed; obtaining a second captured image including the object based on identifying that an object is placed on the removed tray (20); detecting a location of the object and an identifier of the object based on a difference between the first captured image and the second captured image; and providing detailed information of the object obtained from the location information of the object and the identifier to the user device (2) through the communication circuit.

According to the disclosed invention, the location and detailed information of an object stored in a refrigerator can be identified using a single camera, and the refrigerator can be managed more effectively by providing the information to the user through a user device, thereby enhancing user convenience. Furthermore, since only a single camera is used, less power is required, and the product cost can also be reduced.

The technical tasks and effects to be achieved in this document are not limited to those described above, and other technical tasks and effects other than those mentioned can be clearly understood by those skilled in the art to which the present invention pertains from the description below.

FIG. 1 illustrates a network system implemented by a refrigerator according to one embodiment.

Figure 2 illustrates the exterior of a refrigerator according to one embodiment.

FIG. 3 illustrates a tray on which objects are placed in a refrigerator according to one embodiment.

Figure 4 is a control block diagram of a refrigerator according to one embodiment.

Figure 5 is a flowchart illustrating a method for controlling a refrigerator according to one embodiment.

FIG. 6 illustrates a state in which one of a plurality of trays of a refrigerator according to one embodiment is taken out.

FIG. 7 illustrates a first captured image acquired by a camera of a refrigerator according to one embodiment.

FIG. 8 illustrates a reference image pre-stored in the memory of a refrigerator according to one embodiment.

FIG. 9 illustrates a second captured image capable of detecting an identifier of an object contained in a removed tray of a refrigerator according to one embodiment.

FIG. 10 illustrates a second captured image in which an identifier of an object contained in a removed tray of a refrigerator according to one embodiment of the present invention cannot be detected.

Figure 11 illustrates a screen displayed on a user device according to one embodiment.

Figure 12 illustrates a screen displayed on a user device according to one embodiment.

Fig. 13 is a flowchart showing a method for controlling a refrigerator according to one embodiment.

It should be understood that the various embodiments of the present disclosure and the terminology used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, but rather to encompass various modifications, equivalents, or substitutes of the embodiments.

In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

The singular form of a noun corresponding to an item may include one or more of said items, unless the context clearly indicates otherwise.

In this disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

The term "and/or" includes any combination of multiple related described elements or any one of multiple related described elements.

Terms such as "first", "second", or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order).

In addition, the terms 'front', 'rear', 'top', 'bottom', 'side', 'left', 'right', 'upper', 'lower', etc. used in the present disclosure are defined based on the drawings, and the shape and position of each component are not limited by these terms.

The terms "include" or "have" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the present disclosure, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

When a component is said to be “connected,” “coupled,” “supported,” or “contacted” with another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

When we say that a component is "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

Below, a refrigerator (1) according to various embodiments will be specifically described with reference to the attached drawings.

FIG. 1 illustrates a network system implemented by a refrigerator (1) according to one embodiment.

A refrigerator (1) may include a communication module capable of communicating with other home appliances, user devices (2), or servers (3), a user interface (400) that receives user input or outputs information to a user, at least one processor (210) that controls the operation of the refrigerator (1), and at least one memory (220) that stores a program for controlling the operation of the refrigerator (1).

The other home appliance may be at least one of various types of home appliances. For example, the other home appliance may include, but is not limited to, at least one of a refrigerator (1), a dishwasher, an electric range, an electric oven, an air conditioner, a clothes manager, a washing machine, a dryer, and a microwave oven, and may include various types of home appliances, such as a cleaning robot, a vacuum cleaner, and a television, which are not shown in the drawing. In addition, the home appliances mentioned above are only examples.

The server (3) may include a communication module capable of communicating with another server, a refrigerator (1), or a user device (2), at least one processor (210) capable of processing data received from another server, a refrigerator (1), or a user device (2), and at least one memory (220) capable of storing a program for processing data or processed data. The server (3) may be implemented as various computing devices such as a workstation, a cloud, a data drive, a data station, etc. The server (3) may be implemented as one or more servers (3) physically or logically separated based on function, detailed configuration of function, or data, and may transmit and receive data and process the transmitted and received data through communication between each server (3).

The server (3) can perform functions such as managing user accounts, registering a refrigerator (1) by linking it to a user account, and managing or controlling the registered refrigerator (1). For example, a user can access the server (3) through a user device (2) and create a user account. The user account can be identified by an ID and password set by the user. The server (3) can register the refrigerator (1) to the user account according to a set procedure. For example, the server (3) can link identification information (e.g., serial number or MAC address, etc.) of the refrigerator (1) to the user account, thereby registering, managing, and controlling the refrigerator (1).

The user device (2) may include a communication module capable of communicating with a refrigerator (1) or a server (3), an interface for receiving user input or outputting information to a user, at least one processor for controlling the operation of the user device (2), and at least one memory storing a program for controlling the operation of the user device (2).

The user device (2) may be carried by the user or placed in the user's home or office, etc. The user device (2) may include, but is not limited to, a personal computer, a terminal, a portable telephone, a smart phone, a handheld device, a wearable device, etc.

A program for controlling the refrigerator (1), i.e., an application, may be stored in the memory of the user device (2). The application may be sold installed in the user device (2) or downloaded and installed from an external server (3).

A user can access a server (3) by executing an application installed on a user device (2), create a user account, and communicate with the server (3) based on the logged-in user account to register a refrigerator (1).

For example, when the refrigerator (1) is operated so that the refrigerator (1) can be connected to the server (3) according to the procedure guided by the application installed on the user device (2), the refrigerator (1) can be registered to the user account by registering the identification information (e.g., serial number or MAC address) of the refrigerator (1) to the corresponding user account on the server (3).

The user can control the refrigerator (1) using the application installed on the user device (2). For example, when the user logs into the user account using the application installed on the user device (2), the refrigerator (1) registered to the user account appears, and when a control command for the refrigerator (1) is input, the control command can be transmitted to the refrigerator (1) via the server (3).

A network can include both wired and wireless networks. Wired networks include cable networks or telephone networks, and wireless networks include any network that transmits and receives signals via radio waves. Wired and wireless networks can be connected to each other.

A network may include a wide area network (WAN) such as the Internet, a local area network (LAN) formed around an Access Point (AP), and a short-range wireless network that does not use an Access Point (AP). Short-range wireless networks may include, but are not limited to, Bluetooth (IEEE 802.15.1), Zigbee (IEEE 802.15.4), Wi-Fi Direct, Near Field Communication (NFC), Z-Wave, etc.

An access point (AP) can connect a refrigerator (1) or a user device (2) to a wide area network (WAN) to which a server (3) is connected. The refrigerator (1) or the user device (2) can be connected to the server (3) via the wide area network (WAN).

The access point (AP) can communicate with the refrigerator (1) or user device (2) using wireless communication such as Wi-Fi (IEEE 802.11), Bluetooth (IEEE 802.15.1), or Zigbee (IEEE 802.15.4), and can connect to a wide area network (WAN) using wired communication, but is not limited thereto.

According to various embodiments, the refrigerator (1) may be directly connected to a user device (2) or a server (3) without going through an access point (AP).

The refrigerator (1) can be connected to a user device (2) or a server (3) via a long-distance wireless network or a short-distance wireless network.

For example, a refrigerator (1) may be connected to a user device (2) via a short-range wireless network (e.g., Wi-Fi Direct).

As another example, the refrigerator (1) may be connected to a user device (2) or a server (3) via a wide area network (WAN) using a long-distance wireless network (e.g., a cellular communication module).

As another example, a refrigerator (1) may be connected to a wide area network (WAN) using wired communication and may be connected to a user device (2) or a server (3) through the wide area network (WAN).

If the refrigerator (1) can connect to a wide area network (WAN) using wired communication, it can also act as a connection relay. Accordingly, the refrigerator (1) can connect other home appliances to the wide area network (WAN) to which the server (3) is connected. In addition, other home appliances can connect the refrigerator (1) to the wide area network (WAN) to which the server (3) is connected.

A refrigerator (1) can transmit information about its operation or status to another home appliance, a user device (2), or a server (3) through a network. For example, the refrigerator (1) can transmit information about its operation or status to another home appliance, a user device (2), or a server (3) when a request is received from a server (3), when a specific event occurs in the refrigerator (1), or periodically or in real time. When information about its operation or status is received from the refrigerator (1), the server (3) can update the information about the operation or status of the refrigerator (1) that has been stored, and transmit the updated information about the operation and status of the refrigerator (1) to the user device (2) through a network. Here, updating information can include various operations in which existing information is changed, such as an operation of adding new information to existing information, an operation of replacing existing information with new information, etc.

The refrigerator (1) can obtain various information from other home appliances, user devices (2), or servers (3), and provide the obtained information to the user.

The refrigerator (1) can operate according to a control command received from another home appliance, a user device (2), or a server (3). For example, if the refrigerator (1) obtains prior approval from a user so that it can operate according to a control command from the server (3) even without a user input, the refrigerator (1) can operate according to a control command received from the server (3). Here, the control command received from the server (3) may include, but is not limited to, a control command input by the user through the user device (2) or a control command based on a preset condition.

The user device (2) can transmit information about the user to the refrigerator (1) or the server (3) through the communication module. For example, the user device (2) can transmit information about the user's location, the user's health status, the user's preferences, the user's schedule, etc. to the server (3). The user device (2) can transmit information about the user to the server (3) with the user's prior consent.

The refrigerator (1), the user device (2), or the server (3) can determine a control command using technology such as artificial intelligence. For example, the server (3) can receive information about the operation or status of the refrigerator (1) or information about the user of the user device (2), process the information using technology such as artificial intelligence, and transmit the processing result or control command to the refrigerator (1) or the user device (2) based on the processing result.

Fig. 2 shows the appearance of a refrigerator (1) according to one embodiment.

A refrigerator (1) according to one embodiment may include a main body.

The "body" may include an inner case, an outer case arranged on the outside of the inner case, and an insulating material provided between the inner case and the outer case.

The "inner case" may include at least one of a case, a plate, a panel, or a liner forming a chamber (11). The inner case may be formed as a single body, or may be formed by assembling a plurality of plates. The "outer case" may form the outer appearance of the main body, and may be joined to the outer side of the inner case so that an insulating material is placed between the inner case and the outer case.

The "insulation" can insulate the inside of the chamber (11) and the outside of the chamber (11) so that the temperature inside the chamber (11) can be maintained at a set appropriate temperature without being affected by the environment outside the chamber (11). According to one embodiment, the insulation can include a foam insulation. The foam insulation can be formed by injecting and foaming urethane foam, in which polyurethane and a foaming agent are mixed, between the inner and outer layers.

In one embodiment, the insulation may additionally include a vacuum insulation in addition to the foam insulation, or the insulation may consist solely of the vacuum insulation instead of the foam insulation. The vacuum insulation may include a core and an outer shell that accommodates the core and seals the interior at a vacuum or near-vacuum pressure. However, the insulation is not limited to the foam insulation or vacuum insulation described above, and may include various materials that can be used for insulation.

The "chamber (11)" may include a space defined by an inner wall. The chamber (11) may further include an inner wall defining a space corresponding to the chamber (11). Various items such as food, medicine, and cosmetics may be stored in the chamber (11), and the chamber (11) may be formed so that at least one side is open for putting items in and taking them out.

A refrigerator (1) may include one or more chambers (11). When two or more chambers (11) are formed in the refrigerator (1), each chamber (11) may have a different purpose and may be maintained at a different temperature. To this end, each chamber (11) may be partitioned from each other by a partition wall including an insulating material.

The chamber (11) may be referred to as a "storage room". The storage room may be provided to be maintained at an appropriate temperature range depending on the intended use, and may include a "refrigerator", a "freezer" or a "variable temperature room" which are distinguished depending on the intended use and/or temperature range. The refrigerator may be maintained at an appropriate temperature for refrigerating an item, and the freezer may be maintained at an appropriate temperature for freezing an item. "Refrigeration" may mean cooling an item to a temperature that does not freeze, and for example, a refrigerator may be maintained in a range of 0 degrees Celsius to +7 degrees Celsius. "Freezing" may mean cooling an item to be frozen or kept in a frozen state, and for example, a freezer may be maintained in a range of -20 degrees Celsius to -1 degree Celsius. The variable temperature room may be used as either a refrigerator or a freezer, at the user's choice or not.

In addition to being called by names such as "refrigerator", "freezer", and "variable temperature room", a storage room may also be called by various names such as "vegetable room", "freshness room", "cooling room", and "ice room". The terms "refrigerator", "freezer", and "variable temperature room" used hereinafter should be understood to encompass storage rooms having corresponding uses and temperature ranges.

The chamber (11) may include a "tray (20)". Here, the tray (20) may be plural. Referring to FIG. 2, a plurality of trays (20) may be stored in different layers within the chamber (11). That is, a plurality of trays (20) may form a plurality of layers spaced apart at a constant distance in the vertical direction.

For example, as shown in Fig. 2, six trays (20) can be stored in the chamber (11) of the refrigerator (1). The six trays can include a first-layer tray (20-1), a second-layer tray (20-2), a third-layer tray (20-3), a fourth-layer tray (20-4), a fifth-layer tray (20-5), and a sixth-layer tray (20-6).

FIG. 3 illustrates a tray (20) on which an object is placed in a refrigerator (1) according to one embodiment.

Referring to FIG. 3, the trays (20) may be arranged so that at least one object can be placed on the upper surface of each layer. Each tray (20) may include a plurality of tray columns (21) spaced apart at a certain distance so that an object can be placed.

For example, as shown in Fig. 3, a total of four spaces where objects can be placed can be provided according to the tray pillar (21).

The tray (20) may be provided so that the user can move it to place an object. For example, the tray (20) may be provided in a slide manner so that it is removed by sliding toward the front.

Accordingly, the user can move the tray (20) toward the user's body before placing an object on it to better position the tray (20) for storing the object.

In addition to the method described above, the method of moving the tray (20) can be any method that allows the user to move the tray (20) in a state where an object can be placed.

According to one embodiment, the refrigerator (1) may include at least one door (10) configured to open and close an open side of a chamber (11). The door (10) may be provided to open and close each of one or more chambers (11), or one door (10) may be provided to open and close a plurality of chambers (11). The door (10) may be installed on the front of the main body in a rotatably or slidably manner.

The "door (10)" may be configured to seal the storage compartment when the door (10) is closed. The door (10) may include insulating material, like the main body, to insulate the storage compartment when the door (10) is closed.

According to one embodiment, the door (10) may include a door (10) outer panel forming the front of the door (10), a door (10) inner panel forming the rear of the door (10) and facing the storage compartment, an upper cap, a lower cap, and a door (10) insulation provided inside these.

A gasket may be provided on the edge of the inner panel of the door (10) to seal the storage compartment by coming into close contact with the front of the main body when the door (10) is closed. The inner panel of the door (10) may include a dyke that protrudes rearward to allow a door (10) basket for storing items to be mounted.

According to one embodiment, the door (10) may include a door (10) body and a front panel that is detachably coupled to the front side of the door (10) body and forms the front of the door (10). The door (10) body may include a door (10) outer plate that forms the front of the door (10) body, a door (10) inner plate that forms the rear of the door (10) body and faces the chamber (11), an upper cap, a lower cap, and a door (10) insulation material provided inside these.

The refrigerator (1) can be classified into a French door (10) type, a side-by-side type, a BMF (Bottom Mounted Freezer), a TMF (Top Mounted Freezer), or a 1-door (10) refrigerator (1) depending on the arrangement of the door (10) and chamber (11).

According to one embodiment, the refrigerator (1) may include a cold air supply device arranged to supply cold air to the chamber (11).

A “cold air supply device” may include a machine, mechanism, electronic device and/or a system combining these that can generate cold air and guide the cold air to cool the chamber (11).

According to one embodiment, the cold air supply device can generate cold air through a refrigeration cycle including compression, condensation, expansion, and evaporation processes of a refrigerant. To this end, the cold air supply device can include a refrigeration cycle device having a compressor, a condenser, an expansion device, and an evaporator capable of driving the refrigeration cycle. According to one embodiment, the cold air supply device can include a semiconductor such as a thermoelectric element. The thermoelectric element can cool the chamber (11) through heat generation and cooling through the Peltier effect.

According to one embodiment, the refrigerator (1) may include a machine room in which at least some components belonging to the cold air supply device are arranged.

The "machine room" may be provided with a chamber (11) and partitioned and insulated to prevent heat generated from components placed in the machine room from being transferred to the chamber (11). The interior of the machine room may be configured to be in communication with the exterior of the main body to dissipate heat from components placed inside the machine room.

Figure 4 is a control block diagram of a refrigerator (1) according to one embodiment.

Referring to FIG. 4, the refrigerator (1) may include a camera (100) for obtaining images of a plurality of trays (20), a control unit (200) for controlling the refrigerator (1), a communication unit (300) including a communication circuit for communicating with the outside, and a user interface (400) for receiving user input or outputting information to the user.

Additionally, the control unit (200) can be electrically connected to the camera (100) and the communication unit (300) and control each component.

The camera (100) can obtain images of objects stored inside the chamber (11).

In the case of a chamber (11) including a plurality of trays (20) as illustrated in FIG. 2, a plurality of cameras (100) that acquire images for each tray (20) may be included. However, the present invention will describe a method of controlling a refrigerator (1) based on acquiring images for a plurality of trays (20) with one camera (100).

Since the present invention uses only one camera (100), it requires less power and has the effect of lowering the cost of the product.

According to one embodiment, a camera (100) is provided on the upper part of the chamber (11) and can obtain an image of an area where a plurality of trays (20) are removed to the front of the chamber (11).

That is, the camera (100) may be installed above the uppermost tray (20) among the plurality of trays (20), and the shooting direction may be arranged to obtain images of the plurality of trays (20) in a vertically downward direction.

For example, the camera (100) may be attached to the upper surface of the inner case as shown in Fig. 2 and may be provided to face vertically downward. The position and shape of the camera (100) are not limited thereto, and any camera capable of obtaining an image of the area where the tray (20) is removed to the front of the chamber (11) may be used.

This can be understood as a physical arrangement for obtaining images of all of the multiple trays (20) forming multiple vertical layers with one camera (100).

In order for the camera (100) to obtain an image of the tray (20), the user must move the position of the tray (20) so that the tray (20) is positioned within the field of view of the camera (100). To this end, as described above, the tray (20) may be configured to be opened to the front through a slide method.

Fig. 5 is a flowchart showing a control method of a refrigerator (1) according to one embodiment.

The control unit (200) can analyze the image acquired by the camera (100). That is, the control unit (200) can identify the movement of at least one of the plurality of trays (20) by performing a predetermined image processing on the image acquired by the camera (100). Here, the movement of the tray (20) can include taking out or storing the tray (20).

Additionally, the control unit (200) can determine information about an object stored in the tray (20) based on an image acquired by the camera (100).

FIG. 6 illustrates a state in which one of a plurality of trays (20) of a refrigerator (1) according to one embodiment is taken out.

Specifically, the control unit (200) can identify whether one of the plurality of trays (20) is to be removed based on the image acquired by the camera (100) (510).

The control unit (200) can identify that the tray (20) has been removed if the image acquired in real time includes an image of the tray (20).

Referring to FIG. 6, since the area captured by the camera (100) is an area including a space in which multiple trays (20) are pulled out to the front, when the trays (20) are pulled out, the trays (20) are included in the image acquired by the camera (100).

Meanwhile, the control unit (200) can control the camera (100) to start acquiring images based on the door (10) being removed. To this end, the door (10) can include a separate sensor that can identify whether the door (10) is opened or closed.

FIG. 7 illustrates a first captured image (C1) acquired by a camera (100) of a refrigerator (1) according to one embodiment.

Referring to FIG. 6 and FIG. 7, when the control unit (200) identifies that the tray (20) has been removed, it can obtain a first captured image (C1) of the removed tray (20) (520). Since the first captured image (C1) is captured immediately after the tray (20) has been removed, it means an image of the previous arrangement state of the object placed on the removed tray (20).

That is, the first captured image (C1) may mean that it was acquired by the camera (100) at the current point in time when the tray (20) was taken out.

Additionally, the control unit (200) may obtain a first capture image (C1) at a point in time that includes an identifier of an object that has been previously placed based on identifying that one of the plurality of trays (20) is being removed.

The control unit (200) needs to detect the identifier of an object that has been placed previously when an object that has been placed previously is removed or a new object is placed.

Therefore, the control unit (200) needs to obtain the first capture image (C1) at a point in time when the object identifier can be detected.

In other words, the control unit (200) can obtain the first capture image (C1) at a point in time when the object identifier can be detected before the tray (20) is fully opened. In order for the object identifier to be detected, the object identifier must first be included in the first capture image (C1).

Since the shooting range of the camera (100) varies depending on the angle of view of the camera (100), if the object identifier is outside the shooting range of the camera (100), the object identifier may not be detected.

This situation can occur mainly when the layer of the exported tray (20) is high and the distance from the camera (100) is close. For example, when the topmost tray (20) is exported, the image acquired by the camera (100) may not include the object identifier portion because a sufficient angle of view is not secured depending on the performance of the camera (100).

In addition, the control unit (200) may acquire multiple capture images at various points in time at regular intervals until the tray (20) completes removal, and select a capture image that includes an object identifier among them as the first capture image (C1).

Meanwhile, if the same tray (20) has already been removed, the first captured image (C1) may be provided from the memory (220) that stores the image (second captured image (C2) described below) acquired by the camera (100) after the object was placed at the most recent time the tray (20) was removed. In this case, the control unit (200) does not need to perform a process of acquiring a separate captured image.

FIG. 8 illustrates a reference image (R) pre-stored in the memory (220) of a refrigerator (1) according to one embodiment.

The control unit (200) can determine the layer of the exported tray (20) based on the acquired first captured image (530). Specifically, the layer of the exported tray (20) can be determined by comparing the first captured image with the reference image (R) of each of the plurality of trays (20).

Here, the reference image (R) of each of the plurality of trays (20) may be pre-stored in the memory (220). The reference image (R) of each of the plurality of trays (20) may include each tray (20) on which no object is placed.

Referring to FIG. 8, the control unit (200) can compare the size of the tray (20) included in the first captured image with the size of each tray (20) included in the reference image (R) of each of the plurality of trays (20), thereby identifying a reference image (R) including a tray (20) having the same size.

The smaller the distance (Z) from the camera (100) of a tray, the larger the size of the tray (20) included in the image. The larger the distance (Z) from the camera (100) of a tray, the smaller the size of the tray (20) included in the image. Therefore, the control unit (200) can determine which tray (20) has been removed by comparing the sizes of the trays (20) included in the image.

The size of the tray (20) can be compared using at least one of the horizontal length (X), vertical length (Y), and occupied area.

Accordingly, the control unit (200) can finally determine the layer of the exported tray (20) by determining that the tray (20) corresponding to the reference image (R) including the tray (20) having the same size as the tray (20) included in the first captured image (C1) has been exported.

The control unit (200) can store in advance the size information of each tray (20) included in the reference image (R) of each of the plurality of trays (20) in the memory (220). The size information of the tray (20) included in the reference image can include the horizontal length and vertical length of the tray (20). Here, the horizontal length (X) and vertical length (Y) can vary depending on the performance of the camera (100).

The control unit (200) can determine the layer of the exported tray (20) by comparing the size information of each tray (20) stored in advance with the size of the tray (20) included in the first captured image (R).

The control unit (200) can identify that an object is placed on the exported tray (20) based on the image acquired by the camera (100) (540). That is, the control unit (200) can identify that a new object is placed on the exported tray (20) when the image acquired in real time by the camera (100) changes to an image including a new object.

When an object is placed, the tray (20) included in the image acquired by the camera (100) does not change, so the control unit (200) can determine that there is a change in the object placed on the tray (20) that has been removed when the image changes.

The control unit (200) can obtain a second capture image (C2) including the placed object based on identifying that an object is placed on the removed tray (20) (550). That is, since the second capture image (C2) is an image obtained after the object is placed, there may be differences in the placed object from the first capture image (C1).

Additionally, the control unit (200) may obtain a second capture image (C2) at a point in time that includes the identifier of the placed object based on identifying that the removed tray (20) is being stored again after the object has been placed.

As described below, since the object identifier must be detected using the second captured image (C2), the control unit (200) needs to acquire the second captured image (C2) at a point in time when the object identifier can be detected.

In other words, the control unit (200) can obtain the second capture image (C2) at a point in time when the object identifier can be detected from the state where the tray (20) is fully opened to the state where it is fully stored after the object is placed. In order for the object identifier to be detected, the object identifier must first be included in the second capture image (C2).

Since the shooting range of the camera (100) varies depending on the angle of view of the camera (100), if the object identifier is outside the shooting range of the camera (100), the object identifier may not be detected.

This situation can occur mainly when the layer of the exported tray (20) is high and the distance from the camera (100) is close. For example, when the topmost tray (20) is exported, the image acquired by the camera (100) may not include the object identifier portion because a sufficient angle of view is not secured depending on the performance of the camera (100).

In addition, the control unit (200) may acquire multiple capture images at various points in time at regular intervals until the removed tray (20) completes storage, and select a capture image that includes an object identifier among them as a second capture image (C2).

Meanwhile, since the size of the tray (20) included in the second captured image (C2) is the same as the size of the tray (20) included in the first captured image (C1), the control unit (200) can determine the layer of the exported tray (20) based on the acquired second captured image (C2). That is, the layer of the exported tray (20) can be determined by comparing the second captured image (C2) with the reference image (R) of each of the plurality of trays (20).

The control unit (200) can detect the position of a placed object and the identifier of the object based on the difference between the first captured image (C1) and the second captured image (C2) (560, 571).

That is, the control unit (200) can detect a specific space within the tray (20) corresponding to an area including a changed portion of the image as the location of the placed object.

Additionally, the control unit (200) can detect an identifier of a placed object included in a changing portion of the image. Here, the identifier can include a label, QR code, barcode, etc. attached to the object.

FIG. 9 illustrates a second captured image (C2) capable of detecting an identifier (L1) of an object included in a removed tray (20) of a refrigerator (1) according to one embodiment.

Referring to FIG. 9, it can be confirmed that the control unit (200) acquires a second captured image (C2) as it is identified that a new object has been placed on the exported tray (20).

The control unit (200) can detect the position and identifier (L1) of the placed object based on the difference between the first captured image (C1) and the second captured image (C2). For example, the difference between the second captured image (C2) of FIG. 9 and the first captured image (C1) of FIG. 7 is that the object is newly placed in the leftmost space, so it can be detected that the position of the placed object is the leftmost space. In addition, the identifier (L1) attached to the object placed in the leftmost space can be detected.

However, the control unit (200) may fail to detect the identifier of the object depending on the form in which the placed object is placed.

FIG. 10 illustrates a second captured image (C2) in which an identifier of an object contained in a taken-out tray (20) of a refrigerator (1) according to one embodiment cannot be detected.

Referring to FIG. 10, it can be confirmed that the control unit (200) has acquired the second capture image (C2) as it is identified that the object located on the far left side of the exported tray (20) is newly placed, similar to FIG. 9.

However, it can be confirmed that the object identifier (L1) included in the second captured image of Fig. 9 is placed so that its contents can be fully known, whereas the object identifier (L2) included in the second captured image (C2) of Fig. 10 is placed so that its contents cannot be fully known.

Therefore, when an object is placed on a tray (20) that has been removed as shown in Fig. 10, a measure is required to induce the user to rearrange the object in order to detect the object identifier.

That is, the control unit (200) may output a notification message regarding relocation of the object through at least one of the user device (2) and the user interface (400) based on a failure to detect the object identifier (572).

The notification message regarding object repositioning may include content guiding the user to reposition the object so that the object identifier is facing vertically upward from where the camera (100) is located so that the object identifier can be detected by the camera (100).

When the object is repositioned in a direction in which the identifier can be detected by the user, the control unit (200) can again acquire a second captured image (C2) to detect the identifier.

The control unit (200) can obtain detailed information about the object through the detected identifier (580). That is, if the identifier has detailed information expressed externally, such as a label, the control unit (200) can obtain detailed information about the object through characters or images included in the label. In addition, if the identifier is in an encrypted form, such as a QR code or barcode, the control unit (200) can obtain detailed information about the object by executing a separate recognition program.

Detailed information about an object may include the name, type, size, capacity, and weight of the object. In addition, if the object is wine, detailed information about the object may further include the producer, production region, variety, vintage, alcohol content, etc. of the object.

Meanwhile, if the identifier of an object placed on a tray (20) as shown in Fig. 10 is not completely detected, a method of inferring detailed information about the object can be utilized in addition to outputting a notification message for relocation.

In cases where the identifier of an object is not completely detected and only a part of the identifier is included in the second captured image (C2) as in Fig. 10, the control unit (200) may obtain detailed information about the object based on a part of the identifier included in the second captured image (C2).

This is because there may be detailed information about an object that can be obtained with only a part of the identifier. In this case, only a part of the detailed information about the object can be obtained.

The control unit (200) can obtain the remaining part of the detailed information by using the object's placement history or the object's detailed information previously stored in the memory (220) in addition to the detailed information obtained based on a part of the identifier.

Here, the object placement history may include information about objects that the user has placed in the past. That is, the object placement history may include an identifier of an object that has been placed in the past, detailed information about the object, and a captured image showing the appearance of the object.

For example, the control unit (200) can select detailed information having the highest similarity to detailed information obtained based on a portion of an identifier among detailed information of an object placed in the past.

Here, highest similarity means that among the details containing multiple items, there are the most matching items.

Additionally, the control unit (200) can select detailed information having the highest similarity to a portion of detailed information acquired based on a portion of the identifier among the detailed information of the pre-stored object.

Here, highest similarity means that among the details containing multiple items, there are the most matching items.

In addition, the control unit (200) can obtain detailed information about the object by predicting the remaining part of the identifier based on a part of the identifier. In this case, the control unit (200) can also utilize the placement history of the object.

For example, one could select an identifier that is most similar to a portion of the acquired identifier among identifiers of objects deployed in the past.

Here, the highest similarity means that when the identifier is expressed externally as text or an image, such as a label, the area where such text or image is identical is the largest.

In addition, the control unit (200) can obtain some of the detailed information of the object based on the appearance of the object included in the second captured image (C2) in addition to a part of the identifier. For example, the control unit (200) can obtain information on the color, size, and shape of the object based on the second captured image (C2). This is because such information can be obtained regardless of the identifier of the object.

In this case, the control unit (200) can also utilize the placement history of the object. For example, the control unit (200) can select detailed information with the highest similarity in information about the color, size, and shape of the object among detailed information about objects that were placed in the past.

When detailed information obtained by the method of inferring detailed information of the aforementioned object is transmitted to the user device (2) or output through the user interface (400), the control unit (200) may also include information that the obtained detailed information is inferred.

The control unit (200) can store the object location information, object detailed information, and the second captured image (C2) in the memory (220) based on detecting that the removed tray (20) is stored again.

Here, the position information of the object may include the direction in which the object is placed in addition to the space in which the object is placed within the tray (20). For example, if the object is a container containing a beverage, the direction in which the object is placed may include the case where the entrance faces the front of the chamber (11) or the case where it faces the rear of the chamber (11).

When the control unit (200) detects that the stored tray (20) is taken out again, it can obtain a captured image again by the camera (100) as described above. Furthermore, even if the control unit (200) does not obtain a captured image again by the camera (100), it can receive a second captured image (C2) of the same content stored in the memory (220). This is because while the stored tray (20) is taken out again, the object placed on the tray (20) does not change.

The control unit (200) can transmit the acquired object location information and object detailed information to the user device (2) through the communication circuit (590). In addition, the control unit (200) can transmit the second captured image to the user device (2) through the communication circuit.

Fig. 11 illustrates a screen displayed on a user device (2) according to one embodiment.

Fig. 12 illustrates a screen displayed on a user device (2) according to one embodiment.

Referring to FIG. 11 and FIG. 12, a user can check the location and detailed information of an object stored in a refrigerator (1) using a user device (2).

Through the screen (S1) of the user device (2) according to Fig. 11, the user can check which wine is placed in which space. Fig. 11 shows a screen that briefly shows a refrigerator (1) in which a total of 6 layers of trays (20) are stored, and 4 objects can be placed on each layer.

The user can input a floor for which he or she wants to search. The user's input can be input through an interface such as a touch screen panel included in the user device (2). For example, if the user touches a portion (S1-1) corresponding to the 4th floor tray on the screen of Fig. 11, a screen can be provided that allows the user to select one of the 4 spaces in which an object can be placed on the 4th floor tray (20).

Next, when the user touches a portion (S1-2) corresponding to the first space among the portions corresponding to the four-layer tray on the screen, the user device (2) can provide a screen (S2) displaying information about an object placed in the first space as shown in FIG. 12.

Specifically, a screen (S2-2) displaying detailed information of an object can be provided.

In addition, the user device (2) can provide a screen (S2-1) that displays the appearance of the object in addition to the detailed information of the object. That is, the user device (2) can enlarge and display a portion corresponding to the first space included in the second captured image (C2).

Therefore, through the screen of the user device (2), the user can check detailed information about an object placed in a selected space as well as the appearance of the object.

Meanwhile, in addition to placing an object on a tray (20) that has been removed, the refrigerator (1) can also be controlled based on the removal of an object that has already been placed.

Fig. 13 is a flowchart showing a control method of a refrigerator (1) according to one embodiment.

Specifically, the control unit (200) can identify whether one of the plurality of trays (20) is to be removed based on the image acquired by the camera (100) (1310).

When the control unit (200) identifies that the tray (20) has been removed, it can obtain a first capture image of the removed tray (20) (1320).

The control unit (200) can determine the layer of the exported tray (20) by comparing the first captured image with the reference image of each of the plurality of trays (20) (1330).

Up to this point, it is the same as 510 to 530 in Fig. 5.

Next, the control unit (200) can identify that an object has been removed from the tray (20) based on the image acquired by the camera (100) (1340). That is, if the image acquired in real time by the camera (100) changes to an image in which an object has been removed from the tray (20), the control unit (200) can identify that an object previously placed on the tray (20) has been removed.

When an object is removed, the tray (20) included in the image acquired by the camera (100) does not change, so the control unit (200) can determine that a change has occurred in the object placed on the removed tray (20) when the image changes.

The control unit (200) can obtain a third capture image (1350) based on identifying that an object placed on the removed tray (20) has been removed. Since the third capture image is an image obtained after the object has been removed, there may be differences in the placed object from the first capture image.

Accordingly, the control unit (200) can detect the position of the dislodged object based on the difference between the first captured image and the third captured image (1360).

The step of detecting an identifier and determining detailed information, as in Fig. 5, can be omitted because it has been performed previously when the object was placed. The control unit (200) can delete detailed information of a detached object from the memory (220).

In addition, the control unit (200) can transmit location information of the detached object, detailed information of the object, and a third captured image to the user device (2) (1370). In addition, the control unit (200) can transmit a notification message regarding the detachment of the object to the user device (2).

Meanwhile, the control unit (200) can identify that two or more trays (20) among a plurality of trays (20) have been removed based on the image acquired by the camera (100). That is, even though one tray (20) has already been removed, there is a case where the user additionally removes a tray (20) located at a higher floor than the removed tray (20).

However, since the camera (100) according to the present invention is installed above the uppermost tray (20), the image acquired by the camera (100) includes only the tray (20) located at the highest level among the trays (20) that are taken out.

In this situation, if an object is placed or removed from a tray (20) other than the tray (20) located at the highest level, the control unit (200) fails to identify a change in the placed object.

Therefore, the control unit (200) can detect the location and identifier of the object being placed only when the object is placed on the tray (20) located at the highest level among two or more exported trays (20), or only one tray (20) is exported.

To solve this problem, when the control unit (200) identifies that two or more trays (20) among a plurality of trays (20) are being taken out, the control unit (200) may output a message guiding the placement of an object on a tray (20) located at the highest level among the two or more taken out trays (20) or a message guiding the sequential use of each of the two or more taken out trays (20) through at least one of the user interface (400) and the user device (2).

Here, the message guiding sequential use may include content guiding the use of trays (20) one by one, since only one tray (20) out of two or more trays (20) that have been taken out must be used. In addition, the message guiding sequential use may include content guiding the selection of a tray (20) to maintain the taken out state and the return of the remaining trays (20).

A refrigerator (1) according to one embodiment comprises: a chamber (11); a plurality of trays (20) stored in different layers within the chamber (11); a camera (100) provided on an upper portion of the chamber (11) and obtaining an image of an area in which the plurality of trays (20) move; a control unit (200) identifying movement of at least one of the plurality of trays (20) based on the image obtained by the camera (100); And a communication circuit for communicating with a user device (2); wherein the control unit (200) obtains a first capture image including the exported tray (20) based on identifying that one of the plurality of trays (20) is exported, obtains a second capture image including the object based on identifying that an object is placed on the exported tray (20), detects a location of the object and an identifier of the object based on a difference between the first capture image and the second capture image, and provides the location information of the object and detailed information of the object obtained from the identifier to the user device (2) through the communication circuit.

In addition, the control unit (200) can obtain a second captured image (C2) at a point in time when the identifier of the object is included in the image obtained by the camera (100) based on identifying that the tray (20) that was removed after the object was placed is stored again.

In addition, the control unit (200) can acquire a first captured image (C1) at a point in time when an identifier of a previously placed object is included in an image acquired by the camera (100) based on identifying that one of the plurality of trays (20) is being taken out.

In addition, the refrigerator (1) according to one embodiment further includes a memory (220) that stores a reference image of each of the plurality of trays (20); and the control unit (200) can determine the layer of the exported tray (20) by comparing the first captured image or the second captured image with the reference image of each of the plurality of trays (20).

In addition, the refrigerator (1) according to one embodiment further includes a user interface (400); and the control unit (200) can output a notification message regarding relocation of the object through at least one of the user device (2) and the user interface (400) based on a failure to detect an identifier of the object.

In addition, the control unit (200) can store the location information of the object and detailed information of the object in the memory (220) based on identifying that the tray (20) that has been removed is being stored again.

Additionally, the location information of the object may include the direction in which the object is placed, and the detailed information of the object may include the name, type, size, capacity, and weight of the object.

In addition, the control unit (200) may provide at least one of a notification message regarding the object's departure and detailed information about the object that has departed to the user device (2) based on identifying that the object has departed from the tray (20).

In addition, the refrigerator (1) according to one embodiment further includes a memory (220); and the control unit (200) can store the location information of the object and the detailed information of the object in the memory (220), and based on identifying that the object is detached from the tray (20), delete the detailed information of the detached object from the memory (220).

In addition, the refrigerator (1) according to one embodiment further includes a user interface (400); and the control unit (200) can output a message guiding sequential use of each of the two or more trays (20) that have been taken out through the user interface (400) based on identifying that two or more of the plurality of trays (20) are taken out.

A method for controlling a refrigerator (1), comprising: a chamber (11) according to one embodiment; a plurality of trays (20) stored in different layers within the chamber (11); a camera (100) provided on an upper portion of the chamber (11) and obtaining an image; and a communication circuit communicating with a user device (2); may include obtaining an image of an area in which the plurality of trays (20) move by the camera (100); obtaining a first captured image including the removed tray (20) based on identifying that one of the plurality of trays (20) is being removed; obtaining a second captured image including the object based on identifying that an object is placed on the removed tray (20); detecting a location of the object and an identifier of the object based on a difference between the first captured image and the second captured image; and providing the location information of the object and detailed information of the object obtained from the identifier to the user device (2) through the communication circuit.

In addition, obtaining the first captured image (C1) may include obtaining the first captured image (C1) at a point in time when an identifier of a previously placed object is included in the image obtained by the camera (100) based on identifying that one of the plurality of trays (20) is being removed.

In addition, obtaining the second captured image (C2) may include obtaining the second captured image (C2) at a point in time when the identifier of the object is included in the image obtained by the camera (100) based on identifying that the tray (20) that was removed after the object was placed is stored again.

In addition, a method of controlling a refrigerator (1) according to one embodiment may include storing a reference image of each of the plurality of trays (20) in a memory (220); and comparing the first captured image or the second captured image with the reference image of each of the plurality of trays (20) to determine the layer of the exported tray (20).

In addition, a method for controlling a refrigerator (1) according to one embodiment may include outputting a notification message regarding relocation of the object through at least one of the user device (2) and the user interface (400) based on a failure to detect an identifier of the object.

In addition, a control method of a refrigerator (1) according to one embodiment may include storing location information of the object and detailed information of the object in a memory (220) based on identifying that the removed tray (20) is stored again.

Additionally, the location information of the object may include the direction in which the object is placed, and the detailed information of the object may include the name, type, size, capacity, and weight of the object.

In addition, a method for controlling a refrigerator (1) according to one embodiment may include providing, to the user device (2), at least one of a notification message regarding the object's departure and detailed information about the object that has departed, based on identifying that the object has departed from the removed tray (20).

In addition, a control method of a refrigerator (1) according to one embodiment may include storing location information of the object and detailed information of the object in the memory (220); and deleting detailed information of the object that has been removed from the memory (220) based on identifying that the object has been removed from the tray (20).

In addition, a method for controlling a refrigerator (1) according to one embodiment may include outputting a message guiding sequential use of each of the two or more trays (20) that have been taken out through the user interface (400) based on identifying that two or more of the plurality of trays (20) have been taken out.

According to the disclosed refrigerator (1) and its control method, the location and detailed information of an object stored in the refrigerator (1) can be identified using one camera (100).

Accordingly, by providing the user with location information and detailed information of an object through the user device (2), the refrigerator (1) can be managed more effectively, thereby enhancing user convenience.

Furthermore, since only one camera (100) is used, lower power is required and the product cost can be reduced.

Meanwhile, the disclosed embodiments may be implemented in the form of a storage medium storing instructions executable by a computer. The instructions may be stored in the form of program codes, and when executed by a processor (210), may generate program modules to perform the operations of the disclosed embodiments.

The storage medium that can be read by the device may be provided in the form of a non-transitory storage medium. Here, the term 'non-transitory storage medium' means only that it is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play StoreTM) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product (e.g., a downloadable app) may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person skilled in the art to which the present disclosure belongs from the description below.

The above has been illustrated and described with respect to specific embodiments. However, it is not limited to the above embodiments, and those skilled in the art to which the invention pertains can make various modifications and implementations without departing from the gist of the technical idea of the invention described in the claims below.

Claims (15)

chamber; A plurality of trays stored in different layers within the chamber; A camera provided on the upper part of the chamber and obtaining an image of the area in which the plurality of trays move; A control unit that identifies movement of at least one of the plurality of trays based on an image acquired by the camera; and A communication circuit for communicating with a user device; The above control unit, Based on identifying that one of the plurality of trays is being removed, a first capture image including the removed tray is acquired, Based on identifying that an object is placed on the above-mentioned exported tray, a second capture image including the object is acquired, Based on the difference between the first captured image and the second captured image, the location of the object and the identifier of the object are detected, A refrigerator that provides location information of the object and detailed information of the object obtained from the identifier to the user device through the communication circuit. In the first paragraph, The above control unit, A refrigerator that acquires a second captured image at a point in time when an identifier of the object is included in the image acquired by the camera, based on identifying that the removed tray is stored again after the object is placed. In the first paragraph, The above control unit, A refrigerator that acquires a first captured image at a point in time when an identifier of a previously placed object is included in the image acquired by the camera based on identifying that one of the plurality of trays is being taken out. In the first paragraph, Further comprising a memory for storing a reference image of each of the plurality of trays; The above control unit, A refrigerator that determines the layer of the exported tray by comparing the first captured image or the second captured image with a reference image of each of the plurality of trays. In the first paragraph, further including a user interface; The above control unit, A refrigerator that outputs a notification message regarding relocation of the object through at least one of the user device and the user interface based on a failure to detect the identifier of the object. In the first paragraph, The above control unit, A refrigerator that stores location information and detailed information of the object in a memory based on identifying that the tray that has been removed is being stored again. In the first paragraph, The position information of the above object includes the direction in which the object is placed, Detailed information of the above object includes the name, type, size, capacity and weight of the above object. In the first paragraph, The above control unit, A refrigerator that provides at least one of a notification message regarding the object's removal and detailed information about the object that has removed to the user device based on identifying that the object has removed from the tray. In the first paragraph, memory; including more; The above control unit, Store the location information of the object and the detailed information of the object in the memory, A refrigerator that deletes detailed information about the detached object from the memory based on identifying that the object has detached from the tray (20). In the first paragraph, further including a user interface; The above control unit, A refrigerator that outputs a message guiding sequential use of each of the two or more trays that have been removed through the user interface based on identifying that two or more of the plurality of trays are being removed. A method for controlling a refrigerator, comprising: a chamber; a plurality of trays stored in different layers within the chamber; a camera provided on the upper part of the chamber and for obtaining an image; and a communication circuit for communicating with a user device; Obtaining an image of an area in which the plurality of trays move by the camera; Based on identifying that one of the plurality of trays is being pulled out, acquiring a first capture image including the pulled out tray; Acquiring a second capture image including the object based on identifying that the object is placed on the tray that has been exported; Detecting the location of the object and the identifier of the object based on the difference between the first captured image and the second captured image; and A method for controlling a refrigerator, comprising providing location information of the object and detailed information of the object obtained from the identifier to the user device through the communication circuit. In Article 11, Obtaining the above first capture image is as follows: A refrigerator comprising: acquiring a first captured image at a point in time when an identifier of a previously placed object is included in the image acquired by the camera, based on identifying that one of the plurality of trays is being removed. In Article 11, Obtaining the above second capture image is as follows: A refrigerator comprising: acquiring a second captured image at a point in time when an identifier of the object is included in the image acquired by the camera, based on identifying that the removed tray is stored again after the object is placed. In Article 11, Storing a reference image of each of the plurality of trays in memory; and A method for controlling a refrigerator, comprising determining a layer of the exported tray by comparing the first captured image or the second captured image with a reference image of each of the plurality of trays. In Article 11, A method of controlling a refrigerator, comprising outputting a notification message regarding relocation of the object through at least one of the user device and the user interface based on a failure to detect the identifier of the object.

Images