TWI747653B - Judgment method for accessing items and smart refrigerator - Google Patents

Abstract

A method for determining access to items is suitable for a refrigerator body, an image capturing unit, and a processing unit of a smart refrigerator, and includes: when a user holds an item into or from the refrigerator body When taken out, the image capturing unit continuously photographs the opening of the refrigerator body to generate a plurality of input images; the processing unit calculates a skin color area and an item area corresponding to each input image, and then calculates an area ratio accordingly , The skin color area corresponds to the user’s hand, and the item area corresponds to the item; the processing unit determines whether the item is stored in the refrigerator body or is It is taken out of the refrigerator body.

Description

Judgment method for accessing items and smart refrigerator

The present invention relates to a judging method and a refrigerator, in particular to a judging method for storing or taking out items used in the refrigerator, and a smart refrigerator for implementing the judging method.

With the advancement of technology, the functions of various smart home appliances are constantly being developed and applied. Among them, the refrigerator is a household appliance that every household must use. For the management and judgment methods used for items stored in the refrigerator, for example, there are already machine learning models using artificial intelligence, which are trained with a large number of photos of items prepared in advance to determine whether the items in the refrigerator have been stored or taken out . However, whether there are other methods or systems that can make effective judgments without requiring a huge amount of image data has become a problem to be improved.

Therefore, the purpose of the present invention is to provide a method for judging access items and a corresponding smart refrigerator that can operate effectively without requiring a huge amount of image data.

Therefore, one aspect of the present invention provides a method for judging access to items, It is suitable for a refrigerator body, an image capture unit, and a processing unit, and includes steps (A) to (C).

In step (A), when a user holds an object into the refrigerator body or takes it out of the refrigerator body, the image capturing unit continuously photographs the opening of the refrigerator body to generate a plurality of input images.

In step (B), the processing unit calculates a skin color area and an item area corresponding to each input image, and then calculates a corresponding area ratio based on the skin color area and the item area. The skin color area should be used The area of the item corresponds to the person’s hand.

In step (C), the processing unit determines whether the item is stored in the refrigerator body or taken out from the refrigerator body according to the change of the area ratio of each input image.

In some embodiments, in step (A), the input images include the length of time corresponding to the user's hand moving from the outside of the refrigerator body to the inside of the refrigerator body and back to the outside of the refrigerator body.

In some embodiments, in step (B), the area ratio of each input image=the area of the item/(area of the item+area of the skin color).

In some implementations, in step (C), the processing unit determines a turning point of the area ratios of the input images according to a time series change, and then respectively calculates the input images before and after the turning point One of the proportions of these areas A first average value and a second average value. When the processing unit determines that the first average value is less than the second average value, it is determined that the item is taken out of the refrigerator body. When the processing unit determines that the first average value is When the value is greater than the second average value, it is determined that the item is put into the refrigerator body, and the turning point is when the two adjacent input images have the largest changes in the area ratio relative to the others.

Therefore, another aspect of the present invention provides a smart refrigerator including a refrigerator body, an image capturing unit, and a processing unit. The image capturing unit is electrically connected to the processing unit and shoots images toward the opening of the refrigerator body .

When a user holds an object into the refrigerator body or takes it out of the refrigerator body, the image capturing unit continuously shoots the opening of the refrigerator body to generate a plurality of input images.

The processing unit calculates a skin color area and an item area corresponding to each input image, and then calculates a corresponding area ratio according to the skin color area and the item area. The skin color area corresponds to the user's hand and the item area corresponds to It should be an item.

The processing unit determines whether the article is stored in the refrigerator body or taken out from the refrigerator body according to the change of the area ratio of each input image.

In some embodiments, the input images include the length of time corresponding to the user's hand moving from the outside of the refrigerator body to the inside of the refrigerator body and back to the outside of the refrigerator body.

In some embodiments, the area ratio of each input image=the item area/(the item area+the skin color area).

In some implementations, the processing unit determines a turning point of the area ratios of the input images according to time series changes, and then respectively calculates a turning point of the area ratios of the input images before and after the turning point. A first average value and a second average value. When the processing unit determines that the first average value is less than the second average value, it is determined that the item is taken out of the refrigerator body. When the processing unit determines that the first average value is When the value is greater than the second average value, it is determined that the item is put into the refrigerator body, and the turning point is when the two adjacent input images have the largest changes in the area ratio relative to the others.

In other embodiments, the image capturing unit is disposed on a left wall surface or a right wall surface inside a refrigerating compartment of the refrigerator body.

The effect of the present invention is to determine whether the item is stored in the refrigerator body or taken out from the refrigerator body by the change of the area ratio of each input image, so that the determining method and the smart refrigerator Compared with the conventional technology that uses artificial intelligence algorithms, it can make correct judgments based on only a small number of such input images.

100: Smart refrigerator

1: Refrigerator body

11: Refrigerator

111: left wall

112: right wall

12: Freezer

2: processing unit

3: Image capture unit

S1~S3: steps

A1, A3, A4, A6: input image

P1, P2: position

The other features and effects of the present invention will be described in the embodiment with reference to the drawings Figure 1 is a block diagram illustrating a smart refrigerator to which the method for determining access to items of the present invention is applicable; Fig. 2 is a flowchart illustrating an embodiment of the method for determining access to items of the present invention Figure 3 is a three-dimensional view illustrating the location of an image capturing unit of this embodiment; and Figure 4 is a schematic diagram illustrating four input images of this embodiment.

Before the present invention is described in detail, it should be noted that in the following description, similar elements are denoted by the same numbers.

Referring to FIGS. 1 and 2, the method for determining access to items of the present invention is applicable to a smart refrigerator 100, which includes a refrigerator body 1, an image capturing unit 3, and a processing unit 2. The processing unit 2 is, for example, a central processing unit or other computing chips, and is, for example, installed on the refrigerator body 1. The image capturing unit 3 is, for example, a camera module, which is electrically connected to the processing unit 2 and shoots images toward the opening of the refrigerator body 1. Referring again to FIG. 3, FIG. 3 is a perspective view of the refrigerator body 1. The refrigerator body includes a refrigerating compartment 11 and a freezing compartment 12, but not limited to this. In this embodiment, the image capturing unit 3 is provided on a left wall 111 of the refrigerator body 1 inside the refrigerating compartment 11 Position P1, or another position P2 on a right wall 112. Among them, the two arrows represent the shooting direction when the image capturing unit is set at the positions P1 and P2, but it is not limited to this.

The method for determining whether to deposit or withdraw items includes steps S1 to S3.

In step S1, when a user holds an object into the refrigerator body 1 or takes it out of the refrigerator body 1, the image capturing unit 3 continuously photographs the opening of the refrigerator body 1 to generate a plurality of input images. For example, the smart refrigerator 100 also includes a dynamic sensor (such as an infrared sensor). When the user opens the refrigerator body 1 and his hand approaches the opening of the refrigerator body 1, the dynamic sensor The sensor detects the user's hand to directly trigger or indirectly trigger the image capturing unit 3 to take a photo via the processing unit 2. In other words, the input images include the length of time corresponding to the user's hand moving from outside the refrigerator body 1 to the refrigerator body 1 and back to the outside of the refrigerator body 1. In this embodiment, the number of the input images is six.

Referring again to FIG. 4, FIG. 4 exemplarily illustrates the first input image A1, the third input image A3, the fourth input image A4, and the fourth input image A1, the third input image A3, and the fourth input image A1 of the six input images shot in chronological order. Six of the input images A6.

In step S2, the processing unit 2 calculates a skin color area and an item area corresponding to each input image. The skin color area corresponds to the user's hand and the item area corresponds to the item. The processing unit 2 then calculates a corresponding area ratio according to the skin color area and the item area of each input image. In more detail, every time you lose The ratio of the area of the incoming image = the area of the item/(area of the item + area of the skin color).

4 again, for example, the first input image A1, the second input image (not shown), the third input image A3, the fourth input image A4, the fifth input image The area ratios of the image (not shown) and the sixth input image A6 are 0, 0.03, 0.11, 0.37, 0.44, and 0.45, respectively.

In step S3, the processing unit 2 determines whether the item is stored in the refrigerator main body 1 or taken out from the refrigerator main body 1 according to the change of the area ratio of each input image. More specifically, the processing unit 2 determines a turning point of the area ratios of the input images according to time series changes, and the turning point is when the two adjacent input images have the largest changes in the area ratios relative to the others. . Continuing the previous example, the change of the two area ratios (ie 0.11 and 0.37) corresponding to the third input image A3 and the fourth input image A4 is equal to 0.26 (ie 0.37-0.11), which is relative to other The changes in the proportions of the two adjacent input images corresponding to the two areas are even greater. Therefore, the turning point is located between the third input image A3 and the fourth input image A4.

The processing unit 2 then respectively calculates a first average value and a second average value of the area proportions of the input images before and after the turning point. When the processing unit 2 determines that the first average value is less than the second average value, it is determined that the article is taken out of the refrigerator body 1. When the processing unit 2 determines that the first average value is greater than the second average value, it is determined that the article is placed in the refrigerator body 1.

Continuing the previous example, the first average value and the second average value are respectively equal to 0.05 and 0.42, that is, 0.05<0.42. Therefore, the input images corresponding to FIG. 4 will be judged as the item being contained in the refrigerator body 1. take out. In addition, it should be noted that: in other embodiments, the image capturing unit 3 can also be arranged in other positions of the refrigerator body 1, as long as the captured input images can show the change of the turning point. Can.

In summary, the processing unit 2 determines whether the item is stored in the refrigerator body 1 or is determined by the change in the area ratio of each input image generated by the image capturing unit 3 The refrigerator body 1 is taken out, so that the judgment method and the smart refrigerator 100, compared with the conventional technology using artificial intelligence algorithms, can make correct judgments based on only a small number of the input images, so it can indeed be achieved The purpose of the present invention.

However, the above are only examples of the present invention. When the scope of implementation of the present invention cannot be limited by this, all simple equivalent changes and modifications made in accordance with the scope of the patent application of the present invention and the content of the patent specification still belong to Within the scope covered by the patent of the present invention.

S1~S3: steps

Claims (7)

A method for determining access to items is suitable for a refrigerator body, an image capture unit, and a processing unit, and includes: (A) When a user holds an item into or from the refrigerator body When taken out, the image capturing unit continuously photographs the opening of the refrigerator body to generate a plurality of input images; (B) the processing unit calculates a skin color area and an item area corresponding to each input image, and then according to The skin color area and the item area are calculated corresponding to an area ratio, the skin color area corresponds to the user's hand, and the item area corresponds to the item; and (C) the processing unit determines the input images according to time series changes A turning point with equal area ratios, and then respectively calculate a first average value and a second average value of the area ratios of the input images before and after the turning point. When the processing unit determines that the first average value is less than When the second average value is used, it is determined that the item is taken out of the refrigerator body. When the processing unit determines that the first average value is greater than the second average value, it is determined that the item is placed in the refrigerator body. The turning point is when the change in the ratio of the two adjacent input images is the largest relative to the others. The method for determining access to items according to claim 1, wherein, in step (A), the input images include the movement of the user's hand from the outside of the refrigerator body to the inside of the refrigerator body and then back to the refrigerator body. The length of time outside the body. The method for determining whether to access an item according to claim 2, wherein, in step (B), the area ratio of each input image=the item area/(the item area+the skin color area). A smart refrigerator includes a refrigerator body, an image capture unit, and a processing unit. The image capture unit is electrically connected to the processing unit and shoots images toward the opening of the refrigerator body. When a user holds an item When put into the refrigerator body or taken out of the refrigerator body, the image capturing unit continuously photographs the opening of the refrigerator body to generate a plurality of input images, and the processing unit calculates a corresponding skin color area for each input image And an item area, and then calculate a corresponding area ratio based on the skin color area and the item area. The skin color area corresponds to the user's hand and the item area corresponds to the item. The processing unit determines the input image according to the time sequence change. A turning point of the area ratios, and then respectively calculating a first average value and a second average value of the area ratios of the input images before and after the turning point, when the processing unit determines the first average value When it is less than the second average value, it is determined that the item is taken out of the refrigerator body, and when the processing unit determines that the first average value is greater than the second average value, it is determined that the item is placed in the refrigerator body, The turning point is when the two adjacent input images have the largest changes in the area ratio relative to the others. The smart refrigerator according to claim 4, wherein the input images include the length of time corresponding to the user's hand moving from the outside of the refrigerator body to the inside of the refrigerator body and back to the outside of the refrigerator body. The smart refrigerator according to claim 5, wherein the area ratio of each input image=the area of the item/(area of the item+area of the skin color). The smart refrigerator according to claim 4, wherein the image capturing unit is provided On a left wall surface or a right wall surface inside a refrigerating compartment of the refrigerator body.

Images