WO2022217935A1 - 冰箱内物品信息识别方法和冰箱 - Google Patents

冰箱内物品信息识别方法和冰箱 Download PDF

Info

Publication number
WO2022217935A1
WO2022217935A1 PCT/CN2021/133222 CN2021133222W WO2022217935A1 WO 2022217935 A1 WO2022217935 A1 WO 2022217935A1 CN 2021133222 W CN2021133222 W CN 2021133222W WO 2022217935 A1 WO2022217935 A1 WO 2022217935A1
Authority
WO
WIPO (PCT)
Prior art keywords
refrigerator
item
image
information
area
Prior art date
Application number
PCT/CN2021/133222
Other languages
English (en)
French (fr)
Inventor
高洪波
Original Assignee
青岛海尔电冰箱有限公司
海尔智家股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 青岛海尔电冰箱有限公司, 海尔智家股份有限公司 filed Critical 青岛海尔电冰箱有限公司
Publication of WO2022217935A1 publication Critical patent/WO2022217935A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25B—REFRIGERATION MACHINES, PLANTS OR SYSTEMS; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS
    • F25B29/00—Combined heating and refrigeration systems, e.g. operating alternately or simultaneously
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching

Definitions

  • the invention relates to the field of refrigeration devices, in particular to a method for identifying information of items in a refrigerator and a refrigerator.
  • the identification of internal items has become a necessary function of smart refrigerators.
  • one or more cameras are installed in the internal storage space of the refrigerator to take pictures of the items stored in the refrigerator, and after processing the photos taken, the identification results are generated and sent to the user terminal. for users to view.
  • the purpose of the present invention is to provide a method for identifying information of items in a refrigerator and a refrigerator.
  • the invention provides a method for identifying information of items in a refrigerator, comprising the steps of:
  • mapping relationship between multiple images, and confirming the positions of the same item in the storage space in different images according to the mapping relationship
  • the identification results of the category information of the same item in different images are inconsistent, detect and identify the size of the area of the area that the item is blocked by other items or the structure of the refrigerator in the multiple images, and use the image with the smallest area of the blocked area. Detect the identified item category information as the final output information.
  • mapping relationship between multiple images specifically includes:
  • Information as final output information specifically includes:
  • the K-Means clustering algorithm is used to cluster the pixels in the item image, and the pixels are divided into occluder area pixels and item area pixels;
  • Information as final output information specifically includes:
  • "acquiring images of the storage space in the same refrigerator taken from at least two different angles” specifically includes:
  • a bird's-eye image of the space inside the bottle holder of the refrigerator is obtained from two different angles to obtain a first image and a second image.
  • "obtaining bird's-eye images of the space inside the bottle holder of the refrigerator taken from two different angles” specifically includes:
  • "acquiring images of the storage space in the same refrigerator taken from at least two different angles” specifically includes:
  • the first image and the second image are obtained by controlling the camera at the first preset point and the second setting point to capture the space inside the bottle holder of the refrigerator, respectively.
  • "respectively detecting and identifying the category information of the item in multiple images” specifically includes:
  • the present invention also provides a refrigerator, wherein the refrigerator is provided with at least two cameras located at different positions at one or both of the top of the box body and the bottom surface of the bottle base;
  • the refrigerator further includes a memory and a processor, the memory stores a computer program that can be executed on the processor, and the processor implements the steps of the above-mentioned method for recognizing item information in the refrigerator when the processor executes the program.
  • the present invention also provides a refrigerator, the refrigerator is provided with a transmission structure on the top of the box body and/or the bottom surface of the bottle base, and the camera moves along the transmission structure;
  • the refrigerator further includes a memory and a processor, the memory stores a computer program that can be executed on the processor, and the processor implements the steps of the above method for identifying item information in the refrigerator when the processor executes the program.
  • the present invention captures images in the storage space of the refrigerator from multiple angles, and after establishing the mapping relationship of the multiple images, detects the occluded area of the items whose identification results are deviated in the multiple images, The detection and recognition results of the items in the images with smaller occluded areas are selected as the output results, which reduces the influence of mutual occlusion between the ingredients on the recognition results, and improves the accuracy of the refrigerator in image recognition.
  • FIG. 1 is a schematic flowchart of a method for identifying information of items in a refrigerator according to an embodiment of the present invention.
  • FIGS. 2a and 2b are schematic diagrams of the inner space of the bottle holder captured by two cameras arranged at different positions on the top of the refrigerator box according to an embodiment of the present invention (except for the bottle holder and its contents for illustration, the drawings are omitted the rest of the content).
  • 3a and 3b are schematic diagrams of the space in the bottle seat below the bottle seat taken by two cameras disposed at the bottom of the bottle seat of the refrigerator according to an embodiment of the present invention.
  • the term used to describe the relative position in space such as “upper”, “lower”, “rear”, “front”, etc., is used to describe one unit or feature shown in the drawings relative to another A unit or feature relationship.
  • the term spatially relative position may include different orientations of the device in use or operation other than the orientation shown in the figures. For example, if the device in the figures is turned over, elements described as “below” or “above” other elements or features would then be oriented “below” or “above” the other elements or features.
  • the exemplary term “below” can encompass both a spatial orientation of below and above.
  • the present invention provides a method for identifying information of items in a refrigerator, comprising the steps of:
  • bird's-eye images of the space in the bottle holder of the refrigerator taken at two different angles are acquired to obtain the first image and the second image.
  • the bird's-eye image can be captured by a camera placed above the storage space to be captured. Shooting from above can reduce the problem of mutual occlusion between the ingredients, and the top image of the food can be captured in the image. Compared with other areas of the food, at different angles The top surface pattern below is relatively fixed, and the detection and recognition rate is high. Shooting from two different angles can obtain two images showing different occlusion relationships, which can cover most of the details of the item, so as to facilitate subsequent item information identification.
  • images of the interior space of the refrigerator bottle holder taken from three or more different angles may also be acquired, so as to improve the accuracy of item information identification.
  • S1a2 Control the cameras that are higher than the top surface of the bottle holder to be photographed and are located at two different positions to photograph the space in the bottle holder of the refrigerator, respectively, to obtain a first image and a second image.
  • the cameras are provided on the left and right sides of the top of the refrigerator box
  • the refrigerator door is provided with multiple rows of bottle seats, which are used to place items such as food materials and medicines that need to be stored at low temperature
  • an angle sensor is provided on the refrigerator door.
  • the two cameras are controlled to capture the first image and the second image.
  • the preset angle is 30°.
  • the camera located at the top of the refrigerator box shoots the bottle seat area located on the door body
  • the upper end surface pattern of the items stored in the bottle seat can be photographed from a top-down angle, so as to facilitate the follow-up Identify the category of the item, and when shooting at a position of 30°, the obstruction between objects is small, easy to identify, and the item occupies a large proportion in the image, which can reduce invalid information in the image.
  • the camera can also be arranged at the bottom of the bottle holder, arranged vertically downward, for capturing images of the items in the bottle holder below it, or at the bottom of the bottle holder and the top of the refrigerator box at the same time. , as long as you can shoot from two different angles.
  • FIG. 2a and 2b it is a schematic diagram of the inner space of the bottle holder captured by two cameras located at different positions on the top of the refrigerator box (except for the bottle holder and its contents for illustration, The rest of the figures are omitted), as shown in FIG. 3a and FIG. 3b, which are schematic diagrams of the space in the bottle seat below the two cameras set at the bottom of the bottle seat of the refrigerator.
  • S1b1 Control the camera to move along a preset path at a position higher than the top surface of the bottle base to be shot.
  • S1b2 Control the camera at the first preset point and the second preset point to capture the space in the bottle holder of the refrigerator to obtain the first image and the second image.
  • the camera is arranged at the bottom of the bottle holder through the slide rail, the camera is arranged vertically downward, the slide rail extends from one end of the bottle holder to the other end, and the camera moves along the slide rail to capture the space image in the bottle holder below it, the first The preset point and the second preset point are respectively located at two ends of the slide rail, that is, the cameras are respectively shooting at opposite sides.
  • the slide rail can also be embedded in the top of the refrigerator box, the camera is arranged toward the door body, and the first preset point and the second preset point are located at both ends of the slide rail, respectively.
  • S2 Establish a mapping relationship between multiple images, and confirm the positions of the same item in different images in the storage space according to the mapping relationship.
  • establishing a mapping relationship specifically includes steps:
  • feature points such as SIFT/SURF/FAST/ORB are extracted for each image, and the descriptor corresponding to each feature point is extracted.
  • an algorithm such as RANSAC can also be used to eliminate incorrectly matched feature points to improve the matching accuracy.
  • the top image information of the items in the refrigerator bottle holder in the first image and the second image is identified. Since the items are photographed from the top, the top image information of the items can be obtained. Detecting and recognizing item type information based on the top image information, compared with identifying the side pattern of the item, the top pattern recognition is simpler and the recognition accuracy is higher.
  • the recognition results in the first image and the second image are consistent, it can be determined that the areas covered by the object in the two images do not affect the recognition results, and the recognition results are highly reliable.
  • the inconsistency mentioned here includes the detection of the same item in two images as two different types of food, or, one image outputs the detection result, and one image is judged as unrecognizable. At this point, it is necessary to compare the occluded area of the item in different images, which specifically includes the following steps:
  • S42a1 Use the K-Means clustering algorithm to cluster the pixels in the object image, and divide the pixels into occluder area pixels and object area pixels.
  • two cluster center points are generated, and the pixel points are divided into two categories according to the cluster center points. For each pixel point, find the nearest cluster center point, perform a clustering, repeat the above steps, until the pixel point is divided into occluder area pixels and item area pixels.
  • the specific steps of the K-Means clustering algorithm are in the prior art, and are not repeated here.
  • S42a2 Calculate the number of pixel points in the item area in the multiple images, and use the item category information detected and identified in the image with the largest number as the final output information.
  • clustering algorithms may also be used to classify pixel points, which are not specifically limited by the present invention. Or compare through other algorithms, such as the following steps:
  • S42b1 Use the edge detection algorithm to identify the edge contour of the item.
  • the edge detection algorithm can adopt the currently common algorithms such as Roberts, Sobel, Prewitt, Canny, Log, etc. The specific algorithm content will not be repeated here.
  • S42b2 Calculate the number of pixels in the edge contour of the item in the multiple images, and use the detected and identified item category information in the image with the largest number as the final output information.
  • the top surface of the article 2 is partially occluded by the article 1.
  • the detection of the category information of the article 1 may be deviated or unrecognized due to the incomplete pattern
  • Fig. 2b Since its shooting angle is different from that in Figure 2a, the top surface of item 2 is not blocked at this time, the detection and recognition results are more reliable, and the number of pixels within the edge contour line of the item area is more than that of item 2 in Figure 2. The number of pixels in 2a, so the detection and identification information of item 2 in Figure 2b will be selected as the final output information.
  • the image is removed, and the judgment and comparison are made according to the detection and recognition results in the remaining images.
  • the present invention also provides a refrigerator, wherein the refrigerator is provided with at least two cameras located at different positions at one or both of the top of the box body and the bottom surface of the bottle base.
  • the refrigerator door body is also provided with an angle sensor for detecting the angle between the refrigerator door body and the refrigerator box body.
  • the refrigerator further includes a memory and a processor.
  • the memory stores a computer program that can be executed on the processor.
  • the processor executes the program, the steps of the above-mentioned method for recognizing item information in the refrigerator are implemented.
  • the invention also provides a refrigerator.
  • the refrigerator is provided with a transmission structure on the top of the box and/or the bottom surface of the bottle base.
  • the camera moves along the transmission structure.
  • the present invention captures images in the refrigerator storage space from multiple angles, and after establishing the mapping relationship of the multiple images, detects the occluded area of the item whose identification results are deviated in the multiple images, and selects the The detection and recognition results of the items in the images with small occluded areas are used as the output results, which reduces the influence of mutual occlusion between the ingredients on the recognition results, and improves the accuracy of the refrigerator in image recognition.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Biology (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Mechanical Engineering (AREA)
  • Thermal Sciences (AREA)
  • Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)

Abstract

本发明提供一种冰箱内物品信息识别方法和冰箱，所述冰箱内物品信息识别方法通过于多个角度拍摄冰箱存储空间内图像，并在建立多幅图像的映射关系后，针对在多幅图像中识别结果出现偏差的物品检测其被遮挡面积，选择以被遮挡区域较小的图像中物品的检测识别结果作为输出结果，降低了食材间相互遮挡对识别结果的影响，提高了冰箱在图像识别中的准确率。

Description

冰箱内物品信息识别方法和冰箱
本申请要求了申请日为2021年04月16日，申请号为202110412306.8，发明名称为“冰箱内物品信息识别方法、冰箱和计算机存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及制冷装置领域，具体地涉及一种冰箱内物品信息识别方法和冰箱。
背景技术
随着家用电器智能化的发展，对内部物品进行识别已经是智能冰箱所必需的功能。通常为了实现对冰箱内存放物品的识别，冰箱的内部储存空间内都安装有一个或多个摄像头用于拍摄冰箱内存放物品，并对拍摄的照片进行处理后生成识别结果发送到使用终端，以供用户查看。
然而，拍摄冰箱内物品图像时，由于物品通常摆放较密集，在图像上物品之间通常存在互相遮挡而导致难以识别。
发明内容
本发明的目的在于提供一种冰箱内物品信息识别方法和冰箱。
本发明提供一种冰箱内物品信息识别方法，包括步骤：
获取多个不同角度拍摄的冰箱内储存空间图像；
建立多幅图像之间的映射关系，根据所述映射关系，确认储存空间内同一物品分别在不同所述图像中的位置；
分别于多幅图像中检测识别所述物品的类别信息；
若同一所述物品在不同图像中类别信息识别结果一致，则判断该所述识别结果为最终输出信息；
若同一物品在不同图像中类别信息识别结果不一致，则检测识别多幅图像中该所述物品被其他物品或冰箱结构所遮挡区域面积的大小，并以被遮挡区域面积最小的一幅图像中所检测识别的物品类别信息作为最终输出信息。
作为本发明的进一步改进，“建立多幅图像之间的映射关系”具体包括：
提取多幅图像中的特征点；
匹配特征点描述子，找到两张图中匹配的特征点对；
计算变换对应变换关系矩阵。
作为本发明的进一步改进，“检测识别多幅图像中该所述物品未被其他物品或冰箱 结构所遮挡区域面积的大小，并以为被遮挡区域面积最大的一幅图像中所检测识别的物品类别信息作为最终输出信息”具体包括：
使用K-Means聚类算法对物品图像中的像素点进行聚类，将像素点划分为遮挡物区域像素点和物品区域像素点；
计算多幅图像中物品区域像素点数量，并以数量最多的一幅图像中所检测识别的物品类别信息作为最终输出信息。
作为本发明的进一步改进，“检测识别多幅图像中该所述物品未被其他物品或冰箱结构所遮挡区域面积的大小，并以为被遮挡区域面积最大的一幅图像中所检测识别的物品类别信息作为最终输出信息”具体包括：
使用边缘检测算法识别物品边缘轮廓线；
计算多幅图像中物品区域像素点数量，并以像素点数量最多的一幅图像中所检测识别的物品类别信息作为最终输出信息。
作为本发明的进一步改进，“获取以至少两个不同角度拍摄的同一冰箱内储存空间的图像”具体包括：
获取以两个不同角度所拍摄的冰箱瓶座内空间的鸟瞰图像，得到第一图像和第二图像。
作为本发明的进一步改进，“获取以两个不同角度所拍摄的冰箱瓶座内空间的鸟瞰图像”具体包括：
检测到冰箱门体运动至与冰箱箱体之间夹角呈预设角度；
控制高于所需拍摄瓶座顶面且位于两处不同位置的摄像机分别拍摄冰箱瓶座内空间，得到所述第一图像和所述第二图像。
作为本发明的进一步改进，“获取以至少两个不同角度拍摄的同一冰箱内储存空间的图像”具体包括：
控制摄像机在高于所需拍摄瓶座顶面的位置沿预设路径运动；
分别在第一预设点位和第二设点位控制摄像机拍摄冰箱瓶座内空间得到所述第一图像和所述第二图像。
作为本发明的进一步改进，“分别于多幅图像中检测识别所述物品的类别信息”具体包括：
识别所述第一图像和所述第二图像中冰箱瓶座内物品的顶面图像信息；
根据所述顶面图像信息检测识别物品种类信息。
本发明还提供一种冰箱，所述冰箱于箱体顶部和瓶座底面中的一处或两处设有至少两个位于不同位置的摄像机；
所述冰箱还包括存储器和处理器，所述存储器存储有可在所述处理器上运行的计算 机程序，所述处理器执行所述程序时实现上述冰箱内物品信息识别方法的步骤。
本发明还提供一种冰箱，所述冰箱于箱体顶部和/或瓶座底面设有传动结构，摄像机沿所述传动结构运动；
所述冰箱还包括存储器和处理器，所述存储器存储有可在所述处理器上运行的计算机程序，所述处理器执行所述程序时实现上述冰箱内物品信息识别方法的步骤。
本发明的有益效果是：本发明通过于多个角度拍摄冰箱存储空间内图像，并在建立多幅图像的映射关系后，针对在多幅图像中识别结果出现偏差的物品检测其被遮挡面积，选择以被遮挡区域较小的图像中物品的检测识别结果作为输出结果，降低了食材间相互遮挡对识别结果的影响，提高了冰箱在图像识别中的准确率。
附图说明
图1是本发明一实施方式中一种冰箱内物品信息识别方法的流程示意图。
图2a和图2b是本发明一实施方式中两个设于冰箱箱体顶部不同位置的摄像机所拍摄瓶座内空间的示意简图(除用以说明的瓶座及其内物品外，省略图中其余内容)。
图3a和图3b是本发明一实施方式中两个设于冰箱瓶座底部的摄像机拍摄的其下方瓶座内空间的示意简图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合本申请具体实施方式及相应的附图对本申请技术方案进行清楚、完整地描述。显然，所描述的实施方式仅是本申请一部分实施方式，而不是全部的实施方式。基于本申请中的实施方式，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施方式，都属于本申请保护的范围。
下面详细描述本发明的实施方式，实施方式的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施方式是示例性的，仅用于解释本发明，而不能理解为对本发明的限制。
为方便说明，本文使用表示空间相对位置的术语来进行描述，例如“上”、“下”、“后”、“前”等，用来描述附图中所示的一个单元或者特征相对于另一个单元或特征的关系。空间相对位置的术语可以包括设备在使用或工作中除了图中所示方位以外的不同方位。例如，如果将图中的装置翻转，则被描述为位于其他单元或特征“下方”或“上方”的单元将位于其他单元或特征“下方”或“上方”。因此，示例性术语“下方”可以囊括下方和上方这两种空间方位。
如图1所示，本发明提供一种冰箱内物品信息识别方法，包括步骤：
S1：获取多个不同角度拍摄的冰箱内储存空间图像。
在本实施方式中，获取以两个不同角度所拍摄的冰箱瓶座内空间的鸟瞰图像，得到第一图像和第二图像。鸟瞰图像可以通过设于所要拍摄储存空间上方的摄像机拍摄得到，从上方拍摄可以减少食材间的互相遮挡问题，并且在图像中拍摄得到食物的顶面图像，相对于食物其他区域图案，在不同角度下的顶面图案相对固定，检测识别率高。从两个不同角度拍摄，可以获得呈现出不同遮挡关系的两幅图像，能够涵盖物品大部分细节，以便于进行后续物品信息识别。
在本发明的另一些实施方式中，也可获取于三个或更多不同角度拍摄的冰箱瓶座内空间图像，以提高物品信息识别准确率。
具体的，根据冰箱结构不同，获取图像有多种方法，下面以几个例子例进行说明。
在实施例一中，其包括步骤：
S1a1：检测到冰箱门体运动至与冰箱箱体之间夹角呈预设角度
S1a2：控制高于所需拍摄瓶座顶面且位于两处不同位置的摄像机分别拍摄冰箱瓶座内空间，得到第一图像和第二图像。
示例性的，摄像机设于冰箱箱体顶部左右两侧，冰箱门体上设有多排瓶座，用于放置诸如食材、药品等需要在低温下储存的物品，冰箱门体处设有角度传感器，在检测到冰箱门体运动至与冰箱箱体之间夹角呈预设角度时，控制两个摄像机拍摄得到第一图像和第二图像。优选的，预设角度为30°，此时位于冰箱箱体顶部的摄像机拍摄位于门体上的瓶座区域时，以俯视角度能够拍摄到瓶座内所存放物品的上端面图案，以方便后续对物品类别识别，而且在30°的位置进行拍摄时，物体之间的阻挡较小，易于识别，物品在图像中占比大，可以减少图像中的无效信息。
在本实施例的另一些实施方式中，摄像机也可设于瓶座底部，竖直向下设置，用于拍摄其下方的瓶座内物品图像，或者同时在瓶座底部和冰箱箱体顶部设置，只要能够从两个不同角度拍摄即可。
示例性的，如图2a和图2b所示，为两个设于冰箱箱体顶部不同位置的摄像机所拍摄瓶座内空间的示意简图(除用以说明的瓶座及其内物品外，省略图中其余内容)，如图3a和图3b所示，为两个设于冰箱瓶座底部的摄像机拍摄的其下方瓶座内空间的示意简图。
在实施例二中，其包括步骤：
S1b1：控制摄像机在高于所需拍摄瓶座顶面的位置沿预设路径运动。
S1b2：分别在第一预设点位和第二设点位控制摄像机拍摄冰箱瓶座内空间得到第一图像和第二图像。
示例性的，摄像机通过滑轨设于瓶座底部，摄像机竖直向下设置，滑轨从瓶座一端延伸至其另一端，摄像机沿滑轨运动拍摄其下方的瓶座内空间图像，第一预设点位和第 二预设点位分别位于滑轨两端，即摄像机分别在相对两侧拍摄。或者，滑轨也可内嵌设于冰箱箱体顶部，摄像机朝向门体设置，第一预设点位和第二预设点位分别位于滑轨两端。
S2：建立多幅图像之间的映射关系，根据映射关系，确认储存空间内同一物品分别在不同图像中的位置。
具体的，在本实施方式中，建立映射关系具体包括步骤：
S21：提取多幅图像中的特征点。
具体的，提取每张图SIFT/SURF/FAST/ORB等特征点，并提取每个特征点所对应的描述子。
S22：匹配特征点描述子，找到两张图中匹配的特征点对。
进一步的，还可使用RANSAC等算法剔除错误匹配的特征点，以提高匹配准确率。
S23：计算变换对应变换关系矩阵。
根据变换矩阵计算变换结果，并绘制变换后图像，从而建立两张图像之间的映射关系。
在本发明的其他实施方式中，也可采用其他算法建立第一图像和第二图像之间的映射关系，具体所用算法为现有技术，在此不再赘述。
S3：分别于多幅图像中检测识别物品的类别信息。
具体的，在本实施方式中，识别第一图像和第二图像中冰箱瓶座内物品的顶面图像信息，由于从顶部对物品进行拍摄，可以获得物品顶面图像信息。根据顶面图像信息检测识别物品种类信息，相比于识别物品侧面图案，顶部图案识别更加简便且识别准确率更高。
S41：若同一物品在不同图像中类别信息识别结果一致，则判断该识别结果为最终输出信息。
由于在第一图像和第二图像中识别结果一致，可以判断物品在两幅图像中所被遮挡的区域均不影响识别结果，该识别结果可信度高。
S42：若同一物品在不同图像中类别信息识别结果不一致，则检测识别多幅图像中该物品被其他物品或冰箱结构所遮挡区域面积的大小，并以被遮挡区域面积最小的一幅图像中所检测识别的物品类别信息作为最终输出信息。
由于在第一图像和第二图像中识别结果不一致，可以判断物品在其中一幅图像中所被遮挡的区域影响识别结果。这里所说的不一致包括两幅图像中对同一物品的检测为两个不同种类食物，或者，一幅图像输出检测结果，一幅图像判断为无法识别。此时，需要对物品在不同图像中的被遮挡面积进行比较，具体包括步骤：
S42a1：使用K-Means聚类算法对物品图像中的像素点进行聚类，将像素点划分为 遮挡物区域像素点和物品区域像素点。
具体的，生成两个聚类中心点，根据聚类中心点，将像素点进行分为两类。针对每个像素点，找到距离其最近的聚类中心点，进行一次聚类，重复上述步骤，直至将像素点划分为遮挡物区域像素点和物品区域像素点。K-Means聚类算法具体步骤为现有技术，在此不再赘述。
S42a2：计算多幅图像中物品区域像素点数量，并以数量最多的一幅图像中所检测识别的物品类别信息作为最终输出信息。
在多幅图像中，物品区域像素点数量越多即近似于物品在图像中越完整，将物品区域像素点数量最多一幅图像中的检测识别结果作为可信度最高的结果。
在本发明的其他实施方式中，也可以使用其它聚类算法对像素点进行分类，对此，本发明不做具体限制。或通过其他算法进行比较，如可通过以下步骤：
S42b1：使用边缘检测算法识别物品边缘轮廓线。
边缘检测算法可以采用诸如Roberts、Sobel、Prewitt、Canny、Log等目前常见的算法，具体算法内容再此不再赘述。
S42b2：计算多幅图像中物品边缘轮廓线内像素点数量，并以数量最多的一幅图像中所检测识别的物品类别信息作为最终输出信息。
示例性的，在图2a中，物品2顶面被物品1部分遮挡，此时在图2a中，对物品1的类别信息检测可能由于图案不全而出现偏差或无法识别，而在图2b中，由于其拍摄角度与图2a不同，此时物品2的顶面未被遮挡，其检测识别结果可信度更高，并且其物品区域内边缘轮廓线之内的像素点数量多于物品2在图2a中的像素点数量，因此会选择将图2b中物品2的检测识别信息作为最终输出信息。类似的，在图3a中，物品4被物品3所遮挡，选择图3b中物品4的检测识别信息作为最终输出信息；在图3b中，物品5被物品6所遮挡，选择图3a中物品5的检测识别信息作为最终输出信息。
进一步的，在本发明的一些实施方式中，还包括步骤：
当物品在图像中存在无法识别的情况时，剔除该图像，根据其余图像中的检测识别结果进行判断比较。
本发明还提供一种冰箱，冰箱于箱体顶部和瓶座底面中的一处或两处设有至少两个位于不同位置的摄像机。
进一步的，冰箱门体处还设有角度传感器，用于检测冰箱门体与冰箱箱体之间的角度。
冰箱还包括存储器和处理器，存储器存储有可在处理器上运行的计算机程序，处理器执行程序时实现上述的冰箱内物品信息识别方法的步骤。
本发明还提供一种冰箱，冰箱于箱体顶部和/或瓶座底面设有传动结构，摄像机沿传 动结构运动，摄像机被配置于在滑轨固定位置对冰箱内储存空间进行拍摄。
综上所述，本发明通过于多个角度拍摄冰箱存储空间内图像，并在建立多幅图像的映射关系后，针对在多幅图像中识别结果出现偏差的物品检测其被遮挡面积，选择以被遮挡区域较小的图像中物品的检测识别结果作为输出结果，降低了食材间相互遮挡对识别结果的影响，提高了冰箱在图像识别中的准确率。
应当理解，虽然本说明书按照实施方式加以描述，但并非每个实施方式仅包含一个独立的技术方案，说明书的这种叙述方式仅仅是为清楚起见，本领域技术人员应当将说明书作为一个整体，各实施方式中的技术方案也可以经适当组合，形成本领域技术人员可以理解的其他实施方式。
上文所列出的一系列的详细说明仅仅是针对本发明的可行性实施方式的具体说明，并非用以限制本发明的保护范围，凡未脱离本发明技艺精神所作的等效实施方式或变更均应包含在本发明的保护范围之内。

Claims (10)

  1. 一种冰箱内物品信息识别方法，其特征在于，包括步骤：
    获取多个不同角度拍摄的冰箱内储存空间图像；
    建立多幅图像之间的映射关系，根据所述映射关系，确认储存空间内同一物品分别在不同所述图像中的位置；
    分别于多幅图像中检测识别所述物品的类别信息；
    若同一所述物品在不同图像中类别信息识别结果一致，则判断该所述识别结果为最终输出信息；
    若同一物品在不同图像中类别信息识别结果不一致，则检测识别多幅图像中该所述物品被其他物品或冰箱结构所遮挡区域面积的大小，并以被遮挡区域面积最小的一幅图像中所检测识别的物品类别信息作为最终输出信息。
  2. 根据权利要求1所述的冰箱内物品信息识别方法，其特征在于，“建立多幅图像之间的映射关系”具体包括：
    提取多幅图像中的特征点；
    匹配特征点描述子，找到两张图中匹配的特征点对；
    计算变换对应变换关系矩阵。
  3. 根据权利要求2所述的冰箱内物品信息识别方法，其特征在于，“检测识别多幅图像中该所述物品未被其他物品或冰箱结构所遮挡区域面积的大小，并以为被遮挡区域面积最大的一幅图像中所检测识别的物品类别信息作为最终输出信息”具体包括：
    使用K-Means聚类算法对物品图像中的像素点进行聚类，将像素点划分为遮挡物区域像素点和物品区域像素点；
    计算多幅图像中物品区域像素点数量，并以数量最多的一幅图像中所检测识别的物品类别信息作为最终输出信息。
  4. 根据权利要求2所述的冰箱内物品信息识别方法，其特征在于，“检测识别多幅图像中该所述物品未被其他物品或冰箱结构所遮挡区域面积的大小，并以为被遮挡区域面积最大的一幅图像中所检测识别的物品类别信息作为最终输出信息”具体包括：
    使用边缘检测算法识别物品边缘轮廓线；
    计算多幅图像中物品区域像素点数量，并以像素点数量最多的一幅图像中所检测识别的物品类别信息作为最终输出信息。
  5. 根据权利要求1所述的冰箱内物品信息识别方法，其特征在于，“获取以至少两个不同角度拍摄的同一冰箱内储存空间的图像”具体包括：
    获取以两个不同角度所拍摄的冰箱瓶座内空间的鸟瞰图像，得到第一图像和第二图 像。
  6. 根据权利要求5所述的冰箱内物品信息识别方法，其特征在于，“获取以两个不同角度所拍摄的冰箱瓶座内空间的鸟瞰图像”具体包括：
    检测到冰箱门体运动至与冰箱箱体之间夹角呈预设角度；
    控制高于所需拍摄瓶座顶面且位于两处不同位置的摄像机分别拍摄冰箱瓶座内空间，得到所述第一图像和所述第二图像。
  7. 根据权利要求5所述的冰箱内物品信息识别方法，其特征在于，“获取以至少两个不同角度拍摄的同一冰箱内储存空间的图像”具体包括：
    控制摄像机在高于所需拍摄瓶座顶面的位置沿预设路径运动；
    分别在第一预设点位和第二设点位控制摄像机拍摄冰箱瓶座内空间得到所述第一图像和所述第二图像。
  8. 根据权利要求7所述的冰箱内物品信息识别方法，其特征在于，“分别于多幅图像中检测识别所述物品的类别信息”具体包括：
    识别所述第一图像和所述第二图像中冰箱瓶座内物品的顶面图像信息；
    根据所述顶面图像信息检测识别物品种类信息。
  9. 一种冰箱，其特征在于，
    所述冰箱于箱体顶部和瓶座底面中的一处或两处设有至少两个位于不同位置的摄像机；
    所述冰箱还包括存储器和处理器，所述存储器存储有可在所述处理器上运行的计算机程序，所述处理器执行所述程序时实现权利要求1-8中任意一项所述冰箱内物品信息识别方法的步骤。
  10. 一种冰箱，其特征在于，
    所述冰箱于箱体顶部和/或瓶座底面设有传动结构，摄像机沿所述传动结构运动；
    所述冰箱还包括存储器和处理器，所述存储器存储有可在所述处理器上运行的计算机程序，所述处理器执行所述程序时实现权利要求1-8中任意一项所述冰箱内物品信息识别方法的步骤。
PCT/CN2021/133222 2021-04-16 2021-11-25 冰箱内物品信息识别方法和冰箱 WO2022217935A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110412306.8A CN113095240B (zh) 2021-04-16 2021-04-16 冰箱内物品信息识别方法、冰箱和计算机存储介质
CN202110412306.8 2021-04-16

Publications (1)

Publication Number Publication Date
WO2022217935A1 true WO2022217935A1 (zh) 2022-10-20

Family

ID=76678350

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/133222 WO2022217935A1 (zh) 2021-04-16 2021-11-25 冰箱内物品信息识别方法和冰箱

Country Status (2)

Country Link
CN (1) CN113095240B (zh)
WO (1) WO2022217935A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113095240B (zh) * 2021-04-16 2023-08-29 青岛海尔电冰箱有限公司 冰箱内物品信息识别方法、冰箱和计算机存储介质
CN114882349B (zh) * 2022-03-29 2024-05-24 青岛海尔制冷电器有限公司 冰箱内物品目标同一性判断方法、冰箱和计算机存储介质
CN114708503A (zh) * 2022-03-31 2022-07-05 青岛海尔电冰箱有限公司 冰箱瓶座内物品信息识别方法、冰箱和计算机存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014168265A1 (ko) * 2013-04-10 2014-10-16 엘지전자 주식회사 영상 인식을 이용한 냉장고의 보관 품목 관리 방법 및 그 냉장고
CN108615015A (zh) * 2018-04-28 2018-10-02 海信（山东）冰箱有限公司 一种冰箱内部食材的识别方法、设备及图像识别系统
CN111553915A (zh) * 2020-05-08 2020-08-18 深圳前海微众银行股份有限公司 物品识别检测方法、装置、设备和可读存储介质
CN111666792A (zh) * 2019-03-07 2020-09-15 阿里巴巴集团控股有限公司 图像识别方法、图像采集和识别方法以及商品识别方法
CN112435223A (zh) * 2020-11-11 2021-03-02 马鞍山市瀚海云星科技有限责任公司 目标检测方法、装置及存储介质
CN113095240A (zh) * 2021-04-16 2021-07-09 青岛海尔电冰箱有限公司 冰箱内物品信息识别方法、冰箱和计算机存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107025683A (zh) * 2017-03-30 2017-08-08 联想(北京)有限公司 一种信息处理方法及电子设备
US11030768B2 (en) * 2018-10-30 2021-06-08 Ncr Corporation Image processing for occluded item recognition
CN110119726B (zh) * 2019-05-20 2023-04-25 四川九洲视讯科技有限责任公司 一种基于YOLOv3模型的车辆品牌多角度识别方法
CN112149465A (zh) * 2019-06-27 2020-12-29 海信（山东）冰箱有限公司 一种物品识别方法及设备
CN112006525B (zh) * 2020-08-11 2022-06-03 杭州九阳小家电有限公司 一种烹饪设备内食物的焦糊检测方法及烹饪设备
CN112052878B (zh) * 2020-08-11 2024-04-16 浙江大华技术股份有限公司 一种雷达的遮挡识别的方法、装置及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014168265A1 (ko) * 2013-04-10 2014-10-16 엘지전자 주식회사 영상 인식을 이용한 냉장고의 보관 품목 관리 방법 및 그 냉장고
CN108615015A (zh) * 2018-04-28 2018-10-02 海信（山东）冰箱有限公司 一种冰箱内部食材的识别方法、设备及图像识别系统
CN111666792A (zh) * 2019-03-07 2020-09-15 阿里巴巴集团控股有限公司 图像识别方法、图像采集和识别方法以及商品识别方法
CN111553915A (zh) * 2020-05-08 2020-08-18 深圳前海微众银行股份有限公司 物品识别检测方法、装置、设备和可读存储介质
CN112435223A (zh) * 2020-11-11 2021-03-02 马鞍山市瀚海云星科技有限责任公司 目标检测方法、装置及存储介质
CN113095240A (zh) * 2021-04-16 2021-07-09 青岛海尔电冰箱有限公司 冰箱内物品信息识别方法、冰箱和计算机存储介质

Also Published As

Publication number Publication date
CN113095240B (zh) 2023-08-29
CN113095240A (zh) 2021-07-09

Similar Documents

Publication Publication Date Title
WO2022217935A1 (zh) 冰箱内物品信息识别方法和冰箱
US9418317B2 (en) End-to end visual recognition system and methods
Seo et al. Nonparametric bottom-up saliency detection by self-resemblance
WO2022217928A1 (zh) 冰箱内物品信息标识方法、冰箱和计算机存储介质
CN109670390B (zh) 活体面部识别方法与系统
WO2022217927A1 (zh) 冰箱内物品信息标识方法、冰箱和计算机存储介质
Susanto et al. 3d object detection with multiple kinects
JP6351243B2 (ja) 画像処理装置、画像処理方法
Bustard et al. Robust 2D ear registration and recognition based on SIFT point matching
CN102004899A (zh) 一种人脸认证系统及方法
CA3115612A1 (en) Animal identification using vision techniques
US9947106B2 (en) Method and electronic device for object tracking in a light-field capture
Hu et al. Posture recognition with a top-view camera
Lakshmi et al. Pedestrian detection in thermal images: An automated scale based region extraction with curvelet space validation
WO2023185779A1 (zh) 冰箱内物品信息识别方法
CN109583296A (zh) 一种防止误检测方法、装置、系统及计算机存储介质
WO2023185280A1 (zh) 冰箱瓶座内物品信息识别方法和冰箱
Yu et al. An eye detection and localization system for natural human and robot interaction without face detection
Bo et al. PhD forum: Illumination-robust foreground detection for multi-camera occupancy mapping
CN113128320B (zh) 基于tof相机的人脸活体检测方法、检测装置及电子设备
WO2022160085A1 (zh) 控制方法、电子设备及存储介质
El-Barkouky et al. Selective part models for detecting partially occluded faces in the wild
TW202103043A (zh) 臉部辨識裝置
Weng et al. Visual knowledge transfer among multiple cameras for people counting with occlusion handling
WO2023109151A1 (zh) 冰箱内物品信息识别方法和冰箱

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21936801

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21936801

Country of ref document: EP

Kind code of ref document: A1