CN111462190B - Intelligent refrigerator and food material input method - Google Patents
Intelligent refrigerator and food material input method Download PDFInfo
- Publication number
- CN111462190B CN111462190B CN202010310655.4A CN202010310655A CN111462190B CN 111462190 B CN111462190 B CN 111462190B CN 202010310655 A CN202010310655 A CN 202010310655A CN 111462190 B CN111462190 B CN 111462190B
- Authority
- CN
- China
- Prior art keywords
- detected
- image
- images
- preset
- frames
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 235000013305 food Nutrition 0.000 title claims abstract description 74
- 239000000463 material Substances 0.000 title claims abstract description 50
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000001514 detection method Methods 0.000 claims abstract description 39
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 14
- 230000033001 locomotion Effects 0.000 claims description 73
- 238000012216 screening Methods 0.000 claims description 13
- 239000013598 vector Substances 0.000 claims description 10
- 230000008569 process Effects 0.000 abstract description 13
- 238000010586 diagram Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 10
- 238000012545 processing Methods 0.000 description 9
- 238000004590 computer program Methods 0.000 description 7
- 230000003068 static effect Effects 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 235000002595 Solanum tuberosum Nutrition 0.000 description 2
- 244000061456 Solanum tuberosum Species 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000009920 food preservation Methods 0.000 description 1
- 238000012905 input function Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000033764 rhythmic process Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D29/00—Arrangement or mounting of control or safety devices
Abstract
The disclosure relates to an intelligent refrigerator and a food material input method, aiming at the problem of low food material input efficiency caused by complex logic implementation of an algorithm in the prior art, the method comprises the following steps: when the intelligent refrigerator determines that the door is opened, an image to be detected, which is acquired by the camera, is acquired, then, a moving target and the moving direction of the moving target exist in a preset detection area are determined, then, the key M frames of images to be detected are sent to the server, the server is enabled to identify the moving target, and then, the intelligent refrigerator determines the food storage and retrieval result based on the identification result fed back by the server and the moving direction. In this way, in the present disclosure, the food material inputting process is avoided occupying too much computing resources, the robustness is improved, in addition, only part of the key images to be detected are sent to the server, the food material inputting efficiency and accuracy are improved, and thus the user experience is improved.
Description
Technical Field
The disclosure relates to the technical field of household appliances, in particular to an intelligent refrigerator and a food material input method.
Background
Along with the continuous development of technology and the increasing speed of social rhythm, the intellectualization of home appliances is also more and more important by various home appliance manufacturers. The intelligent refrigerator is used as one of core household appliances, and besides the food material fresh-keeping function, other related functions are continuously perfected and expanded. The most basic food input function is one of key technologies of the intelligent refrigerator, wherein the key technologies have more powerful functions and are more friendly to user experience.
In the prior art, methods such as application program input, radio frequency identification (Radio Frequency Identification, RFID) tag clip identification, voice identification, image identification and the like are generally adopted to realize food material input of the intelligent refrigerator.
However, although the food material input method of application program input, RFID tag clip recognition and voice recognition can ensure extremely high accuracy, the three methods need to manually input related information or instructions, so that the user experience is poor.
The food material input method based on image recognition mainly comprises two implementation schemes, namely a static scheme and a dynamic scheme. In the static scheme, after the intelligent refrigerator triggers the door opening and closing operation every time, the inside of the refrigerator is shot through a camera which is preset, then, the intelligent refrigerator sends shot pictures to a cloud server, and the cloud server is enabled to conduct food material identification. In the dynamic scheme, the shooting visual field of a camera of the intelligent refrigerator is mainly near a refrigerator door body, when the intelligent refrigerator determines to trigger a door opening operation, the user can determine food entering or taking out through the hand gesture and the hand movement condition shot by the camera, and then the intelligent refrigerator detects the type of the food according to the food picture in the storage and taking process.
Obviously, for the static scheme, the problem that the identification accuracy is low because food materials shielded in the intelligent refrigerator cannot be identified exists, meanwhile, the static scheme is a non-real-time processing process, a large identification error exists, in addition, a plurality of color cameras are usually required to be deployed in the static scheme, and the hardware cost is high.
The dynamic scheme is used as a newly proposed scheme, so that the shielding problem can be well solved, however, certain user cooperation is needed, for example, the food cannot be shielded by the hands too much, meanwhile, as the dynamic scheme relates to a plurality of technologies such as hand detection, target tracking, position detection, food detection and the like, the identification process is too complex, the calculated amount is large, the robustness is low, in addition, a depth camera is needed to be deployed in the dynamic scheme, and the hardware cost is high.
It follows that there is a need to devise a new solution to overcome the above drawbacks.
Disclosure of Invention
The disclosure provides an intelligent refrigerator and a food material input method, which are used for solving the problem of low food material input efficiency caused by complex algorithm realization logic and overlarge calculated amount in the prior art.
The specific technical scheme provided by the embodiment of the disclosure is as follows:
in a first aspect, a smart refrigerator includes:
a housing including a storage chamber, the storage chamber having an opening;
the door is movably connected with the shell and is used for shielding the opening;
the camera is connected with the shell and used for collecting images;
a processor configured to:
when the door is determined to be opened, acquiring N frames of images to be detected acquired by the camera;
determining that a moving object exists in a preset detection area in the N frames of images to be detected based on preset moving object screening conditions, and the moving direction of the moving object in each frame of images to be detected;
determining continuous M frames of images to be detected, of which the moving target keeps the same moving direction, in the N frames of images to be detected;
when the value of M is determined to reach a preset image recognition threshold, sending the M frames of images to be detected to a server, and enabling the server to determine a recognition result of the moving object based on the M frames of images to be detected;
and when the identification result fed back by the server is received, determining a food access result based on the identification result and the movement direction of the M frames of images to be detected, and storing the food access result into a preset database.
In a second aspect, a food material input method includes:
when the door is determined to be opened, acquiring N frames of images to be detected acquired by the camera;
determining that a moving object exists in a preset detection area in the N frames of images to be detected based on preset moving object screening conditions, and the moving direction of the moving object in each frame of images to be detected;
determining continuous M frames of images to be detected, of which the moving target keeps the same moving direction, in the N frames of images to be detected;
when the value of M is determined to reach a preset image recognition threshold, sending the M frames of images to be detected to a server, and enabling the server to determine a recognition result of the moving object based on the M frames of images to be detected;
and when the identification result fed back by the server is received, determining a food access result based on the identification result and the movement direction of the M frames of images to be detected, and storing the food access result into a preset database.
In the method, when the intelligent refrigerator determines that a door is opened, an image to be detected acquired by a camera is acquired, then, a moving object is determined to exist in a preset detection area, and the moving direction of the moving object in each frame of image to be detected is determined, then, continuous M frames of images to be detected, which are kept in the same moving direction, of the moving object are determined, when the value of M reaches a preset image recognition threshold, the M frames of images to be detected are transmitted to a server, the server is enabled to recognize the moving object, and then, based on a recognition result fed back by the server and the moving direction of the M frames of images to be detected, a food material access result is determined, and the food material access result is stored.
Thus, the present disclosure has at least the following beneficial effects:
after the moving object and the moving direction of the moving object are determined, the intelligent refrigerator sends the corresponding image to be detected to the server for moving object identification, so that the food material inputting process is simplified, the calculated amount of the intelligent refrigerator is reduced, excessive calculation resources occupied by the food material inputting process are avoided, the robustness is improved, in addition, only part of key images to be detected are sent to the server, the data transmission efficiency is improved, the food material inputting efficiency and accuracy are improved, the user experience is improved, and meanwhile, the cost of the intelligent refrigerator is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or the solutions in the prior art, the drawings that are required for the description of the embodiments or the prior art will be briefly described, it being apparent that the drawings in the following description are only some embodiments of the present invention and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1A is a schematic illustration of a scenario provided in an embodiment of the present disclosure;
fig. 1B is a schematic structural diagram of an intelligent refrigerator according to an embodiment of the disclosure;
fig. 2A-2C are schematic structural diagrams of three intelligent refrigerators provided in an embodiment of the present disclosure;
fig. 3 is a schematic flow chart of a food material input method provided in an embodiment of the disclosure;
fig. 4A is a schematic perspective view of a smart refrigerator according to an embodiment of the present disclosure;
fig. 4B is a top view of a smart refrigerator provided in an embodiment of the present disclosure;
FIGS. 5A-5C are schematic illustrations of a set of food access provided in embodiments of the present disclosure;
fig. 6 is a flowchart of a moving object determining method provided in an embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of an intelligent refrigerator according to an embodiment of the disclosure.
Detailed Description
In order to better understand the technical solutions in the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only some embodiments of the present disclosure, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without inventive effort, based on the embodiments in this disclosure, shall fall within the scope of the present disclosure.
In the description of the present disclosure, it should be understood that the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like indicate orientations or positional relationships based on the orientation or positional relationships shown in the drawings, merely to facilitate description of the present disclosure and simplify the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present disclosure.
The terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present disclosure, unless otherwise indicated, the meaning of "a plurality" is two or more.
In the description of the present disclosure, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the terms in this disclosure will be understood by those of ordinary skill in the art in the specific context.
Aiming at the problems of low food material input efficiency caused by complex logic realization and overlarge calculated amount of an algorithm in the prior art, in order to simplify the calculation process and improve the food material input efficiency, in the embodiment of the disclosure, an intelligent refrigerator and a food material input method are provided. It should be noted that, the method provided by the embodiment is not only used with an intelligent refrigerator, but also suitable for equipment such as an intelligent refrigerator.
Referring to fig. 1A, an application scenario diagram of an intelligent refrigerator according to some embodiments of the present disclosure is shown.
The intelligent refrigerator 100 may provide functions including, but not limited to, food preservation and food entry. The specific type, size, power, etc. of the intelligent refrigerator are not limited.
The intelligent refrigerator 100 also communicates data with the server 200 through various communication means. The intelligent refrigerator 100 may be allowed to communicate via a local area network (Local Area Network, LAN), a wireless local area network (Wireless Local Area Network, WLAN) and other networks. The server 200 may provide various contents and interactions to the smart refrigerator 100. By way of example, the intelligent refrigerator 100 may send and receive information such as: receiving moving object identification data, accessing a remotely stored digital media library, and transmitting image data to be detected. The servers 200 may be one group, may be multiple groups, and may be one or more types of servers. The server 200 can be deployed locally or at the cloud, and functions such as image recognition are realized through the server 200.
In some embodiments, as shown in fig. 1B, the intelligent refrigerator 100 includes at least a cabinet 110, a door 120, a camera 130, and a processor 140, wherein,
a cabinet 110 including a storage chamber having an opening;
the door 120 is movably connected with the casing 110 and is used for shielding the opening;
the camera 130 is connected with the shell 110 and is used for collecting images;
a processor configured to:
when the door 120 is determined to be opened, acquiring N frames of images to be detected acquired by the camera 130;
determining that a moving object exists in a preset detection area in N frames of images to be detected based on preset moving object screening conditions, and the moving direction of the moving object in each frame of images to be detected;
determining continuous M frames of images to be detected, wherein the continuous M frames of images to be detected are kept in the same motion direction by the moving object;
when the value of M is determined to reach a preset image recognition threshold, sending M frames of images to be detected to the server 200, and enabling the server 200 to determine a recognition result of the moving object based on the M frames of images to be detected;
and when receiving the recognition result fed back by the server 200, determining a food access result based on the recognition result and the movement direction of the M frames of images to be detected, and storing the food access result into a preset database.
The processor may be disposed in the housing 110 or may be disposed in the door 120, which is not limited in this disclosure.
The camera 130 is a color camera, for example, a three primary color (RGB) color mode camera. In some embodiments, to avoid blurring of images during food access, camera 130 is a high frame rate camera, e.g., 120 display frames per second (Frames per Second, FPS), 60FPS, 48FPS, etc.
In some embodiments, referring to fig. 2A-2C, in order to enable the camera to obtain a better shooting view, and reduce the shielding of objects such as a smart refrigerator and a human body from the shooting view, the camera 130 is disposed on the top of the housing 110, for example, the camera 130 is disposed in the center of the top of the housing 110, the camera 130 is disposed on the left side of the top of the housing 110, and the camera 130 is disposed on the right side of the top of the housing 110.
In other embodiments, the camera 130 may be a plurality of cameras, and specifically, the number of cameras used may be determined according to the size of the space to be managed in the intelligent refrigerator and the shooting range of the cameras. If there are multiple cameras 130 in the intelligent refrigerator, image stitching can be performed according to the images collected by each camera, so as to obtain an image to be detected. The process of image stitching of the images acquired by the plurality of cameras is in the prior art, and in the embodiment of the present disclosure, the process is not described in detail.
For convenience of description, only one camera 130 will be described hereinafter as an example.
Referring to fig. 2A, the intelligent refrigerator 100 includes a cabinet 110, a door 120, a camera 130, and a processor, wherein the camera 130 is disposed at a top center of the cabinet 110 for capturing images.
In practical application, when the user opens the intelligent refrigerator, food materials can be stored in the intelligent refrigerator or taken out, so that in order to facilitate management of the food materials, the processor determines that the door 120 is closed and opened, and then the food materials are required to be recorded according to the image acquired by the camera 130. Specifically, when the processor determines that the door 120 is closed and opened, the camera 130 is controlled to start shooting, and an image to be detected is acquired.
The processor acquires an image to be detected acquired by the camera 130, determines a moving object existing in the image to be detected based on the image to be detected, and sends the image to be detected containing the moving object to the server 200, so that the server 200 determines a recognition result of the moving object based on a preset image recognition model, and then the processor determines a food access result according to the recognition result fed back by the server 200 and the moving direction of the moving object, thereby inputting the food access result.
Referring to fig. 3, in the embodiment of the disclosure, a specific flow of food material input is as follows.
Step S301: when the intelligent refrigerator determines that the door is opened, N frames of images to be detected, which are acquired by the camera, are acquired.
In the embodiment of the disclosure, in order to protect privacy of a user, reduce power consumption of a product and prolong the service time of a camera, the camera shoots when a door is opened and stops shooting when the door is closed.
Specifically, in the embodiment of the disclosure, the intelligent refrigerator judges whether the door is opened or not according to the logic level of the preset switch.
For example, when the smart refrigerator determines that the preset switch is at a high level, it is determined that the door 120 is opened.
When the intelligent refrigerator determines that the door is opened, the camera is started to shoot, and a complete image to be detected acquired by the camera is acquired.
For example, referring to fig. 2A, when the intelligent refrigerator determines that the door 120 is opened, the camera 130 is started to shoot, and a complete image to be detected of 120 frames acquired by the camera 130 is acquired.
In the embodiment of the disclosure, because the number of frames of the image acquired in each second is increased under the high frame rate camera, in order to ensure the algorithm processing speed and realize real-time moving target recognition, the complete image to be detected is downsampled.
Specifically, the intelligent refrigerator downsamples the complete image to be detected based on a preset downsampling ratio to obtain an X-frame image to be detected.
In the disclosed embodiments, the downsampling ratio is dependent on the resolution of the camera, and the intelligent refrigerator specifications. In general, the higher the resolution of the camera, the clearer the acquired image to be detected, and therefore, the higher the resolution of the camera, the greater the downsampling ratio. The larger the intelligent refrigerator specification is, the higher the erection position of the camera is, and the smaller the moving target is in the acquired image to be detected, so that in order to ensure the identification accuracy, the larger the intelligent refrigerator specification is, the smaller the downsampling ratio is.
For example, assuming that the preset downsampling ratio is 50%, the intelligent refrigerator downsamples 120 frames of complete images to be detected based on the preset downsampling ratio to obtain 60 frames of images to be detected.
Further, the intelligent refrigerator determines N frames of images to be detected in the X frames of images to be detected based on the downsampling ratio and the frame rate of the camera.
In the embodiment of the disclosure, if the downsampling ratio is too large, subsequent moving object recognition is directly performed on the X-frame image to be detected, that is, N, X has the same value, and if the frame rate of the camera is too high, N-frame images to be detected are selected from the X-frame images to be detected, and subsequent moving object recognition is performed, that is, N, X has different values.
For example, assuming that the preset downsampling ratio is 50%, the frame rate of the camera 130 is 60FPS, the intelligent refrigerator determines 30 frames of images to be detected among 60 frames of images to be detected based on the downsampling ratio of 50%, and the frame rate 60FPS of the camera 130.
For another example, assuming that the preset downsampling ratio is 50%, the frame rate of the camera 130 is 24FPS, the intelligent refrigerator determines 60 frames of images to be detected among 60 frames of images to be detected based on the downsampling ratio of 50%, and the frame rate of 24FPS of the camera 130, that is, directly performs subsequent moving object recognition on the 60 frames of images to be detected.
Step S302: the intelligent refrigerator determines that a moving object exists in a preset detection area in N frames of images to be detected based on preset moving object screening conditions, and the moving direction of the moving object in each frame of images to be detected.
In the embodiment of the disclosure, in order to simplify the working area and reduce the calculation amount, a detection area is preset in the intelligent refrigerator. The setting of the detection area is determined according to the installation position and the erection angle of the camera.
For example, in the smart refrigerator, a detection area shown in fig. 4A and 4B is preset, wherein fig. 4A is a schematic perspective view of the smart refrigerator, and fig. 4B is a plan view of the smart refrigerator, and the detection area is set as a spatial three-dimensional area extending a certain length in the forward and backward directions with the front side (i.e., the side facing the user) of the smart refrigerator as the center when the door 120 is opened. The detection area is mapped onto an image to be detected acquired by a camera, the detection area is shown in fig. 5A, the side projection facing a user is taken as the center, a certain distance is respectively extended to the up-down direction of the image to be used as the longitudinal height of the detection area, the edges on two sides are cut off, and the residual distance is used as the transverse width of the detection area.
After the intelligent refrigerator determines N frames of images to be detected, referring to fig. 6, starting from the second frame of images to be detected, for each frame of images to be detected, the following method may be used to determine a moving target, but is not limited to:
step S3021: the intelligent refrigerator determines a plurality of motion pixel points of an image to be detected of a current frame relative to an image to be detected of a previous frame in a preset detection area based on a preset feature matching algorithm, and determines motion vectors of the plurality of motion pixel points.
In the embodiment of the disclosure, the preset feature matching algorithm may adopt, but is not limited to, a rapid feature point extraction and description (Oriented Fast and Rotated BRIEF, ORB) algorithm, an accelerated robust feature (Speeded Up Robust Features, SURF) algorithm, a Scale-invariant feature transform (Scale-invariant feature transform, SIFT) algorithm, and the like. Hereinafter, an ORB algorithm will be described as an example.
Specifically, the intelligent refrigerator determines all characteristic pixel points of an image to be detected of a current frame in a preset detection area based on image characteristics.
For example, referring to fig. 5B, the intelligent refrigerator determines all feature pixels of the image to be detected in the current frame as a in a preset detection area based on the gray level of the image 1 、a 2 、a 3 、a 4 、a 5 、a 6 、a 7 、a 8 、a 9 、a 10 、a 11 。
Further, the intelligent refrigerator establishes corresponding feature descriptors aiming at all feature pixel points, wherein the feature descriptors are used for describing image features of one feature pixel point.
For example, with characteristic pixel point a 1 For example, the intelligent refrigerator aims at the characteristic pixel point a 1 Establishing a characteristic pixel point a 1 Is a feature descriptor 10101011 (binary data).
And the intelligent refrigerator compares the feature descriptors of all feature pixels of the current frame to-be-detected image with the feature descriptors of all feature pixels of the previous frame to-be-detected image, screens out a plurality of feature pixels with the similarity reaching a preset similarity threshold value from all feature pixels of the current frame to-be-detected image, takes the feature pixels as a plurality of motion pixels of the current frame to-be-detected image relative to the previous frame to-be-detected image, and determines motion vectors of the motion pixels.
For example, assuming that the preset similarity threshold is 90%, the intelligent refrigerator compares feature descriptors of all feature pixels of the image to be detected of the current frame with feature descriptors of all feature pixels of the image to be detected of the previous frame, and screens out feature pixels a with similarity reaching 90% from all feature pixels of the image to be detected of the current frame 1 、a 2 、a 3 、a 4 、a 5 、a 6 、a 7 、a 8 、a 9 、a 10 、a 11 As the current frame to-be-detected image relative to the previous frame to-be-detected imageThe motion pixel point of the image is then used as the motion pixel point a 1 、a 2 、a 3 、a 4 For example, determine a 1 、a 2 、a 3 、a 4 The motion vectors of (1, 17), (-1, -60), (-4, 9), (-8, 8) in that order.
Step S3022: the intelligent refrigerator screens out a plurality of target pixel points with the movement distance meeting a preset effective numerical range from the plurality of movement pixel points based on the movement vectors of the plurality of movement pixel points.
For example, assume that the preset effective value range is 5-30, and the intelligent refrigerator is based on a 1 、a 2 、a 3 、a 4 、a 5 、a 6 、a 7 、a 8 、a 9 、a 10 、a 11 Screening a plurality of target pixel points a with the motion distance of 5-30 from a plurality of motion pixel points 3 、a 4 、a 5 、a 6 、a 7 、a 8 、a 9 、a 10 、a 11 。
Step S3023: when the intelligent refrigerator determines that the number of the plurality of target pixel points exceeds a preset target composition threshold value, determining that a moving target exists in a preset detection area in an image to be detected of a current frame, wherein the moving target is composed of the plurality of target pixel points.
It should be noted that, in the embodiment of the present disclosure, the target composition threshold is set according to the resolution of the camera and the feature matching algorithm.
For example, assuming that the preset target composition threshold is 7, the intelligent refrigerator determines that the number of a plurality of target pixels is 9, and when the target composition threshold is exceeded 7, it determines that a moving target exists in the image to be detected of the current frame in the detection area, wherein the moving target is formed by a target pixel a 3 、a 4 、a 5 、a 6 、a 7 、a 8 、a 9 、a 10 、a 11 The composition is formed.
Step S3024: the intelligent refrigerator determines the motion direction of a moving target in an image to be detected of a current frame based on the motion directions of a plurality of target pixel points.
Specifically, the intelligent refrigerator determines the number of target pixel points corresponding to each type of motion direction based on the motion directions of a plurality of target pixel points, and takes the motion direction with the largest number of corresponding target pixel points as the motion direction of the motion target in the image to be detected of the current frame.
For example, referring to FIG. 5B, a 3 、a 5 、a 6 、a 7 、a 8 、a 9 、a 10 、a 11 Is moved inwards, a 4 The movement direction of the intelligent refrigerator is outward movement, and the intelligent refrigerator is based on the target pixel point a 3 、a 4 、a 5 、a 6 、a 7 、a 8 、a 9 、a 10 、a 11 The number of the target pixel points corresponding to the motion direction representing inward motion is 8, the number of the target pixel points corresponding to the motion direction representing outward motion is 1, and the motion direction representing inward motion is taken as the motion direction of the motion target in the image to be detected of the current frame.
Step S303: and determining continuous M frames of images to be detected of which the moving targets keep the same moving direction in the N frames of images to be detected by the intelligent refrigerator.
For example, the intelligent refrigerator determines, among 30 frames of images to be detected, 10 consecutive frames of images to be detected, in which the moving object maintains the same moving direction, wherein the moving direction of the 10 consecutive frames of images to be detected is inward movement.
Step S304: when the intelligent refrigerator determines that the value of M reaches a preset image recognition threshold, the intelligent refrigerator sends M frames of images to be detected to the server, so that the server determines a recognition result of the moving object based on the M frames of images to be detected.
In the embodiment of the disclosure, in order to avoid excessive computing resources occupied in the process of identifying the moving object, a server adopts a preset image identification model to identify the moving object, wherein the server can be deployed locally or at a cloud end, and the disclosure is not limited to this.
For example, assuming that the preset image recognition threshold is 10, when the intelligent refrigerator determines that the value of M reaches 10, sending 10 frames of images to be detected to the cloud server, and enabling the cloud server to determine the recognition result of the moving object by adopting a preset image recognition model based on the 10 frames of images to be detected.
In other embodiments, in order to reduce transmission delay and improve recognition efficiency, the intelligent refrigerator intercepts a detection area of an M-frame image to be detected, sends the intercepted M-frame image to be detected to a server, and enables the server to determine a recognition result of a moving object based on the intercepted M-frame image to be detected.
For example, referring to fig. 5C, the intelligent refrigerator intercepts a detection area of 10 frames of images to be detected, sends the intercepted 10 frames of images to be detected to a local server, and makes the local server determine a recognition result of a moving object based on the intercepted 10 frames of images to be detected.
Step S305: when the intelligent refrigerator receives the identification result fed back by the server, based on the identification result and the movement direction of the M frames of images to be detected, determining a food material access result, and storing the food material access result into a preset database.
Specifically, the intelligent refrigerator determines the food material access result based on the identification result and the movement direction of the M frames of images to be detected, and the food material access result is shown in table 1.
Table 1 food access results
| Sequence number | Recognition result | Direction of movement | Food material access result |
| 1 | Food material type number | Inward movement of | Storing certain food materials in intelligent refrigerator |
| 2 | Food material type number | Outward movement of | Taking out certain food from intelligent refrigerator |
| 3 | No food material is present | Inward movement of | Invalidation operation |
| 4 | No food material is present | Outward movement of | Invalidation operation |
For example, when the intelligent refrigerator receives the recognition result fed back by the cloud server, it is assumed that the recognition result carries a number representing the potato, based on the recognition result and a movement direction representing 10 frames of images to be detected moving inwards, a food material access result is determined to be that the potato is stored in the intelligent refrigerator, and the food material access result is stored in a preset database.
Based on the same inventive concept, in an embodiment of the present disclosure, an intelligent refrigerator is provided, as shown in fig. 7, including at least: an acquisition unit 701, a detection unit 702, a processing unit 703, an identification unit 704 and a determination unit 705, wherein,
the acquisition unit 701 is used for acquiring N frames of images to be detected acquired by the camera when the door is determined to be opened;
the detecting unit 702 is configured to determine, based on a preset moving object screening condition, that a moving object exists in a preset detection area in the N frames of images to be detected, and a moving direction of the moving object in each frame of images to be detected;
a processing unit 703, configured to determine, among the N frames of images to be detected, consecutive M frames of images to be detected in which the moving object maintains the same moving direction;
the identifying unit 704 is configured to send the M frame image to be detected to a server when determining that the value of M reaches a preset image identifying threshold, so that the server determines an identifying result of the moving object based on the M frame image to be detected;
and the determining unit 705 is configured to determine a food material access result based on the identification result and the motion direction of the M frames of images to be detected when the identification result fed back by the server is received, and store the food material access result in a preset database.
The acquisition unit 701, the detection unit 702, the processing unit 703, the identification unit 704 and the determination unit 705 cooperate with each other to realize the functions of the intelligent refrigerator in the above-described respective embodiments.
Based on the same inventive concept, the embodiments of the present disclosure provide a storage medium, which when instructions in the storage medium are executed by a processor, enable the processor to perform any one of the food material entry methods implemented by the intelligent refrigerator in the above-described flow.
In the method, when the intelligent refrigerator determines that a door is opened, an image to be detected acquired by a camera is acquired, then, a moving object is determined to exist in a preset detection area, and the moving direction of the moving object in each frame of image to be detected is determined, then, continuous M frames of images to be detected, which are kept in the same moving direction, of the moving object are determined, when the value of M reaches a preset image recognition threshold, the M frames of images to be detected are transmitted to a server, the server is enabled to recognize the moving object, and then, based on a recognition result fed back by the server and the moving direction of the M frames of images to be detected, a food material access result is determined, and the food material access result is stored.
Thus, the present disclosure has at least the following beneficial effects:
after the moving object and the moving direction of the moving object are determined, the intelligent refrigerator sends the corresponding image to be detected to the server for moving object identification, so that the food material inputting process is simplified, the calculated amount of the intelligent refrigerator is reduced, excessive calculation resources occupied by the food material inputting process are avoided, the robustness is improved, in addition, only part of key images to be detected are sent to the server, the data transmission efficiency is improved, the food material inputting efficiency and accuracy are improved, the user experience is improved, and meanwhile, the cost of the intelligent refrigerator is reduced.
For system/device embodiments, the description is relatively simple as it is substantially similar to method embodiments, with reference to the description of method embodiments in part.
It will be apparent to those skilled in the art that embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While the preferred embodiments of the present disclosure have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the disclosure.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present disclosure without departing from the spirit or scope of the disclosure. Thus, the present disclosure is intended to include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (8)
1. An intelligent refrigerator, comprising:
a housing including a storage chamber, the storage chamber having an opening;
the door is movably connected with the shell and is used for shielding the opening;
the camera is connected with the shell and used for collecting images;
a processor configured to:
when the door is determined to be opened, acquiring a complete image to be detected, which is acquired by a camera; if the frame rate of the camera exceeds the preset frame rate, acquiring an image to be detected from the complete image to be detected by adopting a preset downsampling ratio; if the frame rate of the camera does not exceed the preset frame rate, the complete image to be detected is taken as an image to be detected;
determining that a moving object exists in a preset detection area in N frames of images to be detected based on preset moving object screening conditions, and the moving direction of the moving object in each frame of images to be detected;
determining continuous M frames of images to be detected, of which the moving target keeps the same moving direction, in the N frames of images to be detected;
when the value of M is determined to reach a preset image recognition threshold, sending the M frames of images to be detected to a server, and enabling the server to determine a recognition result of the moving object based on the M frames of images to be detected;
and when the identification result fed back by the server is received, determining a food access result based on the identification result and the movement direction of the M frames of images to be detected, and storing the food access result into a preset database.
2. The intelligent refrigerator according to claim 1, wherein the camera is a color camera.
3. The intelligent refrigerator according to claim 1, wherein the camera is disposed at a top of the cabinet.
4. The intelligent refrigerator of any one of claims 1-3, wherein, based on a preset moving object screening condition, determining that a moving object exists in a preset detection area in the N frames of images to be detected, and a moving direction of the moving object in each frame of images to be detected, the processor is further configured to:
starting from the second frame of image to be detected, for each frame of image to be detected, performing the following operations:
based on a preset feature matching algorithm, determining a plurality of motion pixel points of an image to be detected of a current frame relative to an image to be detected of a previous frame in a preset detection area, and determining motion vectors of the plurality of motion pixel points;
screening a plurality of target pixel points with the motion distance meeting a preset effective numerical range from the plurality of motion pixel points based on the motion vectors of the plurality of motion pixel points;
when the number of the plurality of target pixel points exceeds a preset target composition threshold, determining that a moving target exists in a preset detection area in the image to be detected of the current frame, wherein the moving target is composed of the plurality of target pixel points;
and determining the motion direction of the moving target in the image to be detected of the current frame based on the motion directions of the target pixel points.
5. The intelligent refrigerator of claim 4, wherein, based on a preset feature matching algorithm, in a preset detection area, determining a plurality of motion pixels of a current frame of an image to be detected relative to a previous frame of the image to be detected, and determining motion vectors of the plurality of motion pixels, the processor is further configured to:
determining all characteristic pixel points of an image to be detected of a current frame in a preset detection area based on image characteristics;
corresponding feature descriptors are established for all feature pixel points, and the feature descriptors are used for describing image features of one feature pixel point;
comparing the feature descriptors of all feature pixels of the image to be detected of the current frame with the feature descriptors of all feature pixels of the image to be detected of the previous frame, screening out a plurality of feature pixels with similarity reaching a preset similarity threshold value from all feature pixels of the image to be detected of the current frame, taking the feature descriptors as a plurality of motion pixels of the image to be detected of the current frame relative to the image to be detected of the previous frame, and determining motion vectors of the motion pixels.
6. The intelligent refrigerator of claim 5, wherein the M frames of images to be detected are sent to a server, and the processor is further configured to, when the server determines the recognition result of the moving object based on the M frames of images to be detected:
intercepting the detection area of the M frames of images to be detected, sending the intercepted M frames of images to be detected to a server, and enabling the server to determine the identification result of the moving object based on the intercepted M frames of images to be detected.
7. A method of food material entry comprising:
when the door is determined to be opened, acquiring a complete image to be detected acquired by a camera; if the frame rate of the camera exceeds the preset frame rate, acquiring an image to be detected from the complete image to be detected by adopting a preset downsampling ratio; if the frame rate of the camera does not exceed the preset frame rate, the complete image to be detected is taken as an image to be detected;
determining that a moving object exists in a preset detection area in N frames of images to be detected based on preset moving object screening conditions, and the moving direction of the moving object in each frame of images to be detected;
determining continuous M frames of images to be detected, of which the moving target keeps the same moving direction, in the N frames of images to be detected;
when the value of M is determined to reach a preset image recognition threshold, sending the M frames of images to be detected to a server, and enabling the server to determine a recognition result of the moving object based on the M frames of images to be detected;
and when the identification result fed back by the server is received, determining a food access result based on the identification result and the movement direction of the M frames of images to be detected, and storing the food access result into a preset database.
8. The method of claim 7, wherein determining that a moving object exists in the N frames of images to be detected in a preset detection area based on a preset moving object screening condition, and a moving direction of the moving object in each frame of images to be detected specifically includes:
starting from the second frame of image to be detected, for each frame of image to be detected, performing the following operations:
based on a preset feature matching algorithm, determining a plurality of motion pixel points of an image to be detected of a current frame relative to an image to be detected of a previous frame in a preset detection area, and determining motion vectors of the plurality of motion pixel points;
screening a plurality of target pixel points with the motion distance meeting a preset effective numerical range from the plurality of motion pixel points based on the motion vectors of the plurality of motion pixel points;
when the number of the plurality of target pixel points exceeds a preset target composition threshold, determining that a moving target exists in a preset detection area in the image to be detected of the current frame, wherein the moving target is composed of the plurality of target pixel points;
and determining the motion direction of the moving target in the image to be detected of the current frame based on the motion directions of the target pixel points.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010310655.4A CN111462190B (en) | 2020-04-20 | 2020-04-20 | Intelligent refrigerator and food material input method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010310655.4A CN111462190B (en) | 2020-04-20 | 2020-04-20 | Intelligent refrigerator and food material input method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111462190A CN111462190A (en) | 2020-07-28 |
| CN111462190B true CN111462190B (en) | 2023-11-17 |
Family
ID=71681893
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010310655.4A Active CN111462190B (en) | 2020-04-20 | 2020-04-20 | Intelligent refrigerator and food material input method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111462190B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114857852B (en) * | 2021-03-29 | 2023-08-15 | 青岛海尔电冰箱有限公司 | High-quality image acquisition method, equipment and refrigerator |
| WO2022262234A1 (en) * | 2021-06-18 | 2022-12-22 | 海信集团控股股份有限公司 | Refrigerator and food management method and device thereof |
| CN114857853A (en) * | 2021-08-05 | 2022-08-05 | 青岛海尔电冰箱有限公司 | Refrigeration equipment, image processing method and device |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6597737B1 (en) * | 1998-09-03 | 2003-07-22 | Sony Corporation | Motion determining apparatus, method thereof, and picture information converting apparatus |
| CN102142058A (en) * | 2010-02-03 | 2011-08-03 | 康佳集团股份有限公司 | Refrigerator and food management method and device thereof |
| CN103593678A (en) * | 2013-10-16 | 2014-02-19 | 长安大学 | Long-span bridge vehicle dynamic load distribution detection method |
| CN106204573A (en) * | 2016-07-07 | 2016-12-07 | Tcl集团股份有限公司 | A kind of food control method and system of intelligent refrigerator |
| JP2017162409A (en) * | 2016-03-11 | 2017-09-14 | ヤンマー株式会社 | Recognizing device, and method, for facial expressions and motions |
| CN107545049A (en) * | 2017-08-18 | 2018-01-05 | 腾讯科技（深圳）有限公司 | Image processing method and related product |
| CN109242011A (en) * | 2018-08-27 | 2019-01-18 | 深圳开立生物医疗科技股份有限公司 | A kind of method and device identifying image difference |
| CN109974743A (en) * | 2019-03-14 | 2019-07-05 | 中山大学 | A kind of RGB-D visual odometry optimized based on GMS characteristic matching and sliding window pose figure |
| CN110084765A (en) * | 2019-05-05 | 2019-08-02 | Oppo广东移动通信有限公司 | A kind of image processing method, image processing apparatus and terminal device |
| WO2019191889A1 (en) * | 2018-04-02 | 2019-10-10 | 北京大学 | Method and device for video processing |
| CN110689560A (en) * | 2019-10-11 | 2020-01-14 | 海信集团有限公司 | Food material management method and equipment |
| WO2020063072A1 (en) * | 2018-09-30 | 2020-04-02 | 中兴通讯股份有限公司 | Method, system, and terminal for image screening, and computer-readable storage medium |
-
2020
- 2020-04-20 CN CN202010310655.4A patent/CN111462190B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6597737B1 (en) * | 1998-09-03 | 2003-07-22 | Sony Corporation | Motion determining apparatus, method thereof, and picture information converting apparatus |
| CN102142058A (en) * | 2010-02-03 | 2011-08-03 | 康佳集团股份有限公司 | Refrigerator and food management method and device thereof |
| CN103593678A (en) * | 2013-10-16 | 2014-02-19 | 长安大学 | Long-span bridge vehicle dynamic load distribution detection method |
| JP2017162409A (en) * | 2016-03-11 | 2017-09-14 | ヤンマー株式会社 | Recognizing device, and method, for facial expressions and motions |
| CN106204573A (en) * | 2016-07-07 | 2016-12-07 | Tcl集团股份有限公司 | A kind of food control method and system of intelligent refrigerator |
| CN107545049A (en) * | 2017-08-18 | 2018-01-05 | 腾讯科技（深圳）有限公司 | Image processing method and related product |
| WO2019191889A1 (en) * | 2018-04-02 | 2019-10-10 | 北京大学 | Method and device for video processing |
| CN109242011A (en) * | 2018-08-27 | 2019-01-18 | 深圳开立生物医疗科技股份有限公司 | A kind of method and device identifying image difference |
| WO2020063072A1 (en) * | 2018-09-30 | 2020-04-02 | 中兴通讯股份有限公司 | Method, system, and terminal for image screening, and computer-readable storage medium |
| CN109974743A (en) * | 2019-03-14 | 2019-07-05 | 中山大学 | A kind of RGB-D visual odometry optimized based on GMS characteristic matching and sliding window pose figure |
| CN110084765A (en) * | 2019-05-05 | 2019-08-02 | Oppo广东移动通信有限公司 | A kind of image processing method, image processing apparatus and terminal device |
| CN110689560A (en) * | 2019-10-11 | 2020-01-14 | 海信集团有限公司 | Food material management method and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111462190A (en) | 2020-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111462190B (en) | Intelligent refrigerator and food material input method | |
| US11538232B2 (en) | Tracker assisted image capture | |
| US10282617B2 (en) | Methods and systems for performing sleeping object detection and tracking in video analytics | |
| US10553091B2 (en) | Methods and systems for shape adaptation for merged objects in video analytics | |
| US9852511B2 (en) | Systems and methods for tracking and detecting a target object | |
| Delannay et al. | Detection and recognition of sports (wo) men from multiple views | |
| US9600744B2 (en) | Adaptive interest rate control for visual search | |
| US9396539B2 (en) | Methods and apparatuses for face detection | |
| US10269123B2 (en) | Methods and apparatus for video background subtraction | |
| CN104883548B (en) | Monitor video face captures processing method and its system | |
| US10229503B2 (en) | Methods and systems for splitting merged objects in detected blobs for video analytics | |
| CN107633208B (en) | Electronic device, the method for face tracking and storage medium | |
| US20180047171A1 (en) | Methods and systems of maintaining object trackers in video analytics | |
| CN109800699A (en) | Image-recognizing method, system and device | |
| CN110532837B (en) | Image data processing method in article picking and placing process and household appliance | |
| CN110929658A (en) | Image recognition method, image recognition device and refrigerator | |
| CN103187083B (en) | A kind of storage means based on time domain video fusion and system thereof | |
| WO2008154825A1 (en) | A method and device for extracting a background image in a motion image | |
| JP2016076791A (en) | Object tracking device, object tracking method, and program | |
| WO2020258889A1 (en) | Tracking method of video tracking device, and video tracking device | |
| CN110780780B (en) | Image processing method and device | |
| AU2009230796A1 (en) | Location-based brightness transfer function | |
| CN113496185A (en) | Refrigerator and food material detection method | |
| CN111767759A (en) | System, method, device and storage medium for food material identification | |
| KR102585604B1 (en) | Image synchronization methods and devices, devices, computer storage media |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |