CN113465286B

CN113465286B - Intelligent refrigerator, angle determination and food material identification method

Info

Publication number: CN113465286B
Application number: CN202010677005.3A
Authority: CN
Inventors: 谢飞学; 曲磊; 赵启东; 高桢; 李正义; 高语函; 孙菁; 高雪松
Original assignee: Qingdao Hisense Electronic Industry Holdings Co Ltd
Current assignee: Hisense Group Holding Co Ltd
Priority date: 2020-07-14
Filing date: 2020-07-14
Publication date: 2022-12-20
Anticipated expiration: 2040-07-14
Also published as: CN113465286A

Abstract

The application discloses an intelligent refrigerator, an angle determining method and a food material identifying method, which are used for solving the problem that the determined refrigerator angle is not accurate enough in the existing angle determining method for opening a refrigerator door body. Because this application installation earth magnetism sensor on the refrigerator door body, when the treater detects the refrigerator door body and is opened, acquire the magnetic field intensity of the earth magnetism that the earth magnetism measured at the current position of earth magnetism sensor at the cross axle magnetic field intensity of the projection of cross axle direction, follow-up based on this cross axle magnetic field intensity, confirm the angle that the refrigerator door body was opened to realized based on the magnetic field intensity that the earth magnetism sensor measured, confirm the angle that the refrigerator door body was opened.

Description

Intelligent refrigerator, angle determination and food material identification method

Technical Field

The application relates to the technical field of intelligent household appliances, in particular to an intelligent refrigerator, a method, a device, equipment and a medium for determining an opening angle of a refrigerator door body and identifying food materials.

Background

At present, the combination of artificial intelligence technology and home appliances is becoming more and more compact, and the intelligent development of the refrigerator as one of the typical home appliances with high use frequency and long use time in the home appliances has become one of the current focusing directions. The difference of the refrigeration technology, the volume and the appearance of the refrigerator is gradually reduced, and the technologies of food material accurate management, intelligent interaction and the like become important implementation modes for improving the product differentiation.

In the development of an intelligent refrigerator, food material identification is one of the largest bright spots, and although a series of problems of various food materials, large shape difference of the same food material, complex access scene and the like exist, the identification accuracy rate is higher and higher along with the rapid development of a deep learning algorithm. However, when food material images on a rack of a refrigerator door are collected, due to the fact that the refrigerator door is opened at different angles, food materials in the food material images can deform to a certain extent, and therefore when food material identification is subsequently performed according to the food material images, the identification result accuracy is low. The method is an important means for improving the accuracy of food identification by acquiring the opening angle of the refrigerator door body and calibrating the food material image on the rack of the refrigerator door body through the opening angle of the refrigerator door body. Therefore, how to obtain the opening angle of the refrigerator door body is a problem which is paid increasing attention in recent years.

In the prior art, in order to obtain an opening angle of a refrigerator door, an acceleration sensor such as a gyroscope is generally mounted on the refrigerator door, and corresponding calculation is performed based on an acceleration measured by the acceleration sensor such as the gyroscope, so as to determine the opening angle of the refrigerator door. However, the method is easy to cause acceleration accumulation errors, so that the obtained acceleration measured by the acceleration sensor is inaccurate, and the angle of the subsequently determined opening angle of the refrigerator door body also has angle errors.

Disclosure of Invention

The application provides an intelligent refrigerator, a method, a device, equipment and a medium for determining an opening angle of a refrigerator door body and identifying food materials, and aims to solve the problem that the determined refrigerator angle is not accurate enough in the existing method for determining the opening angle of the refrigerator door body.

In a first aspect, the present application provides an intelligent refrigerator, comprising: a geomagnetic sensor and a processor;

the geomagnetic sensor is mounted on a refrigerator door body of the intelligent refrigerator and used for measuring the magnetic field intensity of a transverse shaft of the projection of the magnetic field intensity of the geomagnetic field at the current position of the geomagnetic sensor in the transverse shaft direction, wherein the transverse shaft direction is parallel to the ground and is determined according to the magnetic field intensity of the transverse shaft;

the processor is used for acquiring the magnetic field strength of the transverse shaft measured by the geomagnetic sensor if the refrigerator door body is opened, and determining the opening angle of the refrigerator door body according to a first ratio of the acquired magnetic field strength of the transverse shaft to the pre-stored geomagnetic strength.

In a second aspect, the present application further provides a method for determining an opening angle of a refrigerator door, where a geomagnetic sensor is installed on the refrigerator door, and the method includes:

if the refrigerator door body is opened, acquiring a transverse axis magnetic field strength of a projection of the magnetic field strength of the geomagnetism measured by the geomagnetic sensor at the current position of the geomagnetic sensor in the transverse axis direction, wherein the transverse axis direction is parallel to the ground and is determined according to the transverse axis magnetic field strength;

and determining the opening angle of the refrigerator door according to a first ratio of the magnetic field intensity of the transverse shaft to the pre-stored geomagnetic intensity.

In a third aspect, the present application further provides a food material identification method based on the method for determining the opening angle of the refrigerator door body, where a geomagnetic sensor is installed on the refrigerator door body, and the method includes:

if the refrigerator door body is opened, acquiring food material images on a rack of the refrigerator door body and determining the opening angle of the refrigerator door body;

calibrating the food material images according to the food material images and the opening angle of the refrigerator door body, and acquiring the calibrated food material images;

and determining the type of the food material contained in the calibrated food material image through a food material identification model.

In a fourth aspect, the present application further provides a refrigerator door body opening angle determining device, a geomagnetic sensor is installed on the refrigerator door body, and the device includes:

an acquisition unit configured to acquire, if the refrigerator door is opened, a transverse axis magnetic field strength of a projection of a magnetic field strength of geomagnetism measured by the geomagnetic sensor at a current position of the geomagnetic sensor in a transverse axis direction, the transverse axis direction being parallel to a ground surface and being determined according to the transverse axis magnetic field strength;

and the processing unit is used for determining the opening angle of the refrigerator door body according to a first ratio of the magnetic field intensity of the transverse shaft to the pre-stored geomagnetic intensity.

In a fifth aspect, the present application further provides a food material identification device based on the method for determining an opening angle of a refrigerator door body, where a geomagnetic sensor is installed on the refrigerator door body, and the device includes:

the refrigerator comprises an acquisition unit, a storage unit and a control unit, wherein the acquisition unit is used for acquiring food material images on racks of a refrigerator door body and determining the opening angle of the refrigerator door body if the refrigerator door body is opened;

the calibration unit is used for calibrating the food material images according to the food material images and the opening angle of the refrigerator door body, and acquiring the calibrated food material images;

and the processing unit is used for determining the type of the food material contained in the calibrated food material image through a food material identification model.

In a sixth aspect, the present application further provides an electronic device, which at least includes a processor and a memory, where the processor is configured to implement the steps of the method for determining an opening angle of the refrigerator door and the steps of the method for identifying food materials as described above when executing a computer program stored in the memory.

In a seventh aspect, the present application further provides a computer-readable storage medium storing a computer program, which when executed by a processor, implements the steps of the method for determining the opening angle of the refrigerator door as described above, and implements the steps of the method for identifying food materials as described above.

Because this application installation earth magnetism sensor on the refrigerator door body, when the treater detects the refrigerator door body and is opened, acquire the magnetic field intensity of the earth magnetism that the earth magnetism measured at the current position of earth magnetism sensor at the cross axle magnetic field intensity of the projection of cross axle direction, follow-up based on this cross axle magnetic field intensity, confirm the angle that the refrigerator door body was opened to realized based on the magnetic field intensity that the earth magnetism sensor measured, confirm the angle that the refrigerator door body was opened.

Drawings

In order to more clearly illustrate the technical solutions of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.

Fig. 1 is a schematic structural diagram of an intelligent refrigerator according to some embodiments of the present application;

fig. 2 is a schematic diagram of geomagnetic intensity and horizontal axis magnetic field intensity in a coordinate system of horizontal axis and vertical axis of the geomagnetic sensor according to some embodiments of the present application;

FIG. 3 is a bottom view of the installation location of a geomagnetic sensor provided in some embodiments of the present application;

fig. 4 is a schematic view illustrating an opening angle of a refrigerator door according to some embodiments of the present disclosure;

5 (a) -5 (b) are schematic diagrams of measured magnetic field strength variations of RM3100 provided in accordance with some embodiments of the present application;

FIG. 6 is a line graph illustrating magnetic field strength variations measured by a magnetic sensor during rotation of a simulated refrigerator door provided in accordance with some embodiments of the present disclosure;

fig. 7 is a schematic structural diagram of an intelligent refrigerator according to some embodiments of the present application;

fig. 8 is a schematic view illustrating a food material identification process of a specific intelligent refrigerator based on a refrigerator door body opening angle determination method according to some embodiments of the present disclosure;

fig. 9 is a schematic view illustrating a process for determining an opening angle of a refrigerator door according to some embodiments of the present disclosure;

fig. 10 is a schematic view illustrating a food material identification process based on a method for determining an opening angle of a refrigerator door according to some embodiments of the present disclosure;

fig. 11 is a schematic structural diagram of an apparatus for determining an opening angle of a refrigerator door according to some embodiments of the present disclosure;

fig. 12 is a schematic structural diagram of a food material identification apparatus based on a method for determining an opening angle of a refrigerator door according to some embodiments of the present disclosure;

fig. 13 is a schematic structural diagram of an intelligent refrigerator according to some embodiments of the present application;

fig. 14 is a schematic structural diagram of an electronic device according to some embodiments of the present application;

fig. 15 is a schematic structural diagram of an electronic device according to some embodiments of the present application.

Detailed Description

In order to improve the accuracy of the determined opening angle of the refrigerator door body, the application provides an intelligent refrigerator, a method, a device, equipment and a medium for determining the opening angle of the refrigerator door body and identifying food materials.

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the accompanying drawings, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In the application scene of the intelligent refrigerator, food material identification is a key technology in the field of the intelligent refrigerator, and for collected food material images on a rack of a refrigerator door body, because the angles at which the refrigerator door body is opened are different, the collected food material images also have difference, so that the identification result accuracy is low when food material identification is subsequently carried out on the basis of the food material images. Therefore, in order to determine the opening angle of the refrigerator door, a geomagnetic sensor can be installed on the refrigerator door, when a user opens the refrigerator door, the intelligent refrigerator can directly obtain the magnetic field intensity of the transverse shaft measured by the geomagnetic sensor locally, and based on the determination method of the opening angle of the refrigerator door, the magnetic field intensity of the transverse shaft is correspondingly processed, so that the opening angle of the refrigerator door is determined.

Fig. 1 is a schematic structural diagram of an intelligent refrigerator provided in some embodiments of the present application, where the intelligent refrigerator includes: a geomagnetic sensor 11 and a processor 12;

the geomagnetic sensor 11 is mounted on a refrigerator door body of the intelligent refrigerator and is used for measuring a transverse-axis magnetic field intensity of a projection of a magnetic field intensity of geomagnetism at the current position of the geomagnetic sensor 11 in a transverse-axis direction, wherein the transverse-axis direction is parallel to the ground and is determined according to the transverse-axis magnetic field intensity;

the processor 12 is configured to, if the refrigerator door is opened, acquire a transverse-axis magnetic field strength measured by the geomagnetic sensor 11, and determine an angle at which the refrigerator door is opened according to a first ratio of the acquired transverse-axis magnetic field strength to a prestored geomagnetic strength.

In the application, in order to determine the opening angle of the refrigerator door conveniently, the geomagnetic sensor 11 is installed on the refrigerator door of the intelligent refrigerator, and in order to enable the geomagnetic sensor 11 to accurately measure the change of the magnetic field strength of the geomagnetic field in the rotation process of the refrigerator door, the installation position of the geomagnetic sensor 11 should be close to the ground, for example, be installed below the refrigerator door. When the refrigerator door body of intelligence refrigerator was opened to different angles, this geomagnetic sensor 11's position can change, leads to its measuring magnetic field intensity also to produce corresponding change, and intelligence refrigerator can carry out corresponding processing based on this geomagnetic sensor 11 measuring magnetic field intensity to confirm the angle that the refrigerator door body of intelligence refrigerator was opened.

In a specific implementation process, the geomagnetic sensor 11 measures the magnetic field strength of the horizontal axis of the projection of the magnetic field strength of the geomagnetic field at the current position of the geomagnetic sensor 11 in the horizontal axis direction in real time, and the processor 12 monitors the condition that the refrigerator door is opened in real time. When the processor 12 detects that the refrigerator door is opened, the horizontal-axis magnetic field strength of the geomagnetic sensor 11 is obtained according to a preset time interval, and corresponding processing is performed on each obtained horizontal-axis magnetic field strength, so that the angle at which the refrigerator door corresponding to the horizontal-axis magnetic field strength is opened.

When the geomagnetic sensor 11 mounted on the refrigerator door is selected, the geomagnetic sensor 11 should be capable of measuring at least the magnetic field strength of the horizontal axis. Specifically, the direction of the transverse axis is parallel to the ground, and is determined according to the magnetic field intensity of the transverse axis measured by the geomagnetic sensor 11 installed on the refrigerator door body when an operator installs the intelligent refrigerator in the home of the user.

The geomagnetic intensity may vary due to different geographical locations. In the application, in order to accurately determine the opening angle of the refrigerator door, the pre-stored geomagnetic intensity is the transverse-axis magnetic field intensity corresponding to the determined transverse-axis direction in the process that a worker installs the intelligent refrigerator in a user home.

After the transverse-axis magnetic field strength measured by the geomagnetic sensor 11 is acquired based on the above embodiment, corresponding processing is performed according to a first ratio of the transverse-axis magnetic field strength to the pre-stored geomagnetic strength, so as to determine the opening angle of the refrigerator door.

According to the refrigerator door opening method and device, the geomagnetic sensor 11 is installed on the refrigerator door, when the processor 12 detects that the refrigerator door is opened, the magnetic field intensity of the cross shaft of the projection of the magnetic field intensity of the geomagnetic field measured by the geomagnetic sensor 11 at the current position of the geomagnetic sensor 11 in the direction of the cross shaft is obtained, and the opening angle of the refrigerator door is determined subsequently based on the magnetic field intensity of the cross shaft, so that the magnetic field intensity measured by the geomagnetic sensor 11 is used for determining the opening angle of the refrigerator door.

In order to accurately determine the angle at which the refrigerator door is opened, on the basis of the above embodiment, in the present application, the processor 12 is specifically configured to determine the angle corresponding to the first ratio; and taking the angle as the opening angle of the refrigerator door body.

In a specific implementation process, in order to accurately determine the opening angle of the refrigerator door body, a corresponding relation between the ratio and the angle, such as an inverse cosine function relation, is stored in advance. After the first ratio is obtained based on the above embodiment, the angle corresponding to the first ratio is determined according to the corresponding relationship between the ratio and the angle stored in advance, and the angle can be used as the angle at which the refrigerator door body is opened. For example, the pre-stored corresponding relationship between the ratio and the angle is an inverse cosine function relationship, the horizontal axis magnetic field strength measured by the geomagnetic sensor 11 is obtained as 0, and the first ratio between the horizontal axis magnetic field strength and the pre-stored geomagnetic field strength is obtained as 0, then according to the inverse cosine function relationship, the angle corresponding to the first ratio 0 is determined as arcos (0) =90 degrees, and the angle is taken as the angle at which the refrigerator door is opened.

Since in general, the geomagnetic intensity and the horizontal axis magnetic field strength have a cosine relationship in the coordinate system of the horizontal axis and the vertical axis of the geomagnetic sensor 11, fig. 2 is a schematic diagram of the geomagnetic intensity and the horizontal axis magnetic field strength provided in some embodiments of the present application in the coordinate system of the horizontal axis and the vertical axis of the geomagnetic sensor, in the coordinate system of the horizontal axis and the vertical axis of the geomagnetic sensor 11 shown in fig. 2, an angle between the geomagnetic intensity and the horizontal axis magnetic field strength is θ, and the cosine relationship between the geomagnetic intensity and the horizontal axis magnetic field strength can be expressed as: b is a mixture of _x = Bcos θ, wherein, b _x The transverse axis magnetic field intensity, B the pre-stored geomagnetic intensity, and theta the opening angle of the refrigerator door.

Therefore, when the geomagnetic intensity is pre-stored and the horizontal axis magnetic field intensity is obtained, the inverse cosine function value corresponding to the first ratio of the horizontal axis magnetic field intensity to the pre-stored geomagnetic intensity can be determined, so as to determine the opening angle of the refrigerator. Specifically, the opening angle of the refrigerator door can be determined according to the following formula:

the meaning of each parameter in the formula is the same as that in the above embodiment, and is not described herein again. In the practical application process, the opening angle of the refrigerator door body is generally in the angle range of [0, pi ].

In order to facilitate determining the horizontal axis direction of the geomagnetic sensor 11, in the present application, on the basis of the above embodiment, if a trigger operation for adjusting the horizontal axis knob direction of the geomagnetic sensor 11 is received, the processor 12 is further configured to obtain the magnetic field strength of the geomagnetism measured by the geomagnetic sensor 11 at the current measurement position of the geomagnetic sensor 11 and the horizontal axis magnetic field strength of the projection in the current candidate horizontal axis direction; and determining the horizontal axis direction according to the candidate horizontal axis direction corresponding to the maximum horizontal axis magnetic field strength.

In order to facilitate subsequent determination of the opening angle of the refrigerator door, the direction of the geomagnetic sensor 11 mounted on the refrigerator door needs to be adjusted, that is, the direction of the geomagnetic sensor 11 is adjusted within a 360-degree rotation range at the mounting position of the geomagnetic sensor 11. Since the transverse axis direction of the geomagnetic sensor 11 mounted on the intelligent refrigerator is fixed when the intelligent refrigerator is shipped, the transverse axis direction of the geomagnetic sensor 11 is rotated with the rotation of the direction of the geomagnetic sensor 11 when the direction of the geomagnetic sensor 11 at the mounting position is adjusted, and the transverse axis direction after each rotation is taken as a candidate transverse axis direction.

In the specific implementation process, because the mounting positions of the intelligent refrigerator are different, the geomagnetic intensity at the mounting position where the intelligent refrigerator is located is different. Therefore, after the delivery of the intelligent refrigerator, the worker needs to go to the home of the user to adjust the direction of the geomagnetic sensor 11 mounted on the refrigerator door of the intelligent refrigerator, and each time the direction of the geomagnetic sensor 11 is adjusted to a predetermined position, the horizontal axis magnetic field strength measured by the geomagnetic sensor 11 is acquired, and according to each acquired horizontal axis magnetic field strength, the direction corresponding to the maximum horizontal axis magnetic field strength is determined as the target direction of the geomagnetic sensor 11, and when the geomagnetic sensor 11 is in the target direction, it can be determined that the candidate horizontal axis direction of the geomagnetic sensor 11 is the same as the geomagnetic direction, and then the candidate horizontal axis direction can be determined as the horizontal axis direction.

In order to adjust the direction of the geomagnetic sensor 11 conveniently, in this application, a horizontal axis knob is provided on the geomagnetic sensor 11, and a worker may adjust the direction of the geomagnetic sensor 11 by adjusting the orientation of the horizontal axis knob of the geomagnetic sensor 11, so that the horizontal axis direction of the geomagnetic sensor 11 changes with the change of the direction of the geomagnetic sensor 11 at the installation position. Fig. 3 is a bottom view of the installation position of a geomagnetic sensor provided in some embodiments of the present application. Referring to fig. 3, the geomagnetic sensor 11 is installed below the refrigerator door, and a horizontal axis knob is provided on the geomagnetic sensor 11. In order to accurately adjust the direction of the geomagnetic sensor 11 through the horizontal shaft knob, determine the target direction of the geomagnetic sensor 11, the horizontal shaft knob and the geomagnetic sensor 11 may be connected through a gear device, for example, the gear device includes a gear wheel and a pinion, because the direction of the geomagnetic sensor 11 that the staff cannot accurately adjust may occur, in this application, when the horizontal shaft knob is rotated, the horizontal shaft knob may drive the pinion to rotate, the gear wheel is driven by the pinion to rotate, the horizontal shaft that the geomagnetic sensor 11 is driven by the gear wheel to rotate again, through the gear device, thereby realizing the effect of accurately adjusting the direction of the geomagnetic sensor 11.

In a specific implementation process, after the intelligent refrigerator receives a trigger operation for a worker to adjust a horizontal axis knob of the geomagnetic sensor 11 to point, where the trigger operation is an operation for adjusting the horizontal axis knob of the geomagnetic sensor 11 to point to a certain position, that is, adjusting a direction of the geomagnetic sensor 11 at a current installation position, and taking a horizontal axis direction of the geomagnetic sensor 11 while the geomagnetic sensor 11 is in the current direction as a candidate horizontal axis direction of the geomagnetic sensor 11. The magnetic field strength of the geomagnetism measured by the geomagnetic sensor 11 at the current measurement position of the geomagnetic sensor 11 and the magnetic field strength of the abscissa of the projection in the current candidate abscissa direction are obtained. And in order to facilitate the working personnel to know the magnetic field intensity of the transverse shaft measured by the geomagnetic sensor 11 in time, the intelligent refrigerator can be controlled to output the magnetic field intensity of the transverse shaft measured by the magnetic sensor.

The mode that the intelligent refrigerator outputs the magnetic field intensity of the transverse shaft measured by the geomagnetic sensor 11 is various, the magnetic field intensity of the transverse shaft measured by the geomagnetic sensor 11 can be displayed on a display screen of the intelligent refrigerator, and the magnetic field intensity of the transverse shaft measured by the geomagnetic sensor 11 can also be sent to a mobile terminal of a worker in a data transmission mode. Specifically, which output mode is adopted can be preset according to the requirement, or can be selected according to the capacity of the intelligent refrigerator. For example, some intelligent refrigerators do not have a display screen, and for these intelligent refrigerators, the horizontal-axis magnetic field strength measured by the geomagnetic sensor 11 may be transmitted to the mobile terminal of the staff member for output by data transmission.

In practical applications, a worker generally adjusts the orientation of the horizontal axis knob according to a predetermined direction, such as clockwise or counterclockwise, continuously according to each preset position, and in the process of changing the orientation of the geomagnetic sensor 11 by adjusting the orientation of the horizontal axis knob, the horizontal axis direction of the geomagnetic sensor 11 is closer to the geomagnetic direction as the horizontal axis knob is turned, the horizontal axis magnetic field strength measured by the geomagnetic sensor 11 is larger until the horizontal axis is turned to approximate to the geomagnetic direction, the horizontal axis magnetic field strength measured by the geomagnetic sensor 11 reaches a maximum value, and after the horizontal axis direction is approximate to the geomagnetic direction, the horizontal axis knob is turned continuously, that is, the horizontal axis direction of the geomagnetic sensor 11 is further away from the geomagnetic direction, the horizontal axis magnetic field strength measured by the geomagnetic sensor 11 is smaller until the horizontal axis is turned to approximate to the opposite to the geomagnetic direction, and the horizontal axis magnetic field strength measured by the geomagnetic sensor 11 reaches a minimum value.

In this way, when determining that the acquired horizontal-axis magnetic field strength is increasing and then decreasing in the already acquired horizontal-axis magnetic field strengths according to the above-described rule, the processor 12 may determine that the candidate horizontal-axis direction corresponding to the largest horizontal-axis magnetic field strength among the acquired horizontal-axis magnetic field strengths is approximately the geomagnetic direction, and determine the candidate horizontal-axis direction as the horizontal-axis direction.

In a specific implementation process, the method for determining the maximum transverse-axis magnetic field strength by the processor 12 according to the acquired transverse-axis magnetic field strength includes: if the currently acquired horizontal axis magnetic field strength is determined to be not more than the horizontal axis magnetic field strength acquired last time, and the horizontal axis magnetic field strength acquired last time is determined to be not less than the horizontal axis magnetic field strength acquired last time corresponding to the horizontal axis magnetic field strength acquired last time, the acquired horizontal axis magnetic field strength is determined to be the maximum horizontal axis magnetic field strength after being increased all the time and beginning to be decreased all the time.

When the maximum horizontal-axis magnetic field strength is determined, the processor 12 may control the intelligent refrigerator to output a prompt message, for example, "please adjust the horizontal-axis knob to the position of the last adjustment" to prompt the worker about the candidate horizontal-axis direction corresponding to the maximum horizontal-axis magnetic field strength.

The mode of the prompt message output by the intelligent refrigerator may be displaying the prompt message on a display screen of the intelligent refrigerator, or notifying a worker through communication modes such as a short message and a notification, which is not specifically limited herein.

After the operator determines the target direction of the geomagnetic sensor 11 at the installation position, that is, after the transverse axis direction of the magnetic sensor 11 is approximately the geomagnetic direction when the geomagnetic sensor 11 is determined to be at the target direction, it is not necessary to continuously adjust the orientation of the transverse axis knob of the geomagnetic sensor 11, therefore, when the target direction of the geomagnetic sensor 11 is determined, the transverse axis direction of the geomagnetic sensor 11 at the target direction at the installation position is determined, the orientation of the transverse axis knob of the geomagnetic sensor 11, that is, the knob position, is fixed, and the processor 12 stores the transverse axis magnetic field strength corresponding to the currently determined transverse axis direction as the geomagnetic strength, so as to subsequently determine the angle at which the refrigerator door is opened.

In another possible embodiment, since the opening angle of the refrigerator door is not uniform and the angular resolution of the geomagnetic sensor 11 is not linear in the process of opening the refrigerator door by the user, when the collected food material image on the rack of the refrigerator door is calibrated according to the determined opening angle of the refrigerator, the quality of the food material image calibration does not have a linear relationship with the determined opening angle of the refrigerator, and the processor 12 can only make the determined opening angle of the refrigerator door more accurate as much as possible.

In a practical application scenario, when a user opens the refrigerator, the opening angle of the door body of the refrigerator is not too large or too small, as long as objects can be placed, and generally the angle range is not less than 30 degrees and not more than 90 degrees. If when a worker installs the intelligent refrigerator at home, the angle of the intelligent refrigerator door body is opened to a certain angle in the angle range, and then the direction of the geomagnetic sensor 11 installed on the refrigerator door body is adjusted, so that the magnetic field intensity of the transverse shaft measured by the geomagnetic sensor 11 can be more accurate when the subsequent user opens the refrigerator door body to other angles in a certain angle range, and therefore the angle at which the refrigerator door body is opened can be more accurate according to the magnetic field intensity of the transverse shaft. Therefore, the reference angle can be determined within the range, so that other angles whose difference from the reference angle is within a certain angle range can be identified more accurately.

When the intelligent refrigerator leaves a factory, the preset reference angles corresponding to the intelligent refrigerator are recorded into the intelligent refrigerator, wherein the preset reference angles corresponding to each intelligent refrigerator can be the same or different. When determining the target direction of the geomagnetic sensor 11 on the intelligent refrigerator, the staff opens the refrigerator door to a preset reference angle, such as 60 degrees, 70 degrees, and the like, according to the preset reference angle corresponding to the intelligent refrigerator stored in advance, and then determines the target direction of the geomagnetic sensor 11 based on the method of the above embodiment, that is, determines the horizontal axis direction of the geomagnetic sensor 11 in the target direction.

For example, a pre-stored preset reference angle corresponding to a certain intelligent refrigerator is 60 degrees, when the intelligent refrigerator is installed, a worker first opens the refrigerator door of the intelligent refrigerator to about 60 degrees, temporarily fixes the refrigerator door, and then adjusts the direction of the geomagnetic sensor 11 in the installation position by adjusting the direction of the horizontal shaft knob of the geomagnetic sensor 11 on the refrigerator door, so that the horizontal shaft direction of the geomagnetic sensor 11 can rotate along with the rotation of the horizontal shaft knob. When the processor 12 determines that the strength of the magnetic field on the horizontal axis obtained and measured by the geomagnetic sensor 11 is the maximum based on the above method, the change in the strength of the magnetic field on the horizontal axis per unit angle is the maximum, that is, the processor is configured to measure the magnetic field on the horizontal axis

Maximum, wherein, b _x Is the magnetic field strength of the transverse axis,

the maximum transverse magnetic field strength is determined for a unit angle around a preset reference angle betaThe direction of the corresponding geomagnetic sensor 11 is a target direction, and a candidate horizontal axis direction of the geomagnetic sensor 11 in the target direction is determined as a horizontal axis direction, which is approximate to the geomagnetic direction. Subsequently, when the user opens the refrigerator door body to about 60 degrees, the magnetic field strength of the transverse shaft measured by the geomagnetic sensor 11 is relatively large, and the determined opening angle of the refrigerator is relatively accurate.

In order to further accurately determine the opening angle of the refrigerator door, on the basis of the foregoing embodiments, in the present application, the processor 12 is specifically configured to determine a correction angle according to a first ratio of the magnetic field strength of the transverse axis to a pre-stored geomagnetic strength, and a direction of a projection of the magnetic field strength in a direction of the longitudinal axis; and determining the opening angle of the refrigerator door body according to the corrected angle and a preset reference angle.

When the transverse axis direction of the geomagnetic sensor 11 is adjusted to be approximately the geomagnetic direction, the refrigerator door body is opened to a preset reference angle, and the reference angle is determined according to the magnetic field intensity of the transverse axis, in the correspondence relationship between the ratio and the angle stored in advance, the angle corresponding to the ratio is no longer the angle at which the refrigerator door body is opened. For example, the pre-stored ratio and angle corresponding relation is an inverse cosine function

For example, the preset reference angle is β, the actual rotation angle of the refrigerator door is θ, and the preset reference angle is 90 degrees, if the coordinate system formed by the horizontal axis and the vertical axis of the geomagnetic sensor 11 is a right-hand coordinate system, the arccosine function is obtained by using the right-hand coordinate system as the coordinate system

In (1),

the corresponding angle is (theta-beta), and the angle range of the angle (theta-beta) may be (theta-beta)

That is to say it may be

In addition, the

Angle of (1) and

all of which have cosine values between 0 and 1, according to the inverse cosine function

When the angle (theta-beta) corresponding to the first ratio is determined, the angle (theta-beta) corresponding to the first ratio cannot be determined to be within the range

Is also within the range of

Thereby affecting the subsequent determination of the opening angle theta of the refrigerator door body.

Also taking the above as an example, if the coordinate system formed by the horizontal axis and the vertical axis of the geomagnetic sensor 11 is the left-handed coordinate system, the value range [0, pi ] of the angle in the corresponding relationship between the prestored ratio and the angle]The corresponding actual angle (β - θ) may be in the range of

Namely may be

In addition, the

Angle of (1) and

When the angle (beta-theta) corresponding to the first ratio is determined, the angle (beta-theta) corresponding to the first ratio cannot be determined to be within the range

Is also within the range of

During the practical application process, the method is applied to

An angle in a range of the direction of projection of the magnetic field intensity of the geomagnetism in the direction of the longitudinal axis of the geomagnetic sensor 11 is a negative direction of the longitudinal axis, and

the direction of the projection of the magnetic field strength of the geomagnetism on the direction of the vertical axis of the geomagnetic sensor 11 is the positive direction of the vertical axis. Fig. 4 is a schematic view of an opening angle of a refrigerator door body according to some embodiments of the present disclosure, referring to fig. 4, where a preset reference angle is β, an actual rotation angle of the refrigerator door body is θ, and when the refrigerator door body rotates at the predetermined rotation angle

The direction of the projection of the intensity of the geomagnetic field in the direction of the vertical axis of the geomagnetic sensor 11 is the negative direction of the vertical axis at an angle within the range of (1)

The direction of the projection of the magnetic field intensity of the geomagnetism in the direction of the vertical axis of the geomagnetic sensor 11 is the positive direction of the vertical axis.

Based on the above, in order to further accurately determine the opening angle of the refrigerator door body, in the application, the ratio and the opening angle are stored in advance according to the ratioAnd determining two candidate angles corresponding to a first ratio of the magnetic field strength of the transverse axis to the pre-stored geomagnetic strength according to the corresponding relation of the angles, and then determining a correction angle from the two candidate angle values according to the projection direction of the magnetic field strength in the direction of the longitudinal axis. Specifically, if the direction of the projection of the magnetic field intensity in the direction of the vertical axis is the negative direction of the vertical axis, the magnetic field intensity will be in the direction of the vertical axis

The angle candidate in the range of (2) is set as a correction angle, and if the direction of the projection of the magnetic field intensity in the direction of the vertical axis is the positive direction of the vertical axis, the correction angle is set to be in the range of

The candidate angle within the range of (2) is used as the correction angle.

After the correction angle is determined based on the above embodiment, corresponding processing is performed according to the correction angle and a preset reference angle, and thus the angle at which the refrigerator door body is opened can be determined. In the specific implementation process, due to different coordinate systems, the correction angle and the preset reference angle are processed in different manners. If a coordinate system formed by a horizontal axis and a vertical axis of the geomagnetic sensor 11 is a left-hand coordinate system, determining an opening angle of the refrigerator door body according to a difference between a preset reference angle and the correction angle, namely, taking the difference between the preset reference angle and the correction angle as the opening angle of the refrigerator door body; if a coordinate system formed by the horizontal axis and the vertical axis of the geomagnetic sensor 11 is a right-hand coordinate system, determining the opening angle of the refrigerator door according to the sum of the preset reference angle and the correction angle, that is, the sum of the preset reference angle and the correction angle is used as the opening angle of the refrigerator door.

The preset reference angle and a coordinate system formed by a horizontal axis and a vertical axis of the geomagnetic sensor 11 are both a left-hand coordinate system or a right-hand coordinate system, which are configured in advance when the intelligent refrigerator leaves a factory, and if the coordinate system is the left-hand coordinate system, a determination method for determining the opening angle of the refrigerator door corresponding to the left-hand coordinate system and the preset reference angle are recorded into the intelligent refrigerator; if the coordinate system is a right-hand coordinate system, a determination method for determining the opening angle of the refrigerator door body corresponding to the right-hand coordinate system and a preset reference angle are recorded into the intelligent refrigerator. Subsequently, when the opening angle of the refrigerator door body is determined based on the method provided by the application, the opening angle of the refrigerator door body can be determined based on the recorded determination method for determining the opening angle of the refrigerator door body and the preset reference angle.

In order to facilitate subsequent determination of the opening angle of the refrigerator door body, the geomagnetic sensor 11 is further configured to measure a longitudinal axis magnetic field strength of a projection of the geomagnetic field strength in a longitudinal axis direction;

the processor 12 is specifically configured to, if the refrigerator door is opened, obtain a magnetic field intensity of the longitudinal axis measured by the geomagnetic sensor 11; if the second ratio of the magnetic field strength of the longitudinal axis to the corresponding absolute value is a positive value, determining that the direction of the projection of the magnetic field strength in the direction of the longitudinal axis is the positive direction of the longitudinal axis; and if the second ratio of the magnetic field strength of the transverse axis to the corresponding absolute value is a negative value, determining the projection direction of the magnetic field strength in the direction of the longitudinal axis as the negative direction of the longitudinal axis.

In the present application, in order to facilitate determination of the direction of the projection of the magnetic field strength in the vertical axis direction, the geomagnetic sensor 11 should also be able to measure the vertical axis magnetic field strength of the projection of the magnetic field strength of the geomagnetism at the current position of the geomagnetic sensor 11 in the vertical axis direction. And performing corresponding processing according to the magnetic field strength of the longitudinal axis, namely determining the projection direction of the magnetic field strength of the geomagnetism in the direction of the longitudinal axis.

Specifically, the direction of the projection of the magnetic field strength of the geomagnetism in the direction of the vertical axis may be determined based on the positive or negative of the magnetic field strength of the vertical axis, that is, if the magnetic field strength of the vertical axis is greater than 0, indicating that the direction of the magnetic field strength of the vertical axis is the same as the positive direction of the vertical axis, the direction of the projection of the magnetic field strength of the geomagnetism in the direction of the vertical axis is determined as the positive direction of the vertical axis; if the vertical axis magnetic field strength is less than 0, it is determined that the direction of the projection of the magnetic field strength of the geomagnetism in the vertical axis direction is the negative direction of the vertical axis.

Of course, the direction of the projection of the magnetic field strength of the geomagnetism in the direction of the vertical axis may also be determined according to a second ratio of the magnetic field strength of the vertical axis to the absolute value corresponding thereto, that is, if the second ratio is greater than 0, it indicates that the magnetic field strength of the vertical axis is greater than 0, and the direction of the magnetic field strength of the vertical axis is the same as the positive direction of the vertical axis, then the direction of the projection of the magnetic field strength of the geomagnetism in the direction of the vertical axis is determined as the positive direction of the vertical axis; if the second ratio is smaller than 0, it indicates that the magnetic field strength of the longitudinal axis is smaller than 0, and the direction of the magnetic field strength of the longitudinal axis is the same as the negative direction of the longitudinal axis, and it is determined that the direction of the projection of the magnetic field strength of the geomagnetism in the direction of the longitudinal axis is the negative direction of the longitudinal axis.

After the direction of the projection of the magnetic field strength of the geomagnetism in the longitudinal axis direction is obtained based on the above embodiment, the processor 12 is specifically configured to determine a first angle corresponding to the first ratio; and determining the correction angle according to the product of the second ratio and the first angle.

According to the corresponding relation between the ratio and the angle which are stored in advance, a first angle corresponding to a first ratio of the magnetic field intensity of the transverse axis to the geomagnetic intensity which is stored in advance is determined, wherein the first angle is between

Within an angular range of (c). A corrected angle value is then determined based on the product of the second ratio and the first angle. If the second ratio is a negative value, the first angle is wrong, the product of the second ratio and the first angle is the inverse of the first angle, and the inverse first angle is a correction angle; if the second ratio is positive, the first angle is correct, and the product of the second ratio and the first angle does not change the first angle, which is the correction angle.

Based on the content of the above embodiments, the opening angle of the refrigerator door body can be determined according to the following formula:

if the coordinate system formed by the horizontal axis and the vertical axis of the geomagnetic sensor 11 is a left-handed coordinate system, the geomagnetic sensor will be referred to as a geomagnetic sensor

Wherein beta is a preset reference angle, theta is an actual rotating angle of the refrigerator door body, b _x Is the magnetic field strength of the transverse axis, b _y The magnetic field intensity of the vertical axis and the magnetic field intensity of the prestored B.

In the present application, in order to accurately determine the opening angle of the refrigerator door, RM3100 manufactured by PIN corporation, usa may be used as the geomagnetic sensor 11. The RM3100 is a high-resolution, low-power consumption geomagnetic sensor 11.RM3100 comprises a sub-sensor in a horizontal axis direction, a sub-sensor in a vertical axis direction, a sub-sensor in a Z axis direction, and a microcontroller, and data measured by the sensors can be directly output to the singlechip/microcontroller through an internal I2C or SPI interface without AD conversion between the sensors and the microcontroller. In the method for determining the opening angle of the refrigerator door according to the present invention, only the magnetic field strength measured in the horizontal axis direction and the vertical axis direction of the magnetic sensor 11 is required, and therefore, only the sub-sensors in the horizontal axis direction and the vertical axis direction of the three-axis sub-sensors of the RM3100 may be used. The transverse axis direction sub-sensor plays a main role in determining the opening angle of the refrigerator door body, the resolution requirement on the transverse axis direction sub-sensor is high, the longitudinal axis direction sub-sensor plays an auxiliary role in determining the opening angle of the refrigerator door body, and the resolution requirement on the longitudinal axis direction sub-sensor is low.

Fig. 5 (a) -5 (b) are schematic diagrams of measured magnetic field strength variations of RM3100 according to some embodiments of the present application. Referring to fig. 5 (a) -5 (b), the magnitude of the magnetic field strength in the graph is increased by 1, representing a practical increase of 50pT in magnetic field strength. It can be seen from fig. 5 (b) that the magnetic field intensity measured in the direction of the transverse axis of the geomagnetic sensor 11 fluctuates up and down while the position of the magnetic field sensor remains unchangedAbout 0.5%, and the reason for the fluctuation of the magnetic field strength is not the precision of RM3100 itself, but the fluctuation of the geomagnetic field strength of the geomagnetic field is about 0.5%. The magnitude of the total magnetic field strength shown in FIG. 5 (a) is about 10X 10 ⁵ When the resolution of RM3100 is 50pT, the magnitude of the measured geomagnetic actual field strength is about 50uT, and the geomagnetic strength of the known geomagnetic field is about 50uT, which indicates that the RM3100 measurement data matches the actual geomagnetic field strength.

Fig. 6 is a line graph illustrating the variation of magnetic field intensity measured by a magnetic sensor when a door of a refrigerator is simulated to rotate according to some embodiments of the present disclosure. The X axis is the frequency, the Y axis is the measured magnetic field intensity, wherein the magnetic field intensity measured when an interference magnetic field is added to the outside to change the magnetic field around the RM3100 is 0-50 times, the magnetic field intensity measured by the RM3100 collected in the process that the refrigerator door body rotates from the beginning to stay to the preset angle is 100-150 times, and the magnetic field intensity measured by the RM3100 collected from the preset angle to the initial position is 150-300 times. Based on the above, in the practical application process, the magnetic field intensity measured by the RM3100 is relatively stable, the resolution of the RM3100 may be much higher than the resolution required for measuring the magnetic field intensity of the geomagnetic field, and even in the case of not filtering the measured magnetic field intensity in the later period, for example, any one of a median filter, a mean filter, a gaussian filter, a kalman filter, and a combination of the above filtering methods is performed on the measured magnetic field intensity, the angle at which the refrigerator door is opened with high precision can be determined.

In this application, the intelligent refrigerator further includes: an image acquisition unit;

the image acquisition unit is used for acquiring food material images on a rack of the refrigerator door body if the refrigerator door body is opened;

the processor 12 is further configured to obtain the food material image sent by the image acquisition unit if the refrigerator door is opened; calibrating the food material images according to the food material images and the determined opening angles of the refrigerator door body corresponding to the food material images, and acquiring the calibrated food material images; and determining the type of the food material contained in the calibrated food material image through a food material identification model.

In the application, in order to identify food materials conveniently, the intelligent refrigerator is further provided with an image acquisition unit, the image acquisition unit can acquire food material images on a rack of a refrigerator door body in real time when the refrigerator of the intelligent refrigerator is opened, wherein the food material images can be acquired through a camera mounted on the intelligent refrigerator, so that the subsequent processor 12 can perform corresponding processing based on the food material images, and therefore the types of food materials contained in the food material images can be identified.

In a specific implementation process, after detecting that the refrigerator door body is opened, the processor 12 simultaneously acquires the food material image on the refrigerator door body rack acquired by the image acquisition unit and the corresponding magnetic field strength measured by the geomagnetic sensor 11 according to a preset time interval, performs corresponding processing on the magnetic field strength corresponding to the food material image for each food material image, determines an angle at which the refrigerator door body is opened when the food material image is acquired, and then performs subsequent processing based on the food material identification method provided by the application.

The specific process for determining the opening angle of the refrigerator door is the same as the description of the above embodiment, and is not described again here.

Fig. 7 is a schematic structural diagram of an intelligent refrigerator according to some embodiments of the present application. Referring to fig. 7, a color camera is installed in the center of a refrigerator door frame of the intelligent refrigerator, racks are installed on two refrigerator door bodies, and geomagnetic sensors 11 are respectively installed below the two refrigerator door bodies. When the refrigerator door body is opened, the camera of the intelligent refrigerator collects food material images on the refrigerator door body rack according to a preset time interval, and meanwhile, the intelligent refrigerator also records the magnetic field intensity measured by the geomagnetic sensor 11 when the food material images are collected. The intelligent refrigerator carries out corresponding processing based on the magnetic field intensity corresponding to each food material image, and determines the opening angle of the refrigerator door body when each food material image is collected.

In a specific implementation process, after the processor 12 obtains the food material image on the rack of the refrigerator door and the opening angle of the refrigerator door, the angle of the food material image is calibrated according to the opening angle of the refrigerator door, so as to obtain the calibrated food material image.

The specific process of calibrating the angle of the food material image belongs to the prior art, and is not described herein again.

And after the calibrated food material image is obtained, inputting the food material image into a pre-trained food material identification model. And determining the food material types contained in the calibrated food material image through the food material identification model.

Fig. 8 is a schematic diagram of a food material identification process of a specific intelligent refrigerator based on a method for determining an opening angle of a refrigerator door body according to some embodiments of the present application, where the process includes two links of horizontal axis direction calibration of a geomagnetic sensor 11 and food material identification, and each link is described below:

a first link: the transverse axis direction calibration of the geomagnetic sensor comprises the following steps:

s801: the refrigerator door is opened to a preset reference position.

S802: the direction of the transverse axis of the geomagnetic sensor is determined.

After the horizontal axis direction of the geomagnetic sensor is determined based on the above steps, if the geographic position of the intelligent refrigerator does not change, the above steps do not need to be executed again in the process of food material identification.

A second ring section: food material identification, which is to perform subsequent food material identification based on the transverse axis direction of the geomagnetic sensor 11 determined in the above steps, and the process includes the following steps:

s803: if the refrigerator door body is opened, food material images on a rack of the refrigerator door body are obtained, and the opening angle of the refrigerator door body is determined.

S804: calibrating the angle of the food material image according to the food material image and the opening angle of the refrigerator door body, and obtaining the calibrated food material image.

S805: and determining the type of the food material contained in the calibrated food material image through a food material identification model.

According to the method and the device, when the refrigerator door body is opened, the food material images on the rack of the refrigerator door body are collected, the opening angle of the refrigerator door body is determined, the collected food material images are calibrated according to the determined opening angle of the refrigerator door body, the calibrated food material images are obtained, and the food materials contained in the calibrated food material images are obtained through the food material recognition model subsequently, so that the accuracy rate of the recognized food material images is improved.

The present application further provides a method for determining an opening angle of a refrigerator door body, and fig. 9 is a schematic diagram illustrating a process for determining an opening angle of a refrigerator door body according to some embodiments of the present application, where the method includes:

s901: if the refrigerator door is opened, acquiring the magnetic field intensity of a transverse axis of the projection of the magnetic field intensity of the geomagnetic field measured by the geomagnetic sensor at the current position of the geomagnetic sensor in the transverse axis direction, wherein the transverse axis direction is parallel to the ground and is determined according to the magnetic field intensity of the transverse axis.

S902: and determining the opening angle of the refrigerator door body according to a first ratio of the magnetic field strength of the transverse shaft to the pre-stored geomagnetic strength.

The method for determining the opening angle of the refrigerator door body is applied to electronic equipment, and the electronic equipment can be intelligent equipment such as an intelligent refrigerator and can also be a server.

In the present application, the concepts, explanations, details, and other steps related to the method for determining the opening angle of the refrigerator door body in the present application related to the technical solution provided in the present application refer to the descriptions of the foregoing method or other embodiments, and are not repeated herein.

The present application further provides a food material identification method based on the method for determining the opening angle of the refrigerator door body according to the above embodiments, and fig. 10 is a schematic diagram of a food material identification process based on the method for determining the opening angle of the refrigerator door body according to some embodiments of the present application, where the method includes:

s1001: if the refrigerator door body is opened, acquiring food material images on racks of the refrigerator door body and determining the opening angle of the refrigerator door body.

S1002: and calibrating the food material image according to the food material image and the opening angle of the refrigerator door body to obtain the calibrated food material image.

S1003: and determining the type of the food material contained in the calibrated food material image through a food material identification model.

The food material identification method is applied to electronic equipment, and the electronic equipment can be intelligent equipment such as an intelligent refrigerator and can also be a server.

If the electronic equipment for food material identification is an intelligent refrigerator, the intelligent refrigerator directly acquires the collected food material images on the refrigerator door rack and the corresponding magnetic field strength measured by the geomagnetic sensor according to the preset time interval after detecting that the refrigerator door body is opened, performs corresponding processing on the magnetic field strength corresponding to the food material images aiming at each food material image, determines the angle at which the refrigerator door body is opened when the food material images are collected, and then performs subsequent processing based on the food material identification method provided by the application; if the electronic equipment for food identification is a server, the intelligent refrigerator can send a notification instruction for opening the refrigerator door body to the server after detecting that the refrigerator door body is opened, the server determines that the refrigerator door body is opened after receiving the notification instruction, and acquires food material images on a rack of the refrigerator door body sent by the intelligent refrigerator according to a preset time interval, and magnetic field intensity information corresponding to geomagnetic sensor measurement.

This application still provides a refrigerator door body opening angle's determining means, install geomagnetic sensor on the refrigerator door body, figure 11 is the structure sketch map of a refrigerator door body opening angle's determining means that this application some embodiments provided, and the device includes:

an obtaining unit 1101 configured to obtain, if the refrigerator door is opened, a horizontal-axis magnetic field strength measured by the geomagnetic sensor, where the horizontal-axis magnetic field strength is a projection of a magnetic field strength of a current position of the geomagnetic sensor in a horizontal-axis direction, and the horizontal-axis direction is parallel to the ground and is determined according to the horizontal-axis magnetic field strength;

the processing unit 1102 is configured to determine an angle at which the refrigerator door is opened according to a first ratio of the magnetic field strength of the transverse axis to a pre-stored geomagnetic strength.

In the present application, the concept, explanation, detailed description and other steps related to the device for determining the opening angle of the refrigerator door body related to the technical solution provided in the present application refer to the description of the foregoing method or other embodiments, and are not repeated herein.

The present application further provides a food material identification device based on any one of the above-described methods for determining an opening angle of a refrigerator door body, where a geomagnetic sensor is installed on the refrigerator door body, and fig. 12 is a schematic structural view of the food material identification device based on the method for determining an opening angle of a refrigerator door body according to some embodiments of the present application, where the device includes:

an obtaining unit 1201, configured to obtain an image of food materials on a rack of the refrigerator door and determine an angle at which the refrigerator door is opened, if the refrigerator door is opened;

the calibration unit 1202 is configured to calibrate the food material image according to the food material image and the opening angle of the refrigerator door, and obtain a calibrated food material image;

the processing unit 1203 is configured to determine the type of the food material included in the calibrated food material image through a food material recognition model.

In the present application, for the concepts, explanations and detailed descriptions and other steps related to the technical solution provided in the present application related to the food material identification apparatus based on the method for determining the opening angle of the refrigerator door body, please refer to the descriptions of the foregoing methods or other embodiments about these contents, and no further description is given here.

Take the execution subject as an intelligent refrigerator as an example. Fig. 13 is a schematic structural diagram of an intelligent refrigerator according to some embodiments of the present application. Referring to fig. 13, the intelligent refrigerator includes an image recognition unit, a display unit, a network unit, a geomagnetic sensor unit, and a data processing unit.

In the practical application process, a camera, such as an RGB camera and a depth camera, is installed on a general intelligent refrigerator, and the food material images of food materials in the refrigerator, such as the food material images on the rack of the refrigerator door body, can be collected in real time through the camera. After the refrigerator door body is opened, the food material image is collected by the image recognition unit through the camera and then sent to the data processing unit through the network unit.

When food material images are obtained, a geomagnetic sensor in a geomagnetic sensor unit of the intelligent refrigerator can measure the magnetic field intensity of a transverse axis and the magnetic field intensity of a longitudinal axis. And determining the opening angle of the refrigerator door according to the magnetic field intensity of the transverse shaft, the magnetic field intensity of the longitudinal shaft, the reference angle stored in advance and the geomagnetic intensity stored in advance. And then, the determined angle at which the refrigerator door body is opened is sent to a data processing unit through a network unit.

After the data processing unit receives the food material image on the rack of the refrigerator door body and the determined opening angle of the refrigerator door body, the food material image is calibrated according to the opening angle of the refrigerator door body, and then the food materials contained in the calibrated food material image are obtained through a food material identification model.

In order to improve the user experience, after the data processing unit obtains the food materials contained in the food material image, the determined food materials are sent to the display unit through the network unit. The display unit displays the type of the food material on a display screen after receiving the type of the food material.

In addition, in order to further improve the interactivity of the intelligent refrigerator, the display screen can also display the recipe of each food material, and when the transverse axis direction of the geomagnetic sensor installed on the refrigerator door body is determined, the transverse axis magnetic field intensity measured by the geomagnetic sensor is displayed on the display screen.

Fig. 14 is a schematic structural diagram of an electronic device according to some embodiments of the present application, and on the basis of the foregoing embodiments, the present application further provides an electronic device, as shown in fig. 14, including: the system comprises a processor 1401, a communication interface 1402, a memory 1403 and a communication bus 1404, wherein the processor 1401, the communication interface 1402 and the memory 1403 are communicated with each other through the communication bus 1404;

the memory 1403 has stored therein a computer program which, when executed by the processor 81, causes the processor 1401 to perform the steps of:

if the refrigerator door is opened, acquiring a transverse-axis magnetic field intensity of a projection of the magnetic field intensity of the geomagnetism measured by the geomagnetic sensor at the current position of the geomagnetic sensor in a transverse-axis direction, wherein the transverse-axis direction is parallel to the ground and is determined according to the transverse-axis magnetic field intensity;

and determining the opening angle of the refrigerator door body according to a first ratio of the magnetic field strength of the transverse shaft to the pre-stored geomagnetic strength.

Because the principle of the electronic equipment for solving the problems is similar to the method for determining the opening angle of the refrigerator door body, the implementation of the electronic equipment can refer to the implementation of the method, and repeated details are not repeated.

The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface 1402 is used for communication between the above-described electronic apparatus and other apparatuses.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Alternatively, the memory may be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a central processing unit, a Network Processor (NP), and the like; but may also be a Digital instruction processor (DSP), an application specific integrated circuit, a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc.

Fig. 15 is a schematic structural diagram of an electronic device according to some embodiments of the present application, and on the basis of the foregoing embodiments, the present application further provides an electronic device, as shown in fig. 15, including: the system comprises a processor 1501, a communication interface 1502, a memory 1503 and a communication bus 1504, wherein the processor 1501, the communication interface 1502 and the memory 1503 complete communication with each other through the communication bus 1504;

the memory 1503 stores therein a computer program that, when executed by the processor 81, causes the processor 1501 to perform the steps of:

calibrating the food material image according to the food material image and the opening angle of the refrigerator door body to obtain the calibrated food material image;

Because the principle of solving the problems of the electronic equipment is similar to the food material identification method based on the method for determining the opening angle of the refrigerator door body, the implementation of the electronic equipment can refer to the implementation of the method, and repeated parts are not described again.

The communication interface 1502 is used for communication between the above-described electronic apparatus and other apparatuses.

The Processor may be a general-purpose Processor, including a central processing unit, a Network Processor (NP), and the like; but may also be a Digital instruction processor (DSP), an application specific integrated circuit, a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like.

On the basis of the foregoing embodiments, the present application further provides a computer-readable storage medium, in which a computer program executable by a processor is stored, and when the program is run on the processor, the processor is caused to execute the following steps:

if the refrigerator door body is opened, acquiring the magnetic field intensity of a transverse shaft of the projection of the magnetic field intensity of the geomagnetic field measured by the geomagnetic sensor at the current position of the geomagnetic sensor in the transverse shaft direction at preset time intervals, wherein the transverse shaft direction is parallel to the ground and is determined according to the magnetic field intensity of the transverse shaft;

Since the principle of solving the problem of the computer readable medium is similar to the method for determining the opening angle of the refrigerator door body, the implementation of the electronic device can refer to the implementation of the method, and repeated details are not repeated.

Because this application installation earth magnetism sensor on the refrigerator door body, when the treater detects the refrigerator door body and is opened, obtain the magnetic field intensity of earth magnetism at the current position of earth magnetism sensor measured at the cross axle direction's projection cross axle magnetic field intensity according to preset time interval, follow-up based on this cross axle magnetic field intensity, confirm the angle that the refrigerator door body was opened to realized confirming the angle that the refrigerator door body was opened based on the magnetic field intensity of earth magnetism sensor measurement.

if the refrigerator door body is opened, acquiring food material images on a rack of the refrigerator door body according to a preset time interval, and determining the opening angle of the refrigerator door body;

Since the principle of solving the problem of the computer readable medium is similar to the food material identification method based on the method for determining the opening angle of the refrigerator door body, the implementation of the electronic device can refer to the implementation of the method, and repeated details are not repeated.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. An intelligent refrigerator, characterized in that the intelligent refrigerator comprises: a geomagnetic sensor and a processor;

the processor is used for acquiring the magnetic field strength of a transverse shaft measured by the geomagnetic sensor if the refrigerator door body is opened, and determining the opening angle of the refrigerator door body according to a first ratio of the acquired magnetic field strength of the transverse shaft to the prestored geomagnetic strength;

the processor is further configured to, if a trigger operation for adjusting the orientation of a horizontal axis knob of the geomagnetic sensor is received, obtain a magnetic field strength of the geomagnetism measured by the geomagnetic sensor at a current measurement position of the geomagnetic sensor, and a horizontal axis magnetic field strength of a projection in a current candidate horizontal axis direction; determining the direction of the transverse shaft according to the candidate transverse shaft direction corresponding to the maximum transverse shaft magnetic field intensity; storing the determined transverse-axis magnetic field strength corresponding to the transverse-axis direction as the geomagnetic strength;

the processor is specifically configured to determine that the cross-axis magnetic field strength acquired last time is the maximum cross-axis magnetic field strength if it is determined that the cross-axis magnetic field strength acquired last time is not greater than the cross-axis magnetic field strength acquired last time, and the cross-axis magnetic field strength acquired last time is not less than the cross-axis magnetic field strength acquired last time corresponding to the cross-axis magnetic field strength acquired last time.

2. The intelligent refrigerator according to claim 1, wherein the processor is specifically configured to determine an angle corresponding to the first ratio; and taking the angle as the angle for opening the refrigerator door body.

3. The intelligent refrigerator according to claim 1, wherein the processor is specifically configured to determine a correction angle according to a first ratio of the horizontal-axis magnetic field strength to a pre-stored geomagnetic field strength, and a direction of a projection of the magnetic field strength in a vertical-axis direction; and determining the opening angle of the refrigerator door body according to the corrected angle and a preset reference angle.

4. The intelligent refrigerator according to claim 3, wherein the geomagnetic sensor is further configured to measure a vertical axis magnetic field strength of a projection of the magnetic field strength of the geomagnetism in a vertical axis direction;

the processor is specifically configured to, if the refrigerator door is opened, obtain a magnetic field intensity of a longitudinal axis measured by the geomagnetic sensor; if the second ratio of the magnetic field strength of the longitudinal axis to the corresponding absolute value is a positive value, determining that the direction of the projection of the magnetic field strength in the direction of the longitudinal axis is the positive direction of the longitudinal axis; and if the second ratio of the magnetic field strength of the transverse axis to the corresponding absolute value is a negative value, determining the projection direction of the magnetic field strength in the direction of the longitudinal axis as the negative direction of the longitudinal axis.

5. The intelligent refrigerator according to claim 4, wherein the processor is specifically configured to determine a first angle corresponding to the first ratio; and determining the correction angle according to the product of the second ratio and the first angle.

6. The intelligent refrigerator according to any one of claims 3 to 5, wherein the preset reference angle is not less than 30 degrees and not more than 90 degrees.

7. The intelligent refrigerator of claim 1, further comprising: an image acquisition unit;

the processor is further configured to acquire the food material image sent by the image acquisition unit if the refrigerator door is opened; calibrating the food material images according to the food material images and the determined opening angles of the refrigerator door body corresponding to the food material images, and acquiring the calibrated food material images; and determining the type of the food material contained in the calibrated food material image through a food material identification model.

8. A method for determining the opening angle of a refrigerator door body is characterized in that a geomagnetic sensor is mounted on the refrigerator door body, and the method comprises the following steps:

determining the opening angle of the refrigerator door body according to a first ratio of the magnetic field intensity of the transverse shaft to the pre-stored geomagnetic intensity;

the method further comprises the following steps:

if a trigger operation for adjusting the orientation of a transverse-axis knob of the geomagnetic sensor is received, acquiring the magnetic field intensity of the geomagnetism measured by the geomagnetic sensor at the current measurement position of the geomagnetic sensor and the transverse-axis magnetic field intensity of the projection in the current candidate transverse-axis direction; determining the direction of the transverse shaft according to the candidate transverse shaft direction corresponding to the maximum transverse shaft magnetic field intensity; storing the determined transverse-axis magnetic field strength corresponding to the transverse-axis direction as the geomagnetic strength;

the process of determining the maximum transverse-axis magnetic field strength includes:

if the cross-axis magnetic field strength obtained at present is determined to be not more than the cross-axis magnetic field strength obtained last time, and the cross-axis magnetic field strength obtained last time is not less than the cross-axis magnetic field strength obtained last time corresponding to the cross-axis magnetic field strength obtained last time, the cross-axis magnetic field obtained last time is determined to be the maximum cross-axis magnetic field strength.

9. A food material identification method based on the method for determining the opening angle of the refrigerator door according to claim 8, wherein a geomagnetic sensor is mounted on the refrigerator door, and the method comprises the following steps: