CN114322409B

CN114322409B - Refrigerator and method for displaying indoor scenery pictures

Info

Publication number: CN114322409B
Application number: CN202110697463.8A
Authority: CN
Inventors: 孔祥键; 卢可敬
Original assignee: Hisense Visual Technology Co Ltd
Current assignee: Hisense Visual Technology Co Ltd
Priority date: 2021-06-23
Filing date: 2021-06-23
Publication date: 2023-09-19
Anticipated expiration: 2041-06-23
Also published as: CN114322409A

Abstract

The application discloses a refrigerator and a storage room internal view picture display method. The refrigerator provided by the embodiment of the application can respond to the input internal view instruction to obtain at least one group of internal view pictures; and displaying the at least one group of internal scene pictures in a user interface, wherein the display position relationship among the internal scene pictures in the same group is consistent with the position relationship among the accommodating grids corresponding to the internal scene pictures. According to the application, the interior scene picture is displayed to the user, and the arrangement mode of the interior scene picture in the user interface accords with the setting position of the accommodating grid in the storage room, so that the display effect of the interior scene in the storage room is better and more vivid, and the reduction degree is high.

Description

Refrigerator and method for displaying indoor scenery pictures

Technical Field

The application relates to the technical field of refrigerators, in particular to a refrigerator and a storage indoor scenery picture display method.

Background

At present, the intelligent refrigerator with the display can provide multiple functions such as intelligent management, intelligent storage, multifunctional reminding and the like of food materials for users. For example, existing food materials in the refrigerator are presented in a user interface on a display in the form of food material icons, and a user knows which of the existing food materials in the refrigerator are based on the food material icons presented in the user interface. The food material icon may be generated according to food material information corresponding to existing food materials in the refrigerator, where the food material information includes a food material name, a food material shelf life, a food material number, and the like.

For example, after a user puts a new food material into the refrigerator, food material information corresponding to the new food material may be input into the refrigerator through some operations, and further, the refrigerator may display a food material icon of the new food material in the user interface according to the food material information corresponding to the new food material. Alternatively, for intelligent refrigerators having a radio frequency identification (radio frequency identification, RFID) function, the food materials in the refrigerator may be identified by RFID tags. The user can control and display a food editing interface through operation, food information for associating with the RFID tag is input in the food editing interface, and after the refrigerator receives the food information input by the user, association is established with the tag information of the RFID tag and is stored. The tag information of the RFID tag on the food material is read through the antenna built in the storage room, and then the food material information related to the tag can be determined according to the tag information.

Disclosure of Invention

The application provides a refrigerator and a method for displaying indoor scenery pictures.

In a first aspect, the present application provides a refrigerator comprising:

the box body is internally provided with a storage chamber, and the storage chamber comprises a plurality of containing grids for containing food materials;

the box door is arranged at the opening of the storage chamber;

The camera is arranged on one side of the box door, which is positioned in the storage room, and is used for collecting an internal view picture of the storage room;

the display is arranged on the box door and used for displaying a user interface;

a controller coupled to the camera and the display configured to:

responding to an input storage indoor scene viewing instruction, and acquiring at least one group of internal scene pictures, wherein each group of internal scene pictures comprises internal scene pictures of each accommodating grid acquired at the same time;

and displaying the at least one group of internal scene pictures in the user interface, wherein the display position relationship among the internal scene pictures in the same group is consistent with the position relationship among the accommodating grids corresponding to the internal scene pictures.

In a second aspect, the present application also provides an electronic device comprising a memory for storing program instructions and a processor for executing the program instructions to perform the steps of:

responding to an input storage indoor scene viewing instruction, and acquiring at least one group of internal scene pictures, wherein each group of internal scene pictures comprises internal scene pictures of each accommodation grid acquired at the same time;

In a third aspect, an embodiment of the present application further provides a method for displaying an indoor scene picture, which is applied to a refrigerator, where the method includes:

According to the technical scheme, the refrigerator provided by the embodiment of the application has the advantages that the storage room comprises the plurality of containing boxes, and the internal view pictures of the storage room comprise a group of internal view pictures of each containing box shot in the same time period. The refrigerator provided by the embodiment of the application can respond to the input internal view instruction to obtain at least one group of internal view pictures; and displaying the at least one group of internal scene pictures in a user interface, wherein the display position relationship among the internal scene pictures in the same group is consistent with the position relationship among the accommodating grids corresponding to the internal scene pictures. According to the application, the interior scene picture is displayed to the user, and the arrangement mode of the interior scene picture in the user interface accords with the setting position of the accommodating grid in the storage room, so that the display effect of the interior scene in the storage room is better and more vivid, and the reduction degree is high.

Drawings

In order to more clearly illustrate the technical solution of the present application, the drawings that are needed in the embodiments will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.

Fig. 1A is a perspective view of a refrigerator according to an exemplary embodiment of the present application;

fig. 1B is a schematic view showing an external appearance of a refrigerator according to an exemplary embodiment of the present application;

fig. 2 is a schematic view showing a hardware configuration of a refrigerator according to an exemplary embodiment of the present application;

FIG. 3 is a diagram illustrating a food management interface according to an exemplary embodiment of the present application;

fig. 4 is a flowchart illustrating a method for a refrigerator controller to be configured to perform according to an exemplary embodiment of the present application;

FIG. 5 is an intra-scene picture according to an exemplary embodiment of the application;

fig. 6 is a flowchart illustrating a method for a refrigerator controller to be configured to perform according to an exemplary embodiment of the present application;

FIG. 7 is a template view illustrating the present application according to an exemplary embodiment;

FIG. 8 is an illustration of an internal scene showing interface according to an exemplary embodiment of the present application;

FIG. 9 is another internal view presentation interface according to an exemplary embodiment of the present application;

Fig. 10 is a view showing a food management interface displayed on a terminal device according to an exemplary embodiment of the present application;

FIG. 11 is an illustration of an internal scene showing interface displayed on a terminal device according to an exemplary embodiment of the present application;

fig. 12 is a flowchart illustrating a method for a refrigerator controller configured to perform according to an exemplary embodiment of the present application;

FIG. 13 is an illustration of an internal scene showing interface according to an exemplary embodiment of the present application;

FIG. 14 is an illustration of an internal scene showing interface according to an exemplary embodiment of the present application;

fig. 15 is a schematic view illustrating a refrigerator and server interaction process according to an exemplary embodiment of the present application;

fig. 16 is a schematic diagram illustrating another refrigerator-server interaction process according to an exemplary embodiment of the present application.

Detailed Description

In order to make the technical solution of the present application better understood by those skilled in the art, the technical solution of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.

In the description of the present invention, it should be understood that the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like indicate orientations or positional relationships based on the orientation or positional relationships shown in the drawings, merely to facilitate describing the present invention and simplify the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention.

The terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present invention, unless otherwise indicated, the meaning of "a plurality" is two or more.

In the description of the present invention, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.

Fig. 1A is a perspective view of a refrigerator according to an exemplary embodiment of the present application, and the refrigerator according to the embodiment of the present application has an approximately rectangular parallelepiped shape. The refrigerator is defined by a storage compartment 110 providing a limited storage space and a door 120 provided at an opening of the storage compartment. The storage chamber 110 is a case having an opening. In the example shown in fig. 1A, the storage compartment 110 includes a freezing compartment 111 located at a lower side and a refrigerating compartment 112 located at an upper side, the freezing compartment 111 and the refrigerating compartment 112 each have one or more independent storage spaces such as a drawer-type compartment 111A included in the freezing compartment 111, etc., and a plurality of receiving compartments 1121-1123 included in the refrigerating compartment 112, and the top illumination lamp 140 is located inside the receiving compartment 1121.

In the example shown in fig. 1A, a plurality of drawer compartments 111A are capped by drawer doors 111B. The refrigerating compartment 112 is divided into left and right sides, which are respectively covered by a door 112B pivotably mounted on the cabinet.

It should be noted that, fig. 1A is an example of an embodiment of the present application, and the refrigerator shown in this embodiment is not limited to the refrigerator provided by the present application. For example, in further embodiments of the present application, the refrigerator cabinet is divided into left and right sides, defining the left side cabinet as a freezer compartment and the right side cabinet as a refrigerator compartment, both of which may be capped by a door pivotally mounted to the cabinet. As another example, in further embodiments of the present application, the storage compartment includes a refrigerator compartment, a freezer compartment, and a temperature change compartment.

Based on the refrigerator provided by the embodiment, the interior view pictures of the storage room are acquired by arranging one or more cameras in the refrigerator body.

In some embodiments, the camera may be disposed at a side of the door inside the storage compartment, and the lens is disposed facing the storage compartment. Thus, when the door is closed, the camera is opposite to the inside of the storage room, so that an internal view picture of the storage room is taken. It should be understood that a plurality of illumination lamps are provided on the interior wall of the storage compartment, including but not limited to a top illumination lamp provided on the top wall of the storage compartment and a side backlight lamp provided on the side wall of the storage compartment. In order to provide sufficient light conditions for the camera to shoot the indoor scenery pictures, the top illuminating lamp is kept in an on state and the side backlight lamp is turned off in the working process of the camera.

As shown in fig. 1A, a camera mounting plate 130 is disposed at one side edge of the refrigerating chamber door 112B, and cameras 131, 132 and 133 are embedded in the camera mounting plate 130, and the cameras 131, 132 and 133 are connected to a controller (see, in particular, the controller 220 and description thereof in fig. 2) of the refrigerator to operate under the control of the controller. When the refrigerating compartment door 112B is closed, the cameras 131 to 133 face the receiving compartments 1121 to 1123 included in the refrigerating compartment 112, respectively, and thus, the cameras 131 to 133 can be used to capture the interior view pictures of the receiving compartments 1121 to 1123, respectively.

It should be noted that fig. 1A is an example of the embodiment of the present application, and the number, the installation positions and the installation manners of the cameras are not limited to the refrigerator provided by the present application. For example, in another embodiment of the present application, a camera mounting plate 130 is provided at an edge of one side of the refrigerating chamber door 112B near the freezing chamber door 111B, a mechanical sliding rail and a driving motor are provided in the camera mounting plate 130, the camera is slidably connected to the mechanical sliding rail through a slider, and the driving motor is connected to the slider, so that the camera can slide in a vertical direction along the mechanical sliding rail under the driving of the driving motor. The controller is connected with the driving motor. When the interior picture of the refrigerating chamber needs to be collected, the driving motor is controlled to enable the camera to move to the position opposite to the first layer (or the third layer) containing grid, the camera is controlled to collect the interior picture of the containing grid, after the collection is completed, the driving motor is controlled to enable the camera to move to the position opposite to the second layer containing grid, and the camera is controlled to collect the interior picture of the containing grid until the interior picture of all the containing grids is collected.

Based on the refrigerator provided by the embodiment, the refrigerator can have a display function by arranging the display outside the refrigerator body.

Fig. 1B is a schematic view showing an external appearance of a refrigerator according to an exemplary embodiment of the present application, and as shown in fig. 1B, a display is provided on a refrigerating chamber door 112B, the display being embedded in a door body. The area of the refrigerating compartment door 112B located below the display is provided with a tag reading area in which an antenna for reading an RFID tag is built for recognizing the RFID tag at a close distance.

In some embodiments, the plane of the display screen is flush with the outer surface of the door.

As shown in fig. 1A and 1B, the refrigerator according to the embodiment of the present application is provided with at least two storage chambers, such as a freezing chamber, a refrigerating chamber, a temperature changing chamber, etc., inside the refrigerator. Each storage compartment may have a plurality of independent storage spaces therein, such as drawer compartments, receiving compartments.

Fig. 2 is a schematic view showing a hardware configuration of a refrigerator according to an exemplary embodiment of the present application. As shown in fig. 2, the refrigerator 200 may include a display 210, a controller 220, an antenna 230 for detecting an RFID tag, a detector 240, a camera 250, a speaker 260, a memory 270, and a user input interface. The display 210, antenna 230, detector 240, camera 250, speaker 260, and memory 270 are coupled to the controller 220 via a communication interface.

The display 210 is configured to receive an image signal output from the controller 220, perform a component for displaying video contents and images and a menu manipulation interface, and display a user manipulation UI interface for controlling the refrigerator 200.

The controller 220 may include one or more processing units, such as a system on a chip (SoC), a central processing unit (central processing unit, CPU), a Microcontroller (MCU), a memory controller, and the like. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

In some embodiments, the controller 220 communicates with the antenna 230 via a serial port.

In some embodiments, the controller 220 includes an RFID module, the antenna 230, and the RFID tag form an RFID read/write system, the antenna 230 is used to transmit radio frequency signals between the RFID module and the RFID tag, and the RFID module performs read/write operations on the RFID tag through the antenna 230.

In some embodiments, the RFID module communicates with the antenna through a serial port.

Memory 270, which may include one or more memory units, may include, for example, volatile memory (volatile memory), such as: dynamic random access memory (dynamic random access memory, DRAM), static random access memory (static random access memory, SRAM), etc.; non-volatile memory (NVM) may also be included, such as: read-only memory (ROM), flash memory (flash memory), and the like. The different memory units may be separate devices or may be integrated or packaged in one or more processors or communication interfaces as part of the processors or communication interfaces.

The memory 270 stores program instructions and application programs such as voice applications, food management applications. The controller 220 may call program instructions in the memory 270 or run an application program to cause the refrigerator to perform a related method, such as the method provided by the embodiments of the present application.

Camera 250 is used to capture images of the interior of the storage compartment, including but not limited to capturing images of the interior, for example, an interior video may also be captured. Alternatively, camera 250 may employ a wide angle lens. Since the focal length of the wide-angle lens is short, a large-area interior view of the storage room can be photographed within a short photographing distance range. Alternatively, the camera 250 may employ a common wide-angle lens, which typically has a focal length of 38-24 mm and a viewing angle of 60-84 degrees, or an ultra-wide-angle lens, which typically has a focal length of 20-13 mm and a viewing angle of 94-118 degrees.

The detector 240, including at least a sound collector, such as a microphone, may be used to receive the user's sound. Illustratively, the voice password used by the user to control the refrigerator 200 is collected by a microphone. The detector 240 may further include a door opening/closing state sensor, such as a detecting magneto-sensitive switch, a mechanical switch, for detecting an opening/closing signal of each door, and transmitting opening/closing state information to the controller 220 when the opening/closing state of the door is recorded according to the detected opening/closing signal. Illustratively, when the detector 240 detects an open signal or a close signal of any one of the doors, the latest switch state information is transmitted to the controller 220.

A user input interface including at least one of a microphone, touch pad, sensor, key, etc. Such as: the user may implement a user command input function through actions such as voice, touch, gesture, press, etc., and the input interface converts the received analog signal into a digital signal, and converts the digital signal into a corresponding command signal, and sends the command signal to the controller 220.

In some embodiments, the controller 220 may include a voice recognition module, which further includes a voice parsing unit and a voice command database, so that the refrigerator can independently perform voice recognition of voice data inputted by a user and a process of matching the recognized voice content with voice commands in the voice command database.

The number of antennas 230 is not limited in the present application. For example, the antenna 230 may include an antenna disposed in each storage compartment and an antenna disposed inside the tag reading area, wherein the antenna disposed in each storage compartment is mainly used to scan the RFID tag in each storage compartment to read the tag information of the RFID tag, and the antenna disposed inside the tag reading area is mainly used to scan the RFID tag on the tag reading area to read the tag information of the RFID tag.

In some embodiments, the power of the antennas disposed in different storage compartments may be different. And moreover, the power of the antenna of each storage room can ensure that the antenna can identify food materials contained in the room of the antenna, so that the false identification rate can be increased when the power is too high, and the food materials not contained in the room of the antenna can be identified when the power is too low. In a specific implementation, the power of the antenna may be determined in advance according to the space size of each storage chamber, and the power of each antenna may be set in the refrigerator, the larger the space, the larger the power.

In some embodiments, the antenna used to scan the tag reading area is referred to as a first antenna, and the antenna disposed in the storage compartment is referred to as a second antenna.

In order to achieve intelligent management of food materials, RFID tags may be used to identify food materials in the storage. By establishing the corresponding relation between the RFID tag and the food material information, virtual association between the RFID tag and the food material information is realized. By modifying the corresponding relation between the RFID tag and the food material information, the same RFID tag is related to different food material information in sequence, and the use flexibility of the RFID tag is improved. And storing the food materials with the RFID tags in the refrigerator, detecting the RFID tags in the storage room through the antenna to obtain tag information, and determining the food materials identified by the tags according to the association relation between the tag information and the food material information.

In some embodiments, the tag information includes tag codes, and particular code segments in the tag codes are tag identifications. The corresponding relation between the identification information of the RFID tag and the food material information is established, so that virtual association of the RFID tag and the food material can be realized. Further, after the tag identification of the RFID tag in the storage compartment is obtained, the food corresponding to the RFID tag can be determined according to the tag identification and the pre-established correspondence.

In some embodiments, the controller 220 implements the food management function by running a food management application that provides an application interface comprising: the system can be used as a food material management interface of an application main interface, a label editing interface for establishing the association relation between an RFID label and food material information, a food material warehouse interface for displaying food material information in a preset food material warehouse, a food material adding interface for adding food materials, an internal scene display interface for displaying internal scenes of a storage room, and the like.

In some embodiments, the controller 220 initiates the food management application in response to a power-on or user-entered food management application initiation instruction. After the food management application is started, the user can input a label editing instruction in the food management application. In response to the tag edit instruction, the food management application scans the RFID tag placed on the tag reading area by the user through the first antenna to obtain tag information, such as "a", of the RFID tag, and displays a tag edit interface. The user may enter food material information in the label editing interface for association with "a". The food material management application receives food material information input by a user, and establishes an association relationship between the food material information and 'A'. When the food material management application detects that the RFID tag 'A' exists in the storage room through the second antenna, real food materials in the storage room can be determined according to food material information related to the RFID tag 'A', and therefore intelligent management of the food materials is achieved. It should be appreciated that when the food material management application detects the presence of the RFID tag "a" in the storage compartment via the second antenna, the food material associated with "a" is the existing food material in the storage compartment.

In other embodiments, the user may enter a food addition instruction in the food management application. The food material management application responds to the food material adding instruction and displays a food material warehouse interface, wherein the food material warehouse interface comprises a plurality of selectable food material icons, and each food material icon corresponds to one piece of food material information. The user may select a food material (icon) to be added in the food material library interface. The food material management application determines the food material information corresponding to the food material icon selected by the user as the food material information of the existing food material in the storage room. For example, food material information corresponding to a food material icon selected by a user is added to a local food material information set, which is a set of food material information of existing food materials in a storage room. According to the food material adding operation, the RFID tag does not need to be attached to the real food material, and the food material information of the real food material is input into the food material management application, so that intelligent management of the food material in the refrigerator is realized through the food material management application.

In some embodiments, the food material information includes one or more of a food material ID, a food material name, a storage location identification, a number of food materials, a food material shelf life, a start time, and the like. The storage room for storing the food materials can be determined according to the storage position identification. For example, if the storage location identification contained in the food material information "apple" is "1", it is determined that "apple" is stored in the refrigerator compartment, and if the storage location identification contained in the food material information "apple" is "2", it is determined that "apple" is stored in the freezer compartment. The freshness of the food material may be determined based on the current time, the shelf life of the food material, and the starting time. The starting time may be the actual production date of the food material input by the user, such as the production date 2021 of the bread, 5 months and 1 days, or the time of placing the food material into the storage room.

In some embodiments, food material information corresponding to existing food materials in the storage compartment is stored in a local food material information set. The existing food material refers to food material currently stored in a refrigerator.

In other embodiments, the local food material information set may further store food material information corresponding to the historical food material. The history food material refers to food material stored in the refrigerator in the past and having been currently removed from the refrigerator. In particular, the existing food material and the history food material may be distinguished by storing the status flag, for example, the food material information of the history food material is marked as a moving-out status, and the food material information of the existing food material is marked as a putting-in status.

In other embodiments, the food material information of the existing food material and the food material information of the historical food material are stored in different sets of food material information.

In some embodiments, after the food management application is started, the food material information of the existing food material in the local food material information set, for example, the food material information in which the storage state is the put state, is read. And then displaying a food material management interface according to the food material information of the existing food material, wherein the food material management interface comprises food material icons drawn according to the food material information, so that the existing food material in the refrigerator is displayed in the form of icons, pictures and texts and the like as many as possible. In addition, various functional controls, such as an RFID tag editing control, a food adding control, a menu recommending control, an internal view control, etc., can be displayed in the food management interface, so that the user can operate to trigger the controller 220 to realize corresponding functions.

In some embodiments, the food management interface may include a plurality of food display areas, one food display area corresponding to each storage compartment. For example, a food material icon of the food material in the refrigerating chamber is displayed in the food material display area corresponding to the refrigerating chamber.

Fig. 3 is a user interface, specifically an exemplary food material management interface, according to an exemplary embodiment of the present application. Referring to fig. 3, the food management interface includes an RFID tag editing control 301, an internal view control 302, an add food control 306, and further includes food display areas corresponding to respective storage compartments, such as a food display area 303 corresponding to a refrigerating compartment, a food display area 304 corresponding to a temperature changing compartment, and a food display area 305 corresponding to a cooling compartment. Each of the food material display areas displays therein a food material icon corresponding to each food material in the storage room, such as food material 1, food material 2, and food material 3 in the food material display area 303, food material 4 and food material 5 in the food material display area 304, and food material 6 and food material 7 in the food material display area 305. Each food material icon includes a food material picture and a food material name.

In some embodiments, a user may enter the above-described label editing instructions by manipulating RFID label editing control 301 in the food management interface. The user can input the above-described food material addition instruction by operating the add food material control 306 in the food material management interface. The user can input an internal view instruction by operating the internal view control 302 in the food management interface.

In some embodiments, the food management application captures an interior view picture of the storage compartment through the camera and displays the interior view to the user by displaying the interior view picture. And identifying food material objects in the internal scene picture based on the image identification technology so as to obtain food material information of the existing food materials in the storage room. It is noted that when the food material object in the internal view picture is a new food material, the function of automatically adding the food material can be realized based on the food material information obtained by image recognition, so that the user operation of manually inputting the food material information is omitted; in addition, the existing food materials in the storeroom can be displayed to the user based on the food material information obtained through image recognition, and user experience is improved.

It should be understood that the storage room generally includes a plurality of independent storage spaces, and an internal view of each storage space is a local internal view of the storage room, and an internal view image of each storage space is a local internal view image of the storage room. Therefore, it is possible to obtain a local interior view picture of the storage room by taking an interior view picture of each storage space, and reflect the local interior view or the global interior view of the storage room by partially or completely taking the interior view picture of the storage space.

Taking the refrigerator shown in fig. 1A as an example, cameras 131 to 133 are installed on the refrigerating chamber door. When the refrigerating chamber door is closed, the lenses of the cameras 131 to 133 face the accommodation cases 1121 to 1123, respectively. The food material management application can respectively collect the internal view pictures corresponding to the holding grids 1121-1123 through the cameras 131-133, and the internal view pictures are taken as the internal view pictures of the refrigerating chamber.

In some embodiments, the controller 220 performs the following steps shown in fig. 4 by running a food material management application:

and S410, when the preset triggering condition is met, controlling the camera to acquire the internal view picture of the storage room.

In some embodiments, the camera is controlled to capture an interior view of the storage compartment when a door closing signal is detected. It is noted that the door switch signal can be selectively monitored according to the setting position of the camera. For example, with the refrigerator shown in fig. 1A, since the camera is provided on the cooling chamber door, the food management application can detect the switching signal of the cooling chamber door without detecting the switching signals of other doors. When a closing signal of the refrigerating chamber door is detected, the cameras 131-133 are controlled in sequence, and the internal view pictures corresponding to the accommodating grids 1121-1123 are collected to obtain a group of internal view pictures. It should be understood that a group of interior pictures is an interior picture of a different part of the storage room collected in the same time period. For example, the intra-scene pictures corresponding to the accommodation boxes 1121 to 1123 collected in the same period of time constitute a group of intra-scene pictures.

In particular, the controller 220 detects the opening and closing signals of the respective doors through the detector 240. Alternatively, when the detector 240 detects an open signal or a close signal of any one of the doors, the open/close state of the doors is recorded. And sending switch state information to the controller, wherein the switch state information comprises the last two switch states of each box door, namely a prior state and a later state. The controller determines whether an opening signal or a closing signal of the door is detected according to the opening and closing state information. For example, when the previous state of a certain door is an open state and the latter state is a closed state, determining that a closing signal of the certain door is detected; when the previous state of a certain box door is a closed state and the later state is an open state, determining that an opening signal of the box door is detected; when the previous state and the subsequent state of a certain door are the same state, it is determined that the switching signal of the door is not detected. Illustratively, the contents of each field in the switch status information that the detector sends to the controller are as follows, wherein the controller can distinguish the doors based on the field "Index".

In other embodiments, the camera is controlled to acquire the interior view picture of the storage room when the current time reaches a preset time point or the interval time reaches a preset time length. For example, food management applications control cameras to capture interior pictures of a storage room at 8:00, 12:00, 18:00 a day. For another example, the food management application controls the camera to collect the interior pictures of the storage room every 24 hours.

In other embodiments, the camera is controlled to capture the interior view of the storage compartment when an input interior view capture instruction is received. The user can input an internal scene picture acquisition instruction by clicking an operation control on the food material management interface or by inputting a voice password.

In an alternative implementation of S410, the food management application includes a service for controlling the camera in the background, through which the food management application controls the camera to capture an interior view of the storage compartment. For example, when a preset trigger condition is satisfied, the background service turns on the camera; and after the first duration of the camera opening, controlling the camera to shoot an internal scene picture of the storage room. Therefore, the problems of frame loss and low brightness which are very easy to occur immediately after the camera is started can be avoided, and the picture shooting effect is ensured. Illustratively, the first duration is 1 second.

It should be noted that the control logic of the food management application for the cameras is related to factors such as the number of cameras and the setting positions thereof. For example, for the refrigerator shown in fig. 1A, when a preset trigger condition is satisfied, the cameras 131 to 133 need to be controlled in sequence to collect the interior view pictures of the corresponding accommodation compartments. For example, firstly, the camera 131 is turned on, after waiting for 1 second, the camera 131 is controlled to shoot an internal view picture of the accommodating grid 1121, and after shooting is completed, the camera 131 is turned off; then, the camera 132 is started, after waiting for 1 second, the camera 132 is controlled to shoot an internal view picture of the accommodating grid 1122, and after shooting is completed, the camera 132 is closed; finally, the camera 133 is turned on, after waiting for 1 second, the camera 133 is controlled to shoot an internal view picture of the accommodating grid 1123, and after shooting is completed, the camera 133 is turned off. Finally, the internal view pictures of the accommodation grids 1121-1123 are obtained.

However, for refrigerators different from the example structure shown in fig. 1A, the control logic of the food management application for the camera may be different. For example, if the camera can slide along the mechanical sliding track on the camera mounting board 130 (see the foregoing embodiment specifically), when the preset triggering condition is met, the camera needs to be controlled to move to the position opposite to each accommodation lattice successively, and then the camera is controlled to capture the internal view picture of the corresponding accommodation lattice, which is not described herein.

In some embodiments, the operating system installed on the refrigerator is an android system. Based on the working mechanism of the android system, after the camera is started, the internal scene picture can be shot only on the premise of outputting the preview picture. Based on the above, after the food management application starts the camera, controlling to be smaller than the window size of the preset size, displaying a preview window in the area of the user interface, which is positioned outside the view port range of the display, and outputting the preview picture of the camera to the preview window. The size of the preview window is smaller than the preset size and is displayed outside the range of the visual port of the display, so that the preview picture of the camera can be output under the condition that a user does not feel, and the preview window is prevented from influencing the user to view the current user interface and operate the controls in the user interface.

For example, the size of the preview window may be set to 1×1 (smaller than a preset size), and the margin value may be set to a negative value. It should be appreciated that the aforementioned margin value represents the distance between the display location of the preview window and the boundary of the user interface, and when the margin value is negative, the preview window will be displayed outside the viewport of the display of the user interface.

S420, judging whether the detector detects a door opening signal in the process of collecting the interior scene picture by the camera.

In some embodiments, the camera is limited by the position of the camera, and the camera can face the inner space of the storage chamber only when the door is in a closed state, and the camera can shift along with the change of the state of the door when the door is in an open state. For example, with the refrigerator exemplarily shown in fig. 1A, the cameras 131 to 133 are respectively faced with the receiving compartments 1121 to 1123 if and only if the refrigerating chamber door is in a closed state. If the detector detects the opening signal of the refrigerating chamber door in the process of collecting the internal scene picture by the camera, the user opens the refrigerating chamber door, so that the camera is not opposite to the accommodating grid, and the internal scene picture collected by the camera is most likely to be an invalid picture. The invalid picture is a picture which cannot meet the requirements of image recognition and identification and internal view display, such as a picture which does not clearly and completely record food images in a storage room. Correspondingly, the effective picture is a picture which does not meet the requirements of image recognition and internal view display. It is worth noting that if the operation that the user opens the door occurs in the process of collecting the internal scene picture by the camera, the camera is very easy to shoot the user and family scenes, so that the privacy of the user is revealed, and the user is puzzled.

Based on the detection, whether the door is opened or not by judging whether the detector detects the opening signal of the door in the process of collecting the internal scene picture by the camera or not, and whether the user opens the door or not in the process of collecting the internal scene picture by the camera or not is determined; if the user opens the box door once in the process of collecting the internal scene picture by the camera, the internal scene picture collected at this time can be determined as an invalid picture.

In some alternative implementations, the door opening and closing state of the door is monitored in real time during operation of the camera. When the door is detected to be opened, the camera is closed. For example, the food management application polls the door opening and closing state information sent by the detector to the controller, thereby acquiring the real-time opening and closing state of the door. When a certain box door is changed from a closed state to an open state and the box door is the box door provided with the camera, the camera is closed. For example, with the refrigerator shown in fig. 1A, when it is acquired that the refrigerating chamber door is changed from the closed state to the open state, the camera is turned off.

In other alternative implementations, the completion time is recorded when the acquisition of the foreground picture is completed. And determining the time when the detector detects the opening signal of the door, namely the door opening time, which is earlier than the completion time, according to the latest door opening and closing state information sent by the detector. The time difference between the completion time and the aforementioned door opening time is then determined. If the time difference is smaller than the preset time difference, determining that the opening signal of the box door is detected in the process of acquiring the internal scene picture by the camera; if the time difference is not smaller than the preset time difference, determining that the opening signal of the box door is not detected in the process of collecting the internal scene picture by the camera. The preset time difference can be determined according to the time required by the camera to acquire the internal scene picture. For example, if the camera requires 10 seconds of acquisition time, the preset time difference may be 10 seconds. In addition, when the last internal view picture shot by the camera is acquired, the internal view picture can be considered to be acquired. For example, taking the refrigerator shown in fig. 1A as an example, assuming that the control sequence of the cameras 131 to 133 by the food management application is the camera 131, the camera 132, and the camera 133, then the internal view picture shot by the camera 133 is the last internal view picture, and when the internal view picture shot by the camera 133 is acquired, the internal view picture is considered to be acquired. Or, when the camera is closed, the acquisition of the internal scene picture can be considered to be completed. Furthermore, the completion time of the recording may be the shooting time of the last internal view picture or the closing time of the camera.

In other alternative implementations, it is determined whether a door opening signal is detected based on door opening/closing status information received during an operational period of the camera. Taking the refrigerator shown in fig. 1A as an example, assume that the control order of the cameras 131 to 133 by the food management application is the camera 131, the camera 132, the camera 133. The food management application records the opening time when opening the camera 131, and records the closing time when closing the camera 133, so as to determine the working time period of the camera according to the opening time and the closing time. After the camera 133 is turned off, door opening and closing state information received in the aforementioned operation period is acquired. If the acquisition result is empty, determining that the detector does not detect the opening signal of the box door in the process of acquiring the internal scene picture by the camera. If the acquired result is not empty and the door opening and closing state information of the door shows that the door of the refrigerating chamber is changed from the closed state to the open state, the detector detects an opening signal of the door in the process of determining that the camera acquires the interior picture.

It can be seen that the above embodiment determines the validity of the captured internal view picture by determining whether the detector detects the opening signal of the door during the process of capturing the internal view picture by the camera. It should be noted that, in other embodiments of the present application, other ways may be used to analyze the validity of the foreground image. For example, whether the internal scene picture contains specific image features is identified based on an image identification technology, if the internal scene picture contains the specific image features, the acquired internal scene picture is determined to be a valid picture, otherwise, the acquired internal scene picture is determined to be an invalid picture. The specific image feature may be a feature of a scene in the storage room shared by a certain number of effective pictures, such as a feature of a partition board forming the accommodating grid, including but not limited to a shape, a color, a size and the like of the partition board.

In S420, if it is determined that the detector detects the opening signal of the door during the process of capturing the interior picture by the camera, S430 is performed, and if it is determined that the detector does not detect the opening signal of the door during the process of capturing the interior picture by the camera, S440 is performed.

It should be noted that, in some embodiments, if the setting position of the camera is not affected by the door opening and closing state, the step of determining the validity of the foreground image may be skipped, i.e. S420-S440 is skipped, and the processing procedure for the foreground image is directly performed, for example, in some embodiments, the flow goes from S410 to S610 in fig. 6 or the flow goes to S121 in fig. 12. In some embodiments, if the setting position of the camera is not affected by the opening and closing state of the door, the validity of the foreground picture is determined in other manners.

S430, if the detector detects the opening signal of the box door in the process of collecting the internal scene picture by the camera, determining that the collected internal scene picture is an invalid picture.

In the example based on the refrigerator shown in fig. 1A, if an opening signal of the refrigerating chamber door is detected in the process of collecting the interior picture, the interior pictures taken by the cameras 131 to 133 in the period of time are all determined as invalid pictures.

In some alternative implementations, in the case that the intra-scene picture is determined to be an invalid picture, the invalid picture is deleted to avoid revealing privacy and occupying the storage space of the refrigerator.

In some embodiments, the camera is controlled to re-capture the interior view picture of the storage compartment when the captured interior view picture is determined to be an invalid picture. In these embodiments, determining that the acquired intra-scene picture is an invalid picture is a preset trigger condition for controlling the camera to acquire the intra-scene picture.

S440, if the detector does not detect the opening signal of the box door in the process of collecting the internal scene picture by the camera, determining that the collected internal scene picture is a valid picture.

In the example based on the refrigerator shown in fig. 1A, if the opening signal of the refrigerating chamber door is not detected in the process of collecting the interior picture, the interior pictures photographed by the cameras 131 to 133 during the period of time are all determined as valid pictures.

In other embodiments, in the case that the opening signal of the door is not detected in the process of determining to collect the internal view pictures, whether the format size of each internal view picture is a preset format size or not is further determined, or whether the size of each internal view picture is larger than a preset minimum value or not is determined. And if the format size of each internal scene picture is a preset format size or the size of each internal scene picture is larger than a preset minimum value, determining that the acquired internal scene picture is an effective picture. And if the format size of at least one internal scene picture is not the preset format size or the size is not larger than the preset minimum value, determining that the group of internal scene pictures are invalid pictures. In the example based on the refrigerator shown in fig. 1A, if the format sizes of the internal view pictures shot by the cameras 131-133 are all the preset format sizes, or all the sizes are larger than the preset minimum value, it is determined that the group of internal view pictures are all effective pictures. If the format size of at least one of the internal view pictures collected by the cameras 131-133 is not the preset format size or is not greater than the preset minimum value, determining that the group of internal view pictures are all invalid pictures. In these embodiments, by determining the format size or the size of each of the intra-scene pictures, the accuracy of determining the validity of the pictures can be further improved.

In the embodiment of the application, under the condition that the internal scene picture is determined to be an effective picture, the effective picture is stored.

According to the refrigerator provided by the embodiment of the application, when the preset trigger condition is met, the camera is used for collecting the internal view picture of the storage room, so that the internal view picture is displayed for the user. And judging the effectiveness of the acquired internal scene picture by judging whether the detector detects the opening signal of the box door in the process of acquiring the internal scene picture by the camera. If the detector detects the opening signal of the box door in the process of collecting the internal scene picture by the camera, determining that the collected internal scene picture is an invalid picture. Not only can ensure that the acquired internal view picture can meet the image recognition requirement and the internal view display requirement, but also can avoid revealing the hidden of the user

In some embodiments, in order for the camera to be able to capture an entire view of the housing grid, the capture mode of the camera is set to a wide angle mode. However, this results in that when a certain cell is photographed, the upper cell and/or the lower cell of the cell are photographed at the same time, so that there is a redundant area of the photographed internal view picture. In order to remove redundant areas in the internal view picture, the internal view picture can be cut according to a preset cutting rule, so that only key areas of the internal view picture are reserved. The key area is the image area corresponding to the accommodation lattice.

It should be noted that, when the installation position, the shooting focal length and/or the shooting angle of the camera are different, the size and the position of the key area in the internal view picture are different, so that the preset clipping rule can be determined according to the installation position, the shooting focal length, the shooting angle and the like of the camera.

Taking the camera 132 shown in fig. 1A as an example, refer to fig. 5. In fig. 5, (a) shows an internal view of the housing compartments 1121 to 1123 in the refrigerator shown in fig. 1A. If the lower boundary of the shooting range of the camera 132 is flush with the bottom partition board of the accommodating compartment 1122, as shown in fig. 5 (b), it can be seen that, in the inner view picture (b), the region b1 is an image region corresponding to the accommodating compartment 1122, that is, a key region, and the region b2 is a redundant region. For the intra picture (b), the redundant area b2 may be removed using a down clipping rule. If the upper boundary of the shooting range of the camera 132 is flush with the top of the receiving compartment 1122, as shown in fig. 5 (c), it can be seen that, in the internal view picture (c), the region c1 is an image region corresponding to the receiving compartment 1122, that is, a key region, and the region c2 is a redundant region. For the intra picture (c), an on-going cropping rule may be employed to remove the redundant area c2. If the lower boundary of the shooting range of the camera 132 is lower than the bottom partition of the receiving grid 1122 and the upper boundary is higher than the top partition of the receiving grid 1122 by Yu Rongna grid, then the shot internal view picture will be shown in fig. 5 (d), and it can be seen that, in the internal view picture (d), the region d1 is the image region corresponding to the receiving grid 1122, that is, the key region, and the regions d2 and d3 are both redundant regions. For the intra picture (d), then a centered clipping rule may be employed to remove the redundant areas d2 and d3.

It can be appreciated that the cropping rules adopted by the foreground pictures in different accommodation boxes can be the same or different. For example, the internal view pictures of the accommodation boxes 1121 and 1122 are cut using an on-the-fly cutting rule, and the internal view picture of the accommodation box 1123 is cut using an on-the-fly cutting rule.

It can be further understood that, since the sizes of the different receiving compartments may be the same or different, the focal lengths adopted by the camera when shooting the internal views of the different receiving compartments may be the same or different, so that the critical area sizes in the different internal view pictures are the same or different, i.e. the sizes of the cut internal view pictures are the same or different.

In some embodiments, after removing redundant areas of each of the intra-scene pictures, a location identifier is assigned to each of the intra-scene pictures. The location identifier is used for representing the relevant information of the accommodation lattice, such as a location, a name and the like, and the location identifier can be a character containing the relevant information of the accommodation lattice or a character having an association relation with the relevant information of the accommodation lattice. Because a group of internal view pictures respectively correspond to different accommodating grids, different internal view pictures in the group of internal view pictures have different position identifications. Furthermore, according to the position identification, the accommodation lattice corresponding to the interior picture can be determined. Illustratively, for the refrigerator shown in fig. 1A, a first location identifier is assigned to the interior view picture of accommodation compartment 1121, a second location identifier is assigned to the interior view picture of accommodation compartment 1122, and a third location identifier is assigned to the interior view picture of accommodation compartment 1123.

In some embodiments, the same time identifier is assigned to the intra-scene pictures taken during the same time period, by which the time of taking the group of intra-scene pictures is characterized.

In some embodiments, the picture name of the intra picture is generated from the location identification of the intra picture such that its location identification is included in its picture name. Illustratively, the picture name of the internal view picture of the accommodation compartment 1121 may be "01.interior of the refrigerator"; the picture name of the intra-scene picture of the accommodation compartment 1122 may be "02.interior of the refrigerator"; the picture name of the inside view picture of the accommodation compartment 1123 may be "03.interior of the refrigerator". The "01", "02" and "03" in the picture names are location identifiers.

In some embodiments, the picture name of the intra-scene picture is generated according to the position identifier and the time identifier of the intra-scene picture, so that the position identifier and the time identifier of the intra-scene picture are contained in the picture name of the intra-scene picture. Illustratively, the picture name of the internal view picture of the accommodation compartment 1121 may be "01.202106051200"; the picture name of the intra-scene picture of the accommodation compartment 1122 may be "02.01.202106051200"; the picture name of the inside view picture of the accommodation compartment 1123 may be "03.01.202106051200". The picture name is 202106051200, namely a time mark, and the characteristic time of the picture is 2021, 6, 5, 12:00.

In some embodiments, the intra-scene picture with the picture name described above is saved in the intra-scene picture collection. The intra-scene pictures with the same time mark form a group of intra-scene pictures. The position identifiers of the intra-scene pictures in each group of intra-scene pictures are different. The intra-scene pictures in the collection may be arranged in chronological order. When the refrigerator interior is required to be displayed to the user, one or more groups of interior pictures with the latest shooting time can be extracted from the collection according to the time identification or the picture storage time and displayed in the user interface.

Optionally, when the number of pictures in the set of the internal scene pictures exceeds an upper threshold, deleting one or more groups of internal scene pictures with earliest shooting time in the set according to the time identifier, so as to ensure that the number of pictures in the set is always less than the upper threshold. Optionally, the food material management application periodically detects whether an expired picture is generated in the set of the internal scene pictures according to the time identifier, and deletes the expired picture when the expired picture is generated in the set. The expired picture is a picture of which the time interval between the shooting time and the current time exceeds a preset interval. For example, the preset interval may be 7 days, so that the food management application may delete the internal view pictures taken 7 days before in the collection periodically.

In some embodiments, the intra-scene picture with the picture name is stored in the intra-scene picture set, and meanwhile, the intra-scene picture with the picture name is sent to the server together with the identity information of the refrigerator. And the server stores the group of internal scene pictures under a path associated with the identity information according to the identity information. The identity information of the refrigerator can be account information of a food management application logged in the refrigerator, or can be equipment information of the refrigerator. Optionally, before sending the intra-scene picture to the server, a compression operation is performed on the intra-scene picture to be sent, so as to reduce the volume of the intra-scene picture. And then, the compressed interior picture and the identity information of the refrigerator are sent to a server, so that the data transmission quantity is reduced, the resources are saved, and the storage space is saved.

In some embodiments, the user may control the refrigerator to display the storage interior scene picture by inputting a storage interior scene viewing instruction. The controller 220 is configured to perform the following steps shown in fig. 6:

s610, responding to an input storage indoor scene viewing instruction, acquiring at least one group of internal scene pictures, wherein each group of internal scene pictures comprises internal scene pictures of each accommodation lattice acquired in the same time period.

In some embodiments, the user enters a storeroom view instruction in the food material management application. For example, the user may enter a storage interior view instruction by clicking on an interior view control in the food management interface shown in fig. 3.

In other embodiments, the user inputs the storage indoor view instruction using a voice control function of the refrigerator. For example, the user speaks a preset voice password, the intelligent voice module receives the voice password through the microphone and analyzes the voice password, and when the voice password is analyzed as the storage indoor scene checking instruction, the storage indoor scene checking instruction is sent to the food management application.

In some alternative implementations, the food management application obtains one or more recently acquired groups of interior pictures from the collection of interior pictures according to the time identification of the interior pictures or the time of storage of the pictures.

In other alternative implementations, the food management application obtains one or more recently acquired sets of interior view pictures from the server based on the identity information of the refrigerator. For example, the food management application sends a picture acquisition request to the server, where the picture acquisition request includes identity information of the refrigerator. And searching one or more groups of recently acquired internal scene pictures from the associated path according to the identity information by the server, and returning the internal scene pictures to the food management application.

And S620, displaying at least one group of internal scene pictures in the user interface, wherein the display position relationship among the internal scene pictures in the same group is consistent with the position relationship among the accommodating grids corresponding to the internal scene pictures.

In some alternative implementations, at least one template view is displayed in the user interface, each template view including a plurality of picture display positions, the positional relationship between the plurality of display positions conforming to the positional relationship between the plurality of receiving compartments. For each group of internal scene pictures, determining a corresponding accommodation grid of each internal scene picture according to the position identification of each internal scene picture; and determining the display position corresponding to each internal scene picture according to the accommodation lattice corresponding to each internal scene picture. Therefore, when each internal view picture is displayed in the corresponding display position, the position relation among the internal view pictures is consistent with the position relation among the corresponding accommodating grids.

It should be noted that, the size of the picture display bit may be set according to the size of the internal view picture, for example, the size of the picture display bit may be completely equal to the size of the internal view picture, and the width or height of the picture display bit may be larger than the width or height of the internal view picture.

In some alternative implementations, the stencil view further includes spacer regions disposed between the picture display positions for simulating the inter-layer spacers of the receiving compartment. The template view may include one or more layers, and may be generated by setting parameters such as color, transparency, and the like of the layers, to include a picture display position and an interval region. The display effect of the interval region can simulate the effect of the accommodating grid layer interval plate by setting the parameters such as the color, the transparency and the like of the interval region, so that the display effect of the internal scene picture is more vivid and three-dimensional. For example, the color of the space region is set to the same color as the interlayer separator of the receiving compartment, and the height of the space region may be set to be equal to the thickness of the receiving compartment interlayer separator.

For example, the template view may include a first layer and a second layer, the first layer being located on top of the second layer. The first layer is a transparent layer and comprises a preset picture display area, namely a picture display position, and is used for setting an interior picture. And a spacing area with a preset size is arranged between the adjacent picture display areas. The transparency of the second layer was 0% and the fill color was black. When the internal view picture is displayed in the picture display area of the first picture layer, the area, corresponding to the picture display area, in the second picture layer is covered by the internal view picture, and the area, not covered by the internal view picture, in the second picture layer forms a black interval area which can simulate the interlayer partition effect of the accommodating grid.

In another exemplary embodiment, the template view includes a layer including predetermined picture presentation areas with a predetermined size of space between adjacent picture presentation areas. The transparency of this layer was 0% and the fill color was black. When the internal view picture is displayed in the picture display area, the internal view picture covers the picture display area, and the area not covered by the internal view picture forms a black interval area which can simulate the effect of accommodating the grid layer interval plate.

Fig. 7 is a view of a template according to an exemplary embodiment of the present application, which includes display positions 701, 702, and 703, and the positional relationship between the display positions 701, 702, and 703 coincides with the positional relationship between the receiving compartments 1121, 1122, and 1123 of the refrigerator shown in fig. 1A, which are all in an up-down-mid-down relationship. Thus, display bit 701 corresponds to compartment 1121, display bit 702 corresponds to compartment 1122, and display bit 703 corresponds to compartment 1123. There is a spacing area 704 between display sites 701 and 702, the spacing area 704 simulating the effect of a spacer between the compartments 1121 and 1122, and there is a spacing area 705 between display sites 703 and 704, the spacing area 705 simulating the effect of a spacer between the compartments 1122 and 1123. When the internal view pictures of the accommodation compartments 1121, 1122, and 1123 are displayed in the display positions 701, 702, and 703, respectively, the positional relationship of the internal view pictures coincides with the positional relationship between the accommodation compartments, which are all in an up-middle-down relationship.

Along the above example, when the user clicks the view-in control in the food material management interface shown in fig. 3, the view-in display interface shown in fig. 8 is displayed. As shown in fig. 8, the internal view presentation interface includes a picture presentation area 810 and a content recognition area 820. The internal view pictures 81, 812 and 813 are displayed in the picture display area 810, and the positional relationship of the internal view pictures 811, 812 and 813 in the internal view display interface coincides with the positional relationship between the accommodation boxes 1121, 1122 and 1123. In addition, under the internal view pictures, the shooting time of the group of internal view pictures is displayed as "6 months and 5 days". Thus, the user can view the food materials stored in the housing cells 1121 to 1123 at "6 months and 5 days" based on the internal view picture taken at "6 months and 5 days". For example, viewing the interior picture 811 can see that the housing grid 1121 has "grape", "banana" and "potato", viewing the interior picture 812 can see that the housing grid 1122 has "peach", "milk", viewing the interior picture 812 can see that the housing grid 1123 has "pineapple", "durian". In addition, the user can trigger the next group of the internal view pictures to be displayed in the internal view interface, and the shooting time of the next group of the internal view pictures is usually earlier than that of the previous group of the internal view pictures. For example, the user may trigger the display of the next set of intra-scene pictures by sliding the current intra-scene picture to the left. The interface shown in fig. 8 is changed to the interface shown in fig. 9 in response to a leftward sliding operation by the user. Unlike the interface shown in fig. 8, in the internal view interface shown in fig. 9, the picture display area 810 displays internal view pictures 814, 815 and 816, and the internal view pictures 814, 815 and 816 are internal view pictures of the accommodation boxes 1121, 1122 and 1123 collected at "day 4 of 6 months". A "picture identification" control 821 and a "one-key entry" control 822 are displayed below the content identification area 820, and a user can instruct to identify food material objects in the interior scene picture by clicking the "picture identification" control 821, and the food material icons of the identified food materials are displayed in the content identification area. The user may select a food icon displayed in the content identification area and enter the selected food into the food management application by clicking a "one-key enter" control 822.

Note that, the internal view display interface shown in fig. 8 and 9 is taken as an example of the present application, and does not limit the protection scope of the present application. For example, two sets of intra-scene pictures may be displayed simultaneously in an intra-scene presentation interface. Through show two sets of interior scene pictures of taking at different time simultaneously on interior scene show interface, can help the user know which food material has been newly added around and which food material has been reduced. As another example, the foreground picture display region may be defined in a left region of the interface and the content recognition region may be defined in a right region of the interface.

In some embodiments, a user may view refrigerator storage interior views through a refrigerator-associated terminal device. The terminal device associated with the refrigerator can be a terminal device associated with the refrigerator implementation based on the identity information. The identity information may be device information, account information, etc., for example, an account number logged in by the food management application.

For example, an account registered by a food management application on a refrigerator is referred to as a main service account. The terminal device associated with the refrigerator may be a terminal device bound to the main service account, such as a mobile phone, a television, a tablet computer, and the like, bound to the main service account. The terminal device can also be a terminal device corresponding to a service account related to the main service account, such as a mobile phone, a computer and the like corresponding to a WeChat account related to the main service account. Other terminal devices for logging in the main service account, such as a terminal device associated with a refrigerator, can be used for installing food management application on the television by a user and logging in the food management application on the television by using the main service account.

In some embodiments, a user may log in to a food management application on a mobile phone, a television, or other terminal device, and input a refrigerator interior view instruction in the food management application. Fig. 10 is an exemplary illustration of a food material management interface, including a view-in control, displayed on a terminal device. The user may click on the internal view control to enter an internal view instruction. The terminal equipment responds to the instruction and sends a picture acquisition request containing identity information of the terminal equipment to the server, wherein the identity information of the terminal equipment can be account information for logging in a food management application or equipment information of the terminal equipment. And the server searches one or more groups of internal scene pictures of the refrigerator associated with the terminal equipment according to the association relation between the identity information of the terminal equipment and the identity information of the refrigerator, and returns the internal scene pictures to the terminal equipment. And the terminal equipment displays one or more groups of internal scene pictures returned by the server in the user interface, and ensures that the display position relationship among the internal scene pictures in the same group is consistent with the position relationship among the accommodating grids corresponding to the internal scene pictures. Fig. 11 is an internal view display interface displayed on a terminal device, in fig. 11, internal view pictures 111, 112 and 113 are displayed, and the positional relationship of the internal view pictures 111, 112 and 113 in the internal view display interface coincides with the positional relationship between the accommodation boxes 1121, 1122 and 1123, which are all in an up-down-up-down relationship.

As can be seen from the above embodiments, the refrigerator provided by the embodiment of the present application includes a plurality of storage compartments, and the interior view pictures of the storage compartment include a set of interior view pictures of each storage compartment photographed in the same time period. The refrigerator provided by the embodiment of the application can respond to the input internal view instruction to obtain at least one group of internal view pictures; and displaying the at least one group of internal scene pictures in a user interface, wherein the display position relationship among the internal scene pictures in the same group is consistent with the position relationship among the accommodating grids corresponding to the internal scene pictures. According to the application, the interior scene picture is displayed to the user, and the arrangement mode of the interior scene picture in the user interface accords with the setting position of the accommodating grid in the storage room, so that the display effect of the interior scene in the storage room is better and more vivid, and the reduction degree is higher.

In some embodiments, when triggering the display of the storage indoor scene picture, the user can instruct the refrigerator to identify the food material object in the displayed indoor scene picture by inputting a food material identification instruction. It should be noted that, the refrigerator can also automatically identify food objects in the displayed interior scene pictures while displaying the interior scene pictures, without additional user operations.

Based on the above embodiments, the controller 220 is configured to perform the steps shown in fig. 12:

s121, responding to an input internal view instruction, displaying an internal view picture of the storage room, wherein the internal view picture comprises food material objects.

For the refrigerator shown in fig. 1A, the internal view pictures of the storage room may be displayed in the manner shown in S610-620 in the above embodiment, and the display effect may be shown in fig. 7, 8 or 9, which are not repeated here.

Taking fig. 8 as an example, the food material objects in the internal view picture 811 include "grape", "banana" and "potato".

It should be noted that, according to the refrigerator structure, the number and the position of the cameras, the number of the interior pictures in the storage room is the same or different. For example, the interior view picture may be a panoramic view picture of the interior of the storeroom, or may be composed of different local interior view pictures of the storeroom together. According to the number, the size and the like of the interior pictures, the interior pictures can be displayed in different modes.

S122, responding to the input food material identification instruction, and displaying a food material icon matched with the food material object in the internal scene picture.

In some embodiments, the user may click on a "picture identification" control in the interior scene display interface to enter a food material identification instruction that instructs the refrigerator to identify the displayed interior scene picture in the current interface.

In some internal scene display interfaces, a plurality of picture identification controls are displayed, and the picture identification controls display a plurality of internal scene pictures in a one-to-one correspondence. The user can click on a picture identification control corresponding to a certain internal scene picture to input a food material identification instruction aiming at the internal scene picture. And the food material management application responds to the instruction and displays a food material icon matched with the food material object in the internal scene picture. For example, in the internal view display interface shown in fig. 13, internal view pictures 1301, 1302 and 1303 are displayed in the internal view picture display area, wherein the internal view picture 1301 includes a food material object (1), a food material object (2) and a food material object (3), the internal view picture 1302 includes a food material object (4), a food material object (5) and a food material object (6), and the internal view picture 1303 includes a food material object (7) and a food material object (8). The upper right corner of each internal scene picture is provided with a picture identification control. When the user clicks the "picture identification" control in the upper right corner of the internal view picture 1301, a food icon matching the food object in the internal view picture 1301 will be displayed in the content identification area. For example, in the content recognition area in fig. 13, a food icon 1 matching the food object (1), a food icon 2 matching the food object (2), and a food icon 3 matching the food object (3) are displayed.

In other embodiments, the user may input the food material identification instruction for a certain internal view picture by clicking on the internal view picture. And the food material management application responds to the instruction and displays a food material icon matched with the food material object in the internal scene picture. In these embodiments, the "photo identification" control may or may not be displayed in the internal scene presentation interface. In the implementation case of displaying the "picture identification" control, the user may still input the food material identification instruction by operating the "picture identification" control.

In some internal view display interfaces, a "picture identification" control is displayed, which a user can click to input food material identification instructions for all of the displayed internal view pictures. The food material management application then displays, in response to the instruction, in the content recognition area, food material icons that match the food material objects in all of the interior scene pictures. For example, in the internal view display interface shown in fig. 14, internal view pictures 1301, 1302, and 1303 are displayed in the internal view picture display area, and a "picture recognition" control is displayed in the content recognition area. When the user clicks the "picture recognition" control, the food material icons matched with the food material objects in the internal view pictures 1301, 1302 and 1303 are displayed in the content recognition area, and the food material icons include the food material icons 1-5 and 7-8 matched with the food material objects (1) - (5) and the food material objects (7) - (8), wherein only one matched food material icon matched with the food material objects is displayed in the content recognition area because the food material objects (5) and (6) are the same.

In some implementations, the food management application identifies, by the server, the food object in the foreground picture. Referring specifically to fig. 15, a food management application on a refrigerator responds to an input food identification instruction to send a food identification request to a server, wherein the food identification request includes related information of an internal scene picture. After receiving the food material identification request, the server identifies the food material object in the internal view picture based on the related information of the internal view picture, obtains an identification result, and returns the identification result to the refrigerator, wherein the identification result comprises identification information of the food material object, such as food material ID or food material name. After receiving the identification result returned by the server, the food material management application on the refrigerator acquires the food material information matched with the identification result from a preset food material library. For example, food material information including the food material name of the food material object is acquired from a preset food material library. And finally, displaying the food material icon matched with the food material object according to the food material information matched with the identification result.

In the above embodiment, the related information of the intra-scene picture includes a unique identifier of the intra-scene picture and/or picture data of the intra-scene picture. In the embodiment of using the unique identifier of the intra-scene picture as the related information, the server needs to search the intra-scene picture characterized by the unique identifier according to the unique identifier, so as to identify the picture based on the searched intra-scene picture. The unique identification may be a picture name. In the embodiment in which the picture data of an intra-scene picture is used as the related information, the server may directly perform image recognition based on the picture data.

Referring to fig. 16, in some embodiments, a food management application on a refrigerator sends a food identification request to a server in response to an input food identification instruction, where the food identification request includes information about an interior scene picture. After receiving the food material identification request sent by the refrigerator, the server generates a corresponding picture identification task according to the food material identification request, adds the picture identification task into a designated task queue, and sequentially executes the picture identification tasks in the queue, thereby avoiding overlarge processing pressure of the server caused by concurrent execution of a large number of picture identification tasks. After the server generates the picture identification task, a result identifier is generated, and the result identifier is returned to the refrigerator, wherein the result identifier corresponds to the food material identification request or the picture identification task generated based on the food material identification request. The food management application on the refrigerator receives the result identifier and, after a second period of time (e.g., 2 seconds), sends a result acquisition request to the server that includes the result identifier. And the server searches the identification result of the corresponding picture identification task according to the result identification, and returns the identification result to the refrigerator. Therefore, the server does not need to process the picture identification request of the refrigerator in real time, so that excessive processing pressure is not generated, and the server is prevented from collapsing. In addition, by sending a picture identification request to the server and requesting result data to the server, the result request process is separated from the picture identification request process, and program blockage caused by the fact that the food management application always waits for an identification result is avoided.

After the server or the refrigerator obtains the identification result, the duplicate removal process is performed on the identification result. For example, when a plurality of repeated food material names are included in the identification result, redundant food material names are deleted, and only one food material name is reserved. The original recognition result and the recognition result after duplication removal are shown in the following table, for example:

in some embodiments, in S121, in response to the input of the interior view instruction, displaying the recently acquired interior view picture of the storage compartment; in S122, a new food material in the storage room is determined according to the recognition result of the food material object in the internal view picture returned by the server, wherein the new food material is not determined as the existing food material. And then displaying the food material icons of the new food materials in the user interface, but not displaying the food material icons of the non-new food materials, so that the effect of filtering the non-new food materials is realized, the situation that the user selects the food material icons of the non-new food materials to add is avoided, and the accuracy and pertinence of the user operation are improved. In an alternative implementation manner, after receiving the food material name of the food material object returned by the server, acquiring a food material name set of the existing food material (from the local food material information set); if the food material name of the food material object is included in the food material name set of the existing food material, it is determined that the food material object is not a new food material, and if the food material name of the food material object is not included in the food material name set of the existing food material, it is determined that the food material object is a new food material.

In some embodiments, through S122, a food icon that matches a food object in the interior scene picture is displayed as a state to be deleted. Thus, by clicking a certain food icon to delete the food icon, the last remaining food icon is the selected target food icon.

S123, according to the target food material icon selected by the user, determining the food material information corresponding to the target food material icon as the food material information of the existing food material in the storage room.

And determining the food material information corresponding to the target food material icon as the food material information of the existing food material, and finishing the food material adding operation, wherein the added food material is the food material represented by the target food material icon.

In some embodiments, in response to a food material entry instruction input by a user, food material information corresponding to a target food material icon selected by the user is determined as food material information of existing food materials in the storage room.

It should be understood that the user may select a target food icon from among the food icons displayed in the content recognition area through various operation modes. For example, a certain food icon is selected by long pressing the food icon. For another example, a certain food material icon is deleted by long pressing the certain food material icon, and the last remaining food material icon is the selected target food material icon. For another example, all food icons are triggered to enter a to-be-selected state by long-pressing a certain food icon, and then the food icon is selected by clicking the certain food icon. For another example, all the food material icons are triggered to enter a state to be deleted by long-pressing a certain food material icon, then the food material icon is deleted by clicking the certain food material icon, and the last reserved food material icon is the selected target food material icon.

In some embodiments, the content recognition area is updated in real time based on the user's operation on the food icons in the content recognition area, so that the user can confirm the selection of the food icons according to the display condition of the content recognition area. After finishing the selection operation of the target food material icon, the user can input a food material input command by operating the one-key input control in the content identification area, and the food material management application responds to the command to determine the food material information corresponding to the target food material icon as the food material information of the existing food material in the storage room, so that the operation of adding (inputting) new food materials into the food material management application is finished.

In an alternative implementation manner, the food material information corresponding to the target food material icon is determined as the food material information of the existing food material in the storage room, specifically, the food material information corresponding to the target food material icon is added to the local food material information set, and is marked as the put-in state.

In some embodiments, after determining the food material information corresponding to the target food material icon selected by the user as the food material information of the existing food material, the food material information of all the existing food materials at present is read, and the interface is updated to be the food material management interface. And displaying the food material icons of all the current existing food materials in the updated food material management interface.

As can be seen from the above embodiments, the refrigerator provided by the embodiments of the present application can display an internal view picture of a storage room in response to an input internal view viewing instruction, where the internal view picture includes food objects; responding to an input food material identification instruction, and displaying a food material icon matched with a food material object in the internal scene picture; and according to the target food material icon selected by the user, determining the food material information corresponding to the target food material icon as the food material information of the existing food material in the storage room. According to the application, the food material information of the food material in the storage room can be obtained based on image recognition and displayed in the form of the food material icon, the food material adding operation is completed by determining the food material information corresponding to the target food material icon selected by the user as the food material information of the existing food material, and the added food material is the food material represented by the target food material icon, so that the user operation of manually inputting the food material information is omitted.

According to the refrigerator provided by the embodiment, the embodiment of the application also provides an image acquisition method, and an execution subject of the image acquisition method comprises but is not limited to the controller of the refrigerator. The specific implementation of the image acquisition method may refer to the above embodiment, particularly the embodiment shown in fig. 4, which is not described herein.

According to the refrigerator provided by the embodiment, the embodiment of the application also provides a storage indoor scene picture display method, and an execution main body of the storage indoor scene picture display method comprises but is not limited to the controller of the refrigerator. The specific implementation of the method for displaying the indoor scenery pictures can refer to the above embodiment, especially the embodiment shown in fig. 6, which is not described herein again.

According to the refrigerator provided by the embodiment, the embodiment of the application also provides a food material input method, and an execution main body of the food material input method comprises but is not limited to the controller of the refrigerator. The specific implementation of the food material inputting method may refer to the above embodiment, particularly the embodiment shown in fig. 12, which is not described herein.

In a specific implementation, the present application further provides a computer storage medium, where the computer storage medium may store a program, where the program may include some or all of the steps in each embodiment of the method provided by the present application when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (random access memory, RAM), or the like.

It will be apparent to those skilled in the art that the techniques of embodiments of the present invention may be implemented in software plus a necessary general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in essence or what contributes to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments or some parts of the embodiments of the present invention.

The same or similar parts between the various embodiments in this specification are referred to each other. In particular, for refrigerator embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and reference should be made to the description of the method embodiments.

The embodiments of the present invention described above do not limit the scope of the present invention.

Claims

1. A refrigerator, comprising:

the box body is internally provided with a storage chamber, the storage chamber comprises a plurality of containing grids for containing food materials, the containing grids are arranged in the vertical direction, and the containing grids are mutually partitioned through interlayer partitions;

The box door is arranged at the opening of the storage chamber;

the camera is arranged on one side of the box door, faces one side of the interior of the storage chamber when the box door is closed, and is used for collecting an internal view picture of the storage chamber;

a controller coupled to the camera and the display configured to:

generating a template view comprising a plurality of picture display positions and interval areas through preset colors and transparencies of one or more layers; the interval area is used for simulating an interlayer clapboard of the accommodating grid; the picture display position is used for setting the internal scene picture;

displaying at least one template view in the user interface, and displaying each internal view picture in a picture display position corresponding to each internal view picture in the template view according to a containing grid corresponding to each internal view picture in each group of internal view pictures so as to enable the internal view pictures to be consistent with the actual display effect;

responding to the operation of the user on the internal scene pictures, and displaying a second group of internal scene pictures on an internal scene display interface, wherein the shooting time of the second group of internal scene pictures is earlier than that of the first group of internal scene pictures displayed in the previous group;

And responding to the operation of the user, and simultaneously displaying two groups of internal view pictures shot at different times on the internal view display interface so as to enable the user to know the change condition of food materials in the refrigerator in a certain time period.

2. The refrigerator of claim 1, wherein displaying each of the plurality of interior pictures in the corresponding display position according to the corresponding compartment of each of the plurality of interior pictures comprises:

determining a containing grid corresponding to the internal scene picture according to the position identification of the internal scene picture;

and determining the display position corresponding to the internal scene picture according to the accommodation lattice corresponding to the internal scene picture.

3. The refrigerator of claim 1, wherein the controller is further configured to:

when the preset triggering condition is met, controlling the camera to acquire the internal view pictures corresponding to the accommodating grids;

and allocating position identifiers for the internal scene pictures, wherein the position identifiers of the internal scene pictures of different accommodation boxes are different.

4. The refrigerator of claim 3, further comprising:

and distributing time identifiers for the internal scene pictures, wherein the time identifiers of the internal scene pictures of different accommodating grids collected at the same time are different, and the time identifiers are used for representing the collection time of the internal scene pictures.

5. The refrigerator of claim 4, further comprising:

cutting the internal view picture according to a preset cutting rule to reserve a key region of the internal view picture, and removing a redundant region of the internal view picture.

6. The refrigerator of claim 4, further comprising:

generating a picture name containing the position identifier and the time identifier;

and storing the internal scene picture according to the picture name.

7. The refrigerator of claim 1, wherein the controller is further configured to:

displaying a food material management interface, wherein the food material management interface comprises a storage indoor scene viewing control;

and receiving an input storage indoor scene viewing instruction by receiving clicking operation of a user on the storage indoor scene viewing control.

8. An electronic device comprising a memory for storing program instructions and a processor for executing the program instructions to perform the steps of:

displaying at least one template view in a user interface, and displaying each internal view picture in a picture display position corresponding to each internal view picture in the template view according to a containing grid corresponding to each internal view picture in each group of internal view pictures so as to enable the internal view pictures to be consistent with the actual display effect;

9. A method for displaying a scene picture in a storage room, which is applied to a refrigerator, the method comprising: