CN114821236A - Smart home environment sensing method, system, storage medium and electronic device - Google Patents
Smart home environment sensing method, system, storage medium and electronic device Download PDFInfo
- Publication number
- CN114821236A CN114821236A CN202210470795.7A CN202210470795A CN114821236A CN 114821236 A CN114821236 A CN 114821236A CN 202210470795 A CN202210470795 A CN 202210470795A CN 114821236 A CN114821236 A CN 114821236A
- Authority
- CN
- China
- Prior art keywords
- scene
- indoor
- image
- perception
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Abstract
The application discloses a smart home environment sensing method, a smart home environment sensing system, a storage medium and an electronic device, and relates to the technical field of smart homes and smart homes, wherein the smart home environment sensing method comprises the following steps: acquiring an indoor scene image of a target area at the current moment; inputting the indoor scene image into an intelligent home environment perception model to obtain scene perception text information of the target area at the current moment; the intelligent home environment perception model is obtained by training a neural network through indoor scene sample images and sample scene text sentences, and the indoor scene sample images are marked with scene feature labels. According to the method and the device, the technical problem that scene information needs to be predefined manually in the prior art is solved, and the real sensing result of the real scene at the current moment is obtained by sensing the environment of the indoor scene image acquired in real time, so that more accurate scene information is provided for the relevant decision of a subsequent smart home.
Description
Technical Field
The application relates to the technical field of artificial intelligence, in particular to a smart home environment sensing method, a smart home environment sensing system, a storage medium and an electronic device.
Background
At present, during a planning scenario, most of the scenarios are predefined manually, for example, several specific scenario description schemes are preset based on the experience of an enterprise product manager or a planning manager, or imagine a family life scenario of a user.
However, most of the existing scenes are manually defined in advance, and the scene descriptions in the fixed mode are set and are all in a static scene state, and real-time scene perception is not realized from the actual situation of family life. The precondition of subsequent downstream decision in the smart home needs to be realized by sensing the scene environment, but the accuracy of the currently obtained scene sensing result is low, and the scene sensing result is manually defined in advance, so that the scene condition of real family life cannot be captured.
Therefore, there is a need for a smart home environment sensing method, system, storage medium and electronic device to solve the above problems.
Disclosure of Invention
The application provides a smart home environment perception method, a smart home environment perception system, a storage medium and an electronic device, which are used for overcoming the defect that in the prior art, only scene perception results are preset manually, realizing real-time perception of environment scenes in an artificial intelligence mode and obtaining more accurate scene perception results.
The application provides a smart home environment perception method, which comprises the following steps:
acquiring an indoor scene image of a target area at the current moment;
inputting the indoor scene image into an intelligent home environment perception model to obtain scene perception text information of the target area at the current moment;
the intelligent home environment perception model is obtained by training a neural network through indoor scene sample images and sample scene text sentences, and the indoor scene sample images are marked with scene feature labels.
According to the intelligent home environment perception method provided by the application, the indoor scene image is input into the intelligent home environment perception model, and the scene perception text information of the target area at the current moment is obtained, and the method comprises the following steps:
inputting the indoor scene image into the intelligent household environment perception model to obtain a scene recognition result output by a scene recognition model in the intelligent household environment perception model;
determining the scene type of the current moment in the target area according to the scene recognition result, and acquiring a scene perception model corresponding to the scene type from the intelligent home environment perception model;
processing the scene recognition result through the scene perception model to obtain scene perception text information of the target area at the current moment;
the scene recognition model is obtained by training a first neural network through an indoor scene sample image marked with a scene characteristic label;
the scene perception model is obtained by training a second neural network through sample scene text sentences corresponding to indoor scene sample images of different scene types.
According to the intelligent family environment perception method provided by the application, the scene recognition model is obtained by training through the following steps:
acquiring a plurality of indoor scene sample images;
marking a corresponding scene characteristic label in each indoor scene sample image to construct a first training sample set, wherein the scene characteristic label comprises a user attribute label, a user state characteristic label, a furniture attribute label, a household appliance attribute label and a household appliance state characteristic label;
and training the first neural network through the first training sample set to obtain a scene recognition model.
According to the intelligent family environment perception method provided by the application, the scene perception model is obtained by training through the following steps:
acquiring scene image descriptors corresponding to each indoor scene sample image according to indoor scene sample images of different scene types, wherein the scene types comprise a living room scene, a bedroom scene and a balcony scene;
generating a corresponding sample scene text sentence according to the scene image descriptor and a preset text rule;
constructing a second training sample set according to the scene image descriptors and the corresponding sample scene text sentences;
and training a second neural network through the second training sample set to obtain a scene perception model.
According to the intelligent household environment perception method provided by the application, after the obtaining of the plurality of indoor scene sample images, the method further comprises the following steps:
based on an image augmentation method, preprocessing a plurality of indoor scene sample images to obtain preprocessed indoor scene sample images;
marking a corresponding scene characteristic label in each indoor scene sample image to construct and obtain a first training sample set, including:
and marking a corresponding scene characteristic label for each preprocessed indoor scene sample image to obtain a first training sample set.
According to the intelligent home environment perception method provided by the application, after the indoor scene image is input into the intelligent home environment perception model, the scene perception text information of the target area at the current moment is obtained, and the method further comprises the following steps:
according to the scene perception text information, entity extraction and relation extraction are carried out to obtain entity information and relation information;
and constructing a knowledge graph corresponding to the target area through the entity information and the relation information.
According to the intelligent household environment perception method provided by the application, before the indoor scene image of the target area at the current moment is acquired, the method further comprises the following steps:
acquiring an environment perception instruction sent by a user terminal, wherein the environment perception instruction comprises target area information and a preset image acquisition time period;
and if the current moment is within the preset image acquisition time period, acquiring the indoor scene image according to the target area information.
The application also provides a wisdom family environmental perception system, includes:
the image acquisition module is used for acquiring an indoor scene image of a target area at the current moment;
the environment perception module is used for inputting the indoor scene image into an intelligent household environment perception model to obtain scene perception text information of the target area at the current moment;
the intelligent home environment perception model is obtained by training a neural network through indoor scene sample images and sample scene text sentences, and the indoor scene sample images are marked with scene feature labels.
The present application also provides a computer-readable storage medium comprising a stored program, wherein the program is executed to implement the smart home environment sensing method according to any one of the above.
The present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the smart home environment sensing method according to any one of the above methods by executing the computer program.
The intelligent home environment sensing method, the intelligent home environment sensing system, the intelligent home environment sensing storage medium and the electronic device solve the technical problem that scene information needs to be predefined manually in the prior art, and the actual sensing result of the current real scene is obtained by sensing the environment of the indoor scene image acquired in real time, so that more accurate scene information is provided for the relevant decision of a follow-up intelligent home.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
Fig. 1 is a schematic diagram of a hardware environment of an interaction method of a smart device according to an embodiment of the present application;
FIG. 2 is a schematic flowchart of a smart home environment sensing method provided in the present application;
FIG. 3 is a flowchart illustrating an overall method for smart home environment sensing provided by the present application;
FIG. 4 is a schematic diagram of a smart home environment sensing system provided in the present application;
fig. 5 is a schematic structural diagram of an electronic device provided in the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in this application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
According to one aspect of the embodiment of the application, an interaction method of intelligent household equipment is provided. The interaction method of the intelligent Home equipment is widely applied to full-House intelligent digital control application scenes such as intelligent homes (Smart Home), intelligent homes, intelligent Home equipment ecology, intelligent House (Intelligent House) ecology and the like. Optionally, fig. 1 is a schematic diagram of a hardware environment of an interaction method of an intelligent device according to an embodiment of the present application, and in this embodiment, the interaction method of an intelligent home device may be applied to the hardware environment formed by the terminal device 102 and the server 104 shown in fig. 1. As shown in fig. 1, the server 104 is connected to the terminal device 102 through a network, and may be configured to provide a service (e.g., an application service) for the terminal or a client installed on the terminal, set a database on the server or independent of the server, and provide a data storage service for the server 104, and configure a cloud computing and/or edge computing service on the server or independent of the server, and provide a data operation service for the server 104.
The network may include, but is not limited to, at least one of: wired networks, wireless networks. The wired network may include, but is not limited to, at least one of: wide area networks, metropolitan area networks, local area networks, which may include, but are not limited to, at least one of the following: WIFI (Wireless Fidelity), bluetooth. Terminal equipment 102 can be but not limited to be PC, the cell-phone, the panel computer, intelligent air conditioner, intelligent cigarette machine, intelligent refrigerator, intelligent oven, intelligent kitchen range, intelligent washing machine, intelligent water heater, intelligent washing equipment, intelligent dish washer, intelligent projection equipment, intelligent TV, intelligent clothes hanger, intelligent (window) curtain, intelligence audio-visual, smart jack, intelligent stereo set, intelligent audio amplifier, intelligent new trend equipment, intelligent kitchen guarding equipment, intelligent bathroom equipment, intelligence robot of sweeping the floor, intelligence robot of wiping the window, intelligence robot of mopping the ground, intelligent air purification equipment, intelligent steam ager, intelligent microwave oven, intelligent kitchen is precious, intelligent clarifier, intelligent water dispenser, intelligent lock etc..
In the existing planning and planning scene process, an enterprise product manager presets and defines several or more than ten scene descriptions in the smart home system based on self experience of a family life scene possibly appearing in the smart home scene, so that the intelligent home system matches the several or more than ten scene descriptions in combination with a judgment result when judging a home environment, and an intelligent control scheme is decided for subsequent home equipment. However, since the scene descriptions are manually set in the early stage, the types and the number of the scene descriptions are limited, and in the prior art, the states of the home appliances are mainly monitored, that is, simple state information such as turning on or turning off of the home appliances is mainly obtained, and the intelligent home environment cannot be completely and accurately described, for example, when a television in a living room in the home environment is in an on state, the current scene description may define the environment sensing result at this time as that the user is watching a television program, however, the actual situation is that the user is talking with a mobile phone or watching content played by the mobile phone, and the scene sensing result is inaccurate.
The method and the device can actually capture the real scene in the smart home, sense the user scene condition in the smart home environment, generate scene description based on the image through the captured scene, perform entity recognition and relationship recognition on the scene description, generate the scene event knowledge graph, and provide basis for the follow-up intelligent control scheme.
Fig. 2 is a schematic flow chart of the smart home environment sensing method provided by the present application, and as shown in fig. 2, the present application provides a smart home environment sensing method, including:
in the application, the scene description is performed through the image shot by the camera arranged in the room, and besides, the real-time environment in the room can be shot through the mobile terminal products of the user, such as the mobile phone, the iPad, the notebook computer and other terminals with the shooting function, so that the scene image in the room area at the current moment is collected.
Preferably, before the acquiring the indoor scene image of the target area at the current moment, the method further includes:
acquiring an environment perception instruction sent by a user terminal, wherein the environment perception instruction comprises target area information and a preset image acquisition time period;
and if the current moment is within the preset image acquisition time period, acquiring the indoor scene image according to the target area information.
In the application, a user sets corresponding target area information and a preset image acquisition time period through a mobile terminal, so as to determine a room area of an image to be acquired at the next moment, for example, the preset image acquisition time period is 9 to 10 am, when the time period is met, a camera can determine a target room (area) according to the target area information, and then image acquisition is performed on the environment of the target room in the time period, and the image information is sent to a server in real time for scene perception processing.
the intelligent home environment perception model is obtained by training a neural network through indoor scene sample images and sample scene text sentences, and the indoor scene sample images are marked with scene feature labels.
In the application, a camera collects images in a room in real time, the collected images of indoor scenes are sent to a server, based on an intelligent home environment perception model trained through a neural network in the server in the earlier stage, the images of the indoor scenes collected at the current moment are processed, so that the characteristics in the images, such as the placement position characteristics of furniture in the scenes, the state characteristics of household appliances, the behavior characteristics of users and the like, are automatically identified, then according to the identified characteristics, the model generates corresponding scene perception text information which is used as scene description of the current moment of a target area, for example, a television with the view angle of the user being opened facing the model identification is determined, the output result of the model is the text information of the television watched by the user, or two users face away from the television with the view angle interactive between the users, and the mouth moves, the model may output scene-aware text information that the user is chatting.
The intelligent home environment perception method solves the technical problem that scene information needs to be predefined manually in the prior art, and obtains the actual perception result of the real scene at the current moment by carrying out environment perception on the indoor scene image collected in real time, so that more accurate scene information is provided for the relevant decision of a follow-up intelligent home.
On the basis of the above embodiment, the inputting the indoor scene image into the smart home environment perception model to obtain the scene perception text information of the target area at the current time includes:
inputting the indoor scene image into the intelligent household environment perception model to obtain a scene recognition result output by a scene recognition model in the intelligent household environment perception model;
determining the scene type of the current moment in the target area according to the scene recognition result, and acquiring a scene perception model corresponding to the scene type from the intelligent home environment perception model;
processing the scene recognition result through the scene perception model to obtain scene perception text information of the target area at the current moment;
the scene recognition model is obtained by training a first neural network through an indoor scene sample image marked with a scene characteristic label;
the scene perception model is obtained by training a second neural network through sample scene text sentences corresponding to indoor scene sample images of different scene types.
In the application, the smart home environment perception model is composed of a scene recognition model and a plurality of scene perception models. After the camera in the target area sends the image acquired in real time to the server, the image is firstly identified by the scene identification model, for example, for watching a television scene, the identification result may include that a user exists, the television is turned on, the visual direction of the user faces towards the television, and the like; the recognition result of the chat scene can comprise that two or more users exist, visual angle interaction exists among the users, the users do not face the scene back to back, and the users sit on a sofa or other furniture; the recognition result of the sleep scene may include the presence of the user, the user's eyes being closed, the user being in a lying posture, and the like; the recognition result of the getting-up scene may include the presence of the user, the process from closing the eyes to opening the eyes, the process from lying to sitting, and the like; sunning scenes may include the presence of a user, the process of hanging clothes on a laundry rack, and the like.
Further, after the scene recognition model completes recognition in the target area, determining the scene type of the target area according to the room characteristics in the area contained in the recognition result, for example, judging that the target area is the living room scene type based on the characteristics of furniture type, arrangement position and the like; or identifying that the target area contains furniture such as a bed, a bedside cabinet, a wardrobe and the like, and judging that the target area is of a bedroom scene type. And then, determining a corresponding scene perception model according to the scene type, and if the scene type is the living room scene type, training the scene perception model mainly through sample data related to relevant characteristics of the living room in the training process. Then, the scene perception model generates corresponding scene perception text information according to the scene recognition result, for example, for a living room scene, the related scene perception model outputs the scene perception text information of 'the user is watching television' according to the characteristics of the user, the television is turned on, the visual direction of the user faces to the television and the like in the recognition result; or in a bedroom scene, based on a scene perception model of a bedroom scene type, according to the characteristics of the existence of the user, the tight eye closure of the user, the lying posture of the user and the like in the recognition result, outputting scene perception text information of 'the user sleeps in the bedroom'.
On the basis of the above embodiment, after the indoor scene image is input into the smart home environment perception model, and the scene perception text information of the target area at the current time is obtained, the method further includes:
according to the scene perception text information, entity extraction and relation extraction are carried out to obtain entity information and relation information;
and constructing a knowledge graph corresponding to the target area through the entity information and the relation information.
In the application, the entity extraction is to identify information elements in a text, and includes labels such as name, organization/organization name, geographic location, time/date, character value and the like, and the specific label definition can be adjusted according to different tasks. In the embodiment of the present application, entity extraction may be performed based on algorithms such as an existing Convolutional Neural Network (CNN), a Long Short-Term Memory Network (LSTM), and a Conditional Random Field (CRF). Relationship extraction is to extract semantic relationships between two or more entities from text, and can be performed by existing algorithms, for example, based on the LSTM + Attention mechanism.
Further, after the extraction of the entities and the relations is completed, based on the knowledge graph construction requirements of the corresponding fields, all the entities are connected through the relations, so that the knowledge graph of the field is constructed, and the constructed knowledge graph is stored in a graph database to provide data support for the scheme decision process of the subsequent smart home.
On the basis of the above embodiment, the scene recognition model is obtained by training through the following steps:
acquiring a plurality of indoor scene sample images;
marking a corresponding scene characteristic label in each indoor scene sample image to construct a first training sample set, wherein the scene characteristic label comprises a user attribute label, a user state characteristic label, a furniture attribute label, a household appliance attribute label and a household appliance state characteristic label;
and training the first neural network through the first training sample set to obtain a scene recognition model.
In the present application, the first neural network may be AlexNet, YOLOv5, etc., and the indoor scene sample image is labeled, for example, the gender of the user, the state of the user (eye tight state, visual angle towards the television, etc.), the furniture type, the household appliance type, and the household appliance on or off state are labeled, so as to form a corresponding training sample set, the first neural network is trained, and when the training condition satisfies the preset condition (reaches the training times), the training is stopped to obtain a trained model, that is, the scene recognition model.
On the basis of the above embodiment, the scene perception model is obtained by training through the following steps:
acquiring scene image descriptors corresponding to each indoor scene sample image according to indoor scene sample images of different scene types, wherein the scene types comprise a living room scene, a bedroom scene and a balcony scene;
generating a corresponding sample scene text sentence according to the scene image descriptor and a preset text rule;
constructing a second training sample set according to the scene image descriptors and the corresponding sample scene text sentences;
and training a second neural network through the second training sample set to obtain a scene perception model.
In the application, the second neural network can be a Word2vec model, in the training process, a group of training samples are formed based on scene image descriptors under different scenes and corresponding sample scene text sentences, then a second training sample set is constructed to train the Word2vec model, and after the preset training times are met, a scene perception model corresponding to the scene is obtained. In the present application, the scene image descriptor is understood as a word form corresponding to the scene recognition result, for example, if the recognition result indicates that there is a user, a living room, and eyes are facing to a television, the scene image descriptor is referred to as "living room, user, eyes, television, and facing".
On the basis of the foregoing embodiment, after the acquiring a plurality of indoor scene sample images, the method further includes:
based on an image augmentation method, preprocessing a plurality of indoor scene sample images to obtain preprocessed indoor scene sample images;
marking a corresponding scene characteristic label in each indoor scene sample image to construct and obtain a first training sample set, including:
and marking a corresponding scene characteristic label for each preprocessed indoor scene sample image to obtain a first training sample set.
In the application, based on an image augmentation (image augmentation) method, a series of random changes are performed on training images to generate similar but different training samples, so that the scale of a training data set is enlarged, the dependence of a model on certain attributes can be reduced, and the generalization capability of the model is improved. For example, the image may be cropped in different ways so that the object of interest appears at different locations, thereby allowing the model to mitigate dependence on where the object appears; factors such as brightness, color, etc. may also be adjusted to reduce the sensitivity of the model to color.
Fig. 3 is a flowchart illustrating an overall smart home environment sensing method according to the present application, referring to fig. 3, in an embodiment, the overall steps of the smart home environment sensing method are as follows:
step S1, sending the images collected in real time to a server based on the photographing and video recording functions of the camera, and predicting through a scene recognition model in the server to obtain a space recognition result, namely a scene recognition result in a target area;
step S2, according to the scene recognition result, applying an environment perception model corresponding to the corresponding scene to perform spatial scene perception on the scene recognition result to obtain an environment perception result;
step S3, returning the environmental sensing result to the relevant device, for example, to the camera, and determining whether to continue image acquisition or not, or storing the environmental sensing result in the database of the server;
step S4, extracting the entity and the relation of the environmental perception result to obtain the entity information and the relation information of the target scene at the current moment;
and step S5, constructing a spatial event knowledge graph corresponding to the target scene at the current moment according to the entity information and the relationship information, and sending the generated knowledge graph to related equipment.
The smart home environment sensing system provided by the present application is described below, and the smart home environment sensing system described below and the smart home environment sensing method described above may be referred to in correspondence with each other.
Fig. 4 is a schematic structural diagram of the smart home environment sensing system provided in the present application, and as shown in fig. 4, the present application provides a smart home environment sensing system, which includes an image acquisition module 401 and an environment sensing module 402, where the image acquisition module 401 is configured to acquire an indoor scene image of a target area at a current time; the environment perception module 402 is configured to input the indoor scene image into an intelligent home environment perception model to obtain scene perception text information of the target area at the current moment; the intelligent home environment perception model is obtained by training a neural network through indoor scene sample images and sample scene text sentences, and the indoor scene sample images are marked with scene feature labels.
The intelligent home environment sensing system solves the technical problem that scene information needs to be defined in advance in a manual mode in the prior art, and environment sensing is carried out on indoor scene images acquired in real time, so that the actual sensing result of the real scene at the current moment is obtained, and more accurate scene information is provided for follow-up relevant decisions of an intelligent home.
On the basis of the above embodiment, the environment sensing module includes a scene recognition unit, a sensing model determination unit and an environment sensing unit, where the scene recognition unit is configured to input the indoor scene image to the smart home environment sensing model to obtain a scene recognition result output by a scene recognition model in the smart home environment sensing model; the perception model determining unit is used for determining the scene type of the current moment in the target area according to the scene recognition result and acquiring a scene perception model corresponding to the scene type from the intelligent home environment perception model; the environment perception unit is used for processing the scene recognition result to obtain scene perception text information of the target area at the current moment; the scene recognition model is obtained by training a first neural network through an indoor scene sample image marked with a scene characteristic label; the scene perception model is obtained by training a second neural network through sample scene text sentences corresponding to indoor scene sample images of different scene types.
On the basis of the embodiment, the system further comprises an extraction module and a knowledge graph construction module, wherein the extraction module is used for performing entity extraction and relationship extraction according to the scene perception text information to obtain entity information and relationship information; and the knowledge graph building module is used for building a knowledge graph corresponding to the target area through the entity information and the relation information.
On the basis of the above embodiment, the system further includes a first training module, specifically configured to:
acquiring a plurality of indoor scene sample images;
marking a corresponding scene characteristic label in each indoor scene sample image to construct a first training sample set, wherein the scene characteristic label comprises a user attribute label, a user state characteristic label, a furniture attribute label, a household appliance attribute label and a household appliance state characteristic label;
and training the first neural network through the first training sample set to obtain a scene recognition model.
On the basis of the above embodiment, the system further includes a second training module, specifically configured to:
acquiring scene image descriptors corresponding to each indoor scene sample image according to indoor scene sample images of different scene types, wherein the scene types comprise a living room scene, a bedroom scene and a balcony scene;
generating a corresponding sample scene text sentence according to the scene image descriptor and a preset text rule;
constructing a second training sample set according to the scene image descriptors and the corresponding sample scene text sentences;
and training a second neural network through the second training sample set to obtain a scene perception model.
On the basis of the above embodiment, the system further includes:
and the image preprocessing module is used for preprocessing the plurality of indoor scene sample images based on an image augmentation method to obtain preprocessed indoor scene sample images.
On the basis of the embodiment, the system further comprises a perception instruction receiving module and an instruction executing module, wherein the perception instruction receiving module is used for acquiring an environment perception instruction sent by the user terminal, and the environment perception instruction comprises target area information and a preset image acquisition time period; and the instruction execution module is used for acquiring the indoor scene image according to the target area information if the current moment is within the preset image acquisition time period.
Fig. 5 is a schematic structural diagram of an electronic device provided in the present application, and as shown in fig. 5, the electronic device may include: a processor (processor)510, a communication Interface (Communications Interface)520, a memory (memory)530 and a communication bus 540, wherein the processor 510, the communication Interface 520 and the memory 530 communicate with each other via the communication bus 540. Processor 510 may invoke logic instructions in memory 530 to perform a smart home context awareness method comprising: acquiring an indoor scene image of a target area at the current moment; inputting the indoor scene image into an intelligent home environment perception model to obtain scene perception text information of the target area at the current moment; the intelligent home environment perception model is obtained by training a neural network through indoor scene sample images and sample scene text sentences, and the indoor scene sample images are marked with scene feature labels.
Furthermore, the logic instructions in the memory 530 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present application further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a computer-readable storage medium, when the computer program is executed by a processor, a computer can execute the intelligent home environment sensing method provided by the above methods, the method includes: acquiring an indoor scene image of a target area at the current moment; inputting the indoor scene image into an intelligent home environment perception model to obtain scene perception text information of the target area at the current moment; the intelligent home environment perception model is obtained by training a neural network through indoor scene sample images and sample scene text sentences, and the indoor scene sample images are marked with scene feature labels.
In another aspect, the present application further provides a computer-readable storage medium, which includes a stored program, where the program is executed to perform the smart home environment sensing method provided by the above methods, and the method includes: acquiring an indoor scene image of a target area at the current moment; inputting the indoor scene image into an intelligent home environment perception model to obtain scene perception text information of the target area at the current moment; the intelligent home environment perception model is obtained by training a neural network through indoor scene sample images and sample scene text sentences, and the indoor scene sample images are marked with scene feature labels.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (10)
1. A smart home environment sensing method, comprising:
acquiring an indoor scene image of a target area at the current moment;
inputting the indoor scene image into an intelligent home environment perception model to obtain scene perception text information of the target area at the current moment;
the intelligent home environment perception model is obtained by training a neural network through indoor scene sample images and sample scene text sentences, and the indoor scene sample images are marked with scene feature labels.
2. The method as claimed in claim 1, wherein the inputting the indoor scene image into a smart home environment perception model to obtain scene perception text information of the target area at a current time includes:
inputting the indoor scene image into the intelligent household environment perception model to obtain a scene recognition result output by a scene recognition model in the intelligent household environment perception model;
determining the scene type of the current moment in the target area according to the scene recognition result, and acquiring a scene perception model corresponding to the scene type from the intelligent home environment perception model;
processing the scene recognition result through the scene perception model to obtain scene perception text information of the target area at the current moment;
the scene recognition model is obtained by training a first neural network through an indoor scene sample image marked with a scene characteristic label;
the scene perception model is obtained by training a second neural network through sample scene text sentences corresponding to indoor scene sample images of different scene types.
3. The intelligent home environment sensing method of claim 2, wherein the scene recognition model is trained by the following steps:
acquiring a plurality of indoor scene sample images;
marking a corresponding scene characteristic label in each indoor scene sample image to construct a first training sample set, wherein the scene characteristic label comprises a user attribute label, a user state characteristic label, a furniture attribute label, a household appliance attribute label and a household appliance state characteristic label;
and training the first neural network through the first training sample set to obtain a scene recognition model.
4. The intelligent home environment sensing method of claim 2, wherein the scene perception model is trained by the following steps:
acquiring scene image descriptors corresponding to each indoor scene sample image according to indoor scene sample images of different scene types, wherein the scene types comprise a living room scene, a bedroom scene and a balcony scene;
generating a corresponding sample scene text sentence according to the scene image descriptor and a preset text rule;
constructing a second training sample set according to the scene image descriptors and the corresponding sample scene text sentences;
and training a second neural network through the second training sample set to obtain a scene perception model.
5. The smart home environmental perception method of claim 3, wherein after the obtaining a plurality of indoor scene sample images, the method further comprises:
based on an image augmentation method, preprocessing a plurality of indoor scene sample images to obtain preprocessed indoor scene sample images;
marking a corresponding scene characteristic label in each indoor scene sample image to construct and obtain a first training sample set, including:
and marking a corresponding scene characteristic label for each preprocessed indoor scene sample image to obtain a first training sample set.
6. The smart home environment sensing method of claim 1, wherein after inputting the indoor scene image into the smart home environment sensing model and obtaining the scene-sensing text information of the target area at the current time, the method further comprises:
according to the scene perception text information, entity extraction and relation extraction are carried out to obtain entity information and relation information;
and constructing a knowledge graph corresponding to the target area through the entity information and the relation information.
7. The intelligent home environment sensing method according to any one of claims 1 to 6, wherein before the acquiring the indoor scene image of the target area at the current time, the method further comprises:
acquiring an environment perception instruction sent by a user terminal, wherein the environment perception instruction comprises target area information and a preset image acquisition time period;
and if the current moment is within the preset image acquisition time period, acquiring the indoor scene image according to the target area information.
8. A smart home environmental awareness system, comprising:
the image acquisition module is used for acquiring an indoor scene image of a target area at the current moment;
the environment perception module is used for inputting the indoor scene image into an intelligent household environment perception model to obtain scene perception text information of the target area at the current moment;
the intelligent home environment perception model is obtained by training a neural network through indoor scene sample images and sample scene text sentences, and the indoor scene sample images are marked with scene feature labels.
9. A computer-readable storage medium, comprising a stored program, wherein the program when executed performs the method of any of claims 1 to 7.
10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 7 by means of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210470795.7A CN114821236A (en) | 2022-04-28 | 2022-04-28 | Smart home environment sensing method, system, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210470795.7A CN114821236A (en) | 2022-04-28 | 2022-04-28 | Smart home environment sensing method, system, storage medium and electronic device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114821236A true CN114821236A (en) | 2022-07-29 |
Family
ID=82509576
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210470795.7A Pending CN114821236A (en) | 2022-04-28 | 2022-04-28 | Smart home environment sensing method, system, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114821236A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115733705A (en) * | 2022-11-08 | 2023-03-03 | 深圳绿米联创科技有限公司 | Space-based information processing method and device, electronic equipment and storage medium |
| CN116226655A (en) * | 2022-12-05 | 2023-06-06 | 广州视声智能股份有限公司 | Smart home environment sensing method and device, storage medium and electronic equipment |
| CN117706954A (en) * | 2024-02-06 | 2024-03-15 | 青岛海尔科技有限公司 | Method and device for generating scene, storage medium and electronic device |
| CN117708680A (en) * | 2024-02-06 | 2024-03-15 | 青岛海尔科技有限公司 | Method and device for improving accuracy of classification model, storage medium and electronic device |
-
2022
- 2022-04-28 CN CN202210470795.7A patent/CN114821236A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115733705A (en) * | 2022-11-08 | 2023-03-03 | 深圳绿米联创科技有限公司 | Space-based information processing method and device, electronic equipment and storage medium |
| CN116226655A (en) * | 2022-12-05 | 2023-06-06 | 广州视声智能股份有限公司 | Smart home environment sensing method and device, storage medium and electronic equipment |
| CN117706954A (en) * | 2024-02-06 | 2024-03-15 | 青岛海尔科技有限公司 | Method and device for generating scene, storage medium and electronic device |
| CN117708680A (en) * | 2024-02-06 | 2024-03-15 | 青岛海尔科技有限公司 | Method and device for improving accuracy of classification model, storage medium and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114821236A (en) | Smart home environment sensing method, system, storage medium and electronic device | |
| CN108039988B (en) | Equipment control processing method and device | |
| CN109188928A (en) | Method and apparatus for controlling smart home device | |
| US11334933B2 (en) | Method, system, and manufacture for inferring user lifestyle and preference information from images | |
| CN108197635B (en) | Cooking mode display method and device and range hood | |
| CN110490688A (en) | Method of Commodity Recommendation and device | |
| US11966317B2 (en) | Electronic device and method for controlling same | |
| JP6400834B2 (en) | RECOMMENDATION DEVICE, RECOMMENDATION DETERMINING METHOD, AND COMPUTER PROGRAM | |
| CN109856980A (en) | Smart home device recommended method, device, Internet of things system and Cloud Server | |
| CN109118282B (en) | Dual-mode mutual inductance intelligent space user portrait management method and terminal | |
| CN110989390A (en) | Smart home control method and device | |
| CN107883520B (en) | Reminding method and device based on air conditioning equipment and terminal | |
| CN111248716B (en) | Food cooking control method, image processing method and device and cooking equipment | |
| CN116484083A (en) | Dish information display method and device, storage medium and electronic device | |
| CN115481284A (en) | Cosmetic method and device based on cosmetic box, storage medium and electronic device | |
| CN114925158A (en) | Sentence text intention recognition method and device, storage medium and electronic device | |
| CN112188252A (en) | Television reminding-based old people accompanying method and device and readable storage medium | |
| CN116540557A (en) | Audiovisual combined intelligent household appliance terminal control method, device and storage medium | |
| CN110941196A (en) | Intelligent panel, multi-level interaction method based on angle detection and storage medium | |
| CN111009305A (en) | Storage medium, intelligent panel and food material recommendation method thereof | |
| CN111126163A (en) | Intelligent panel, interaction method based on face angle detection and storage medium | |
| CN106406103A (en) | Intelligent household control system based on behavior habits | |
| CN108397974A (en) | Food control method and device | |
| CN114911381B (en) | Interactive feedback method and device, storage medium and electronic device | |
| CN114995726B (en) | Method and device for determining interaction mode, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |