CN116866103A - Indoor household appliance interaction intelligent judgment method based on nerf - Google Patents
Indoor household appliance interaction intelligent judgment method based on nerf Download PDFInfo
- Publication number
- CN116866103A CN116866103A CN202310941429.XA CN202310941429A CN116866103A CN 116866103 A CN116866103 A CN 116866103A CN 202310941429 A CN202310941429 A CN 202310941429A CN 116866103 A CN116866103 A CN 116866103A
- Authority
- CN
- China
- Prior art keywords
- nerf
- model
- household appliance
- appliance
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 91
- 230000003993 interaction Effects 0.000 title claims abstract description 21
- 230000008569 process Effects 0.000 claims abstract description 36
- 230000008447 perception Effects 0.000 claims abstract description 13
- 230000000007 visual effect Effects 0.000 claims description 15
- 238000013527 convolutional neural network Methods 0.000 claims description 12
- 238000005457 optimization Methods 0.000 claims description 9
- 238000009877 rendering Methods 0.000 claims description 9
- 230000006399 behavior Effects 0.000 claims description 6
- 238000013135 deep learning Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 6
- 230000033001 locomotion Effects 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 4
- 230000008901 benefit Effects 0.000 claims description 3
- 230000000295 complement effect Effects 0.000 claims description 3
- 238000013136 deep learning model Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 claims description 3
- 230000005484 gravity Effects 0.000 claims description 3
- 210000003128 head Anatomy 0.000 claims description 3
- 238000005286 illumination Methods 0.000 claims description 3
- 230000010354 integration Effects 0.000 claims description 3
- 210000003127 knee Anatomy 0.000 claims description 3
- 230000004807 localization Effects 0.000 claims description 3
- 230000001902 propagating effect Effects 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 6
- 230000009471 action Effects 0.000 description 3
- 241000238558 Eucarida Species 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/2803—Home automation networks
- H04L12/2807—Exchanging configuration information on appliance services in a home automation network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/724—User interfaces specially adapted for cordless or mobile telephones
- H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Graphics (AREA)
- Human Computer Interaction (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a method for intelligently judging interaction of indoor household appliances based on a surf, which specifically comprises the following steps: s1, scanning by a mobile phone: in the process, a user only needs to shoot a video which covers all the IoT devices in the room by using a mobile phone, and the invention relates to the technical field of intelligent interaction systems. According to the intelligent judging method for indoor household appliance interaction based on the NERF, indoor scanning is carried out by using a mobile phone, so that a user can directly use the mobile phone to acquire information of an indoor environment without purchasing additional hardware, and indoor environment reconstruction is carried out by using a NERF technology: the existing intelligent home system is generally based on preset room layout and equipment positions, in the scheme, the indoor environment of the indoor intelligent home control system based on the Nerf reconstruction information and gesture perception is automatically reconstructed through mobile phone scanning and NeRF technology, so that usability and flexibility of the system are certainly improved, and the intelligent home control is performed by using a digital twin model.
Description
Technical Field
The invention relates to the technical field of intelligent interaction systems, in particular to a method for intelligently judging interaction of indoor household appliances based on a nerf.
Background
NeRF is a technology for generating new and synthesized views, which is suitable for complex scenes, such as indoor and outdoor environments, and the core idea is to learn a neural network, the network takes a point and a viewing direction in a 3D space as input, the color and opacity of the point in the direction are output, intelligent voice interaction is a new generation interaction mode based on voice input, you can obtain feedback results through speaking, a typical application scene, namely a voice assistant, and intelligent voice interaction applications are rapidly developed since iPhone 4S derives SIRI, and Chinese typical intelligent voice interaction applications are as follows: the voice assistant and the message flying word point of the worm hole are accepted by more and more users.
In order to monitor the indoor environment, the existing user also needs to purchase other equipment additionally, the user needs to manually input the position of the equipment or configure the layout of the room, the whole operation has a certain limitation, and the indoor environment cannot be reflected in real time in the prior art, so that the use efficiency of the internal intelligent equipment is lower.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides the indoor household appliance interaction intelligent judging method based on the nerf, which solves the problem that the use efficiency of the internal intelligent equipment is lower due to certain limitation of the whole operation.
In order to achieve the above purpose, the invention is realized by the following technical scheme: an intelligent judging method for indoor household appliance interaction based on a nerf specifically comprises the following steps:
s1, scanning by a mobile phone: in the process, the user only needs to shoot a video covering all IoT devices in the room with the mobile phone;
s2, deducing camera external parameters:
(3) Feature extraction and matching: when the features are extracted from all the images, the COLMAP starts to perform feature matching, wherein feature matching refers to finding out the same feature points between two or more images, feature descriptors, a vector representing the intensity pattern of pixels around the features plays a key role at this stage, and features with the most similar (i.e. closest) descriptors can be found in different images by using, for example, the euclidean distance method, and are considered as matched feature points;
(4) Sparse reconstruction: with the matched feature points, we can create a sparse three-dimensional model by computing the position of the feature points in space and the position and pose of the camera by solving an optimization problem called "beam-method adjustment", which is basically a process that minimizes projection errors. Projection errors are the distances between the points observed on the image and the 3D points projected by the camera model. In this process, the algorithm will try to find a set of camera parameters (including the position and pose of each camera) that minimize all projection errors, as well as the position of the 3D points, this optimization problem is usually solved by iterative linearization methods, such as the gauss-newton method or the Li Wen berg-quard method. In each iteration, a parameter update is searched to further reduce projection errors, and when the optimization converges, the position and the posture of the camera and the three-dimensional coordinates of the matched feature points can be obtained;
s3, reconstructing an indoor environment by using the NeRF according to the image to obtain a NeRF model;
s4, according to the NeRF model, an image of any view angle in any direction in the room can be obtained, and the image can be obtained by the method:
(1) Position of fixed camera in room
The part of the method is the same as that of the part 2, and the position parameters of the fixed camera can be easily obtained through feature point matching and open reconstruction by using the existing image sequence and NeRF complementary rendering;
(2) The position of the home appliances at each position in the room, and two methods are comprehensively judged by the part:
a.3D object identification
i. Generating a 3D model: first, using the NeRF model, we can generate a 3D model of the room from multiple perspectives, including information on the shape, color, and location of objects, which can cover every corner in the room, so we can obtain the specific location of the appliance from this model;
object recognition: then we need to identify the home appliance in the 3D model. This can be achieved by training a deep learning model that can identify various appliances in the 3D model, e.g., we can train a 3D convolutional neural network that can process 3D data and identify the shape of the appliances;
household appliance positioning: once we identify the appliance in the 3D model, we can determine the location of the appliance, in particular we can find the center of gravity of the appliance model, which can be used as the location of the appliance, which is in 3D space, which can be used to guide the subsequent appliance control;
b, depth perception judgment of 2D:
firstly, carrying out target identification on a video sequence, determining the position of a household appliance in space, then respectively rendering a plurality of view angles facing the household appliance at different positions by using the surf for the position of each household appliance, and obtaining the space information of the household appliance according to the depth perception principle of the binocular camera:
i. target identification: first, it is necessary to identify various home appliances in the image, which can be achieved by target identification algorithms in deep learning, such as YOLO, SSD, or fast R-CNN, etc., which can identify predefined object categories in the image and give their positions in the image (typically represented by a bounding box);
spatial localization: next, the position of each home appliance in three-dimensional space needs to be determined. In order to achieve this, the ability of the NeRF model may be utilized, first, for each identified home appliance, depth information of each pixel point in its bounding box may be used as input, and corresponding three-dimensional coordinates may be obtained through the NeRF model, so that a rough position of each home appliance in a three-dimensional space may be obtained;
perspective rendering and depth perception: in order to acquire accurate three-dimensional information of the household appliance, images of the household appliance can be rendered from different visual angles by using a NeRF model, the principle is similar to that of a binocular camera or multi-view stereoscopic vision, the depth information of an object can be calculated by comparing images of the same object at different visual angles, and in the process, the NeRF model can generate images at any visual angle, so that enough visual angles can be provided for depth perception;
three-dimensional modeling: after the accurate three-dimensional position of the household appliance is obtained, a NeRF model can be further used for carrying out three-dimensional modeling on the household appliance, specifically, each part of the household appliance can be used as input, and the shape and the color of the part in the three-dimensional space can be obtained through the NeRF model, so that the three-dimensional model of the household appliance can be obtained, and the method is very useful for subsequent intelligent household control;
s5, modeling the intelligent household appliances in the room, and obtaining a digital twin;
s6, in order to ensure safety and privacy, a follow-up reasoning and control part is arranged at the edge end;
s7, the camera judges and infers the gesture of the person, and then judges according to the arrival of the position of the intelligent household appliance:
a. human body posture estimation: firstly, human body posture estimation needs to be carried out in a video stream acquired from a camera, which is an important problem in computer vision, the aim is to find key points of a human body such as a head, a shoulder, an elbow, a knee, a foot and the like in an image or a video, then determine the spatial relationship among the key points, so as to represent the posture of the human body, and a plurality of deep learning methods can be used for human body posture estimation such as OpenPose, alphaPose, and the methods firstly use a Convolutional Neural Network (CNN) to process the image, then find the key points of the human body through regression or classification tasks, and the algorithms can process the posture estimation of single person or multiple persons and can process the posture estimation of 2D or 3D;
b. posture judgment and reasoning: once the pose information of the person is obtained, we can make further judgment and reasoning, for example, we can judge the behavior of the person by analyzing the position and movement of the key points, such as walking, sitting down, lifting hands, etc., and we can infer the attention of the person by observing the face direction of the person or the gaze point of eyes, which is important for understanding the intention of the person;
c. judging according to the position of the household appliance: having knowledge of the person's posture and behavior, we also need to consider the location of the smart home, e.g. if the person is looking at the television, he may want to turn on the television; if a person is in the kitchen and looking into the refrigerator, he may want to open the refrigerator, during which we need to know the location of each appliance, which can be obtained by the earlier NeRF modeling, and then we can determine which appliance the person may want to interact with based on the person's location and posture and the location of the appliance.
Preferably, in said S2 (1), feature extraction and matching are the first steps in the workflow and also the basic steps in computer vision, the main goal of this stage being to find and describe specific points in the image that are representative of the object or scene in the image, such as corner points, edges or other salient local features that have unique image properties so that they can be identified and matched with each other between multiple images, these feature points often being used to identify images and estimate camera motion, feature extractors can find and describe these features at various scales, typically using SIFT algorithms, with rotational, scale and illumination invariance.
Preferably, in the step S3, neRF is a technique for generating a new and synthesized view, which is suitable for complex scenes, such as indoor and outdoor environments, and its core idea is to learn a neural network, which takes as input a point in 3D space and a viewing direction, and outputs the color and opacity of the point in that direction, and the main advantage of NeRF is that it can generate a high-quality and realistic synthesized view, and at the same time finely model details of the scene (such as shadows and reflections). However, its main disadvantage is the large computational overhead, since 5D integration is required once for every new view generated.
Preferably, in the step S3, the workflow of NeRF is as follows:
a. input: neRF receives as input a set of 2D images, which should include multiple views of the scene, along with the camera parameters of those images (including camera position and orientation of each image), so that global information of the scene can be obtained;
b. training process: in training, neRF takes as input each point in the scene (in 3D space) and a view direction, then predicts the color and volume density of the point in that direction, which determines the degree of attenuation of the light as it passes through the point, this process can be seen as a 5D problem (point in 3D space +2D visual angle), and in training, neRF uses an integral loss function that measures the error between the predicted color and the observed color of the network, this integral process essentially simulates the propagation of the light in the scene, and by back propagation and gradient descent, the parameters of NeRF are optimized so that the predicted color is as close as possible to the observed color;
c. and (3) outputting: after training is completed, the NeRF may generate a new, synthesized view. This is achieved by forward propagating each point in the scene and a new direction of observation, and then integrating the predicted color and volume density.
Preferably, in the step S6, the digital twin is converted to form a data set composed of the room space and the coordinates and types of different objects in the room, and the subsequent judgment is performed by using the data set.
Advantageous effects
The invention provides a method for intelligently judging interaction of indoor household appliances based on a surf. Compared with the prior art, the method has the following beneficial effects: according to the intelligent judging method for indoor household appliance interaction based on the NERF, indoor scanning is carried out by using a mobile phone, so that a user can directly use the mobile phone to acquire information of an indoor environment without purchasing additional hardware, and indoor environment reconstruction is carried out by using a NERF technology: the existing smart home system is usually based on preset room layout and equipment position, in the scheme, the indoor environment of the indoor intelligent home control system is automatically rebuilt based on the NeRF rebuilding information and attitude sensing through a mobile phone scanning and NeRF technology, the usability and flexibility of the system are certainly improved, a digital twin model is used for intelligent home control, the existing smart home system rarely uses the digital twin model for equipment control, a user does not need to manually input the position of equipment or configure the room layout by utilizing the NeRF technology, the system can automatically acquire and model the information, and the real-time performance and precision of the smart home system are improved: the digital twin model can reflect the state of the indoor environment in real time, so that the equipment control is more accurate and efficient, through automatic scanning and modeling of the indoor environment, a user can obtain more concise and efficient experience when configuring and using the intelligent home system, the user can rescan and update the indoor environment through a mobile phone at any time, the system can adapt to the change of the environment, and through the digital twin model updated in real time, the intelligent home system can obtain more accurate equipment state information, so that the equipment control is more accurate.
Drawings
FIG. 1 is a flow chart of the steps of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, the present invention provides a technical solution: an intelligent judging method for indoor household appliance interaction based on a nerf specifically comprises the following steps:
s1, scanning by a mobile phone: in the process, the user only needs to shoot a video covering all IoT devices in the room with the mobile phone;
s2, deducing camera external parameters:
(5) Feature extraction and matching: when the features are extracted from all the images, the COLMAP starts to perform feature matching, wherein feature matching refers to finding out the same feature points between two or more images, feature descriptors, a vector representing the intensity pattern of pixels around the features plays a key role at this stage, and features with the most similar (i.e. closest) descriptors can be found in different images by using, for example, the euclidean distance method, and are considered as matched feature points;
(6) Sparse reconstruction: with the matched feature points, we can create a sparse three-dimensional model by computing the position of the feature points in space and the position and pose of the camera by solving an optimization problem called "beam-method adjustment", which is basically a process that minimizes projection errors. Projection errors are the distances between the points observed on the image and the 3D points projected by the camera model. In this process, the algorithm will try to find a set of camera parameters (including the position and pose of each camera) that minimize all projection errors, as well as the position of the 3D points, this optimization problem is usually solved by iterative linearization methods, such as the gauss-newton method or the Li Wen berg-quard method. In each iteration, a parameter update is searched to further reduce projection errors, and when the optimization converges, the position and the posture of the camera and the three-dimensional coordinates of the matched feature points can be obtained;
s3, reconstructing an indoor environment by using the NeRF according to the image to obtain a NeRF model;
s4, according to the NeRF model, an image of any view angle in any direction in the room can be obtained, and the image can be obtained by the method:
(1) Position of fixed camera in room
The part of the method is the same as that of the part 2, and the position parameters of the fixed camera can be easily obtained through feature point matching and open reconstruction by using the existing image sequence and NeRF complementary rendering;
(2) The position of the home appliances at each position in the room, and two methods are comprehensively judged by the part:
a.3D object identification
i. Generating a 3D model: first, using the NeRF model, we can generate a 3D model of the room from multiple perspectives, including information on the shape, color, and location of objects, which can cover every corner in the room, so we can obtain the specific location of the appliance from this model;
object recognition: then we need to identify the home appliance in the 3D model. This can be achieved by training a deep learning model that can identify various appliances in the 3D model, e.g., we can train a 3D convolutional neural network that can process 3D data and identify the shape of the appliances;
household appliance positioning: once we identify the appliance in the 3D model, we can determine the location of the appliance, in particular we can find the center of gravity of the appliance model, which can be used as the location of the appliance, which is in 3D space, which can be used to guide the subsequent appliance control;
b, depth perception judgment of 2D:
firstly, carrying out target identification on a video sequence, determining the position of a household appliance in space, then respectively rendering a plurality of view angles facing the household appliance at different positions by using the surf for the position of each household appliance, and obtaining the space information of the household appliance according to the depth perception principle of the binocular camera:
i. target identification: first, it is necessary to identify various home appliances in the image, which can be achieved by target identification algorithms in deep learning, such as YOLO, SSD, or fast R-CNN, etc., which can identify predefined object categories in the image and give their positions in the image (typically represented by a bounding box);
spatial localization: next, the position of each home appliance in three-dimensional space needs to be determined. In order to achieve this, the ability of the NeRF model may be utilized, first, for each identified home appliance, depth information of each pixel point in its bounding box may be used as input, and corresponding three-dimensional coordinates may be obtained through the NeRF model, so that a rough position of each home appliance in a three-dimensional space may be obtained;
perspective rendering and depth perception: in order to acquire accurate three-dimensional information of the household appliance, images of the household appliance can be rendered from different visual angles by using a NeRF model, the principle is similar to that of a binocular camera or multi-view stereoscopic vision, the depth information of an object can be calculated by comparing images of the same object at different visual angles, and in the process, the NeRF model can generate images at any visual angle, so that enough visual angles can be provided for depth perception;
three-dimensional modeling: after the accurate three-dimensional position of the household appliance is obtained, a NeRF model can be further used for carrying out three-dimensional modeling on the household appliance, specifically, each part of the household appliance can be used as input, and the shape and the color of the part in the three-dimensional space can be obtained through the NeRF model, so that the three-dimensional model of the household appliance can be obtained, and the method is very useful for subsequent intelligent household control;
s5, modeling the intelligent household appliances in the room, and obtaining a digital twin;
s6, in order to ensure safety and privacy, a follow-up reasoning and control part is arranged at the edge end;
s7, the camera judges and infers the gesture of the person, and then judges according to the arrival of the position of the intelligent household appliance:
a. human body posture estimation: firstly, human body posture estimation needs to be carried out in a video stream acquired from a camera, which is an important problem in computer vision, the aim is to find key points of a human body such as a head, a shoulder, an elbow, a knee, a foot and the like in an image or a video, then determine the spatial relationship among the key points, so as to represent the posture of the human body, and a plurality of deep learning methods can be used for human body posture estimation such as OpenPose, alphaPose, and the methods firstly use a Convolutional Neural Network (CNN) to process the image, then find the key points of the human body through regression or classification tasks, and the algorithms can process the posture estimation of single person or multiple persons and can process the posture estimation of 2D or 3D;
b. posture judgment and reasoning: once the pose information of the person is obtained, we can make further judgment and reasoning, for example, we can judge the behavior of the person by analyzing the position and movement of the key points, such as walking, sitting down, lifting hands, etc., and we can infer the attention of the person by observing the face direction of the person or the gaze point of eyes, which is important for understanding the intention of the person;
c. judging according to the position of the household appliance: having knowledge of the person's posture and behavior, we also need to consider the location of the smart home, e.g. if the person is looking at the television, he may want to turn on the television; if a person is in the kitchen and looking into the refrigerator, he may want to open the refrigerator, during which we need to know the location of each appliance, which can be obtained by the earlier NeRF modeling, and then we can determine which appliance the person may want to interact with based on the person's location and posture and the location of the appliance.
In the present invention, in S2 (1), feature extraction and matching are the first step in the workflow, and also the fundamental step in computer vision, the main goal of this stage is to find and describe special points in the image that are representative of the object or scene in the image, such as corner points, edges or other significant local features that have unique image attributes so that they can be identified and matched with each other between multiple images, these feature points are often used to identify images and estimate camera motion, and feature extractors can find and describe these features at various scales, typically using SIFT algorithms, with rotational, scale and illumination invariance.
In the present invention, in S3, neRF is a technique for generating new, synthesized views, suitable for complex scenes such as indoor and outdoor environments, whose core idea is to learn a neural network that takes as input a point in 3D space and a viewing direction, and outputs the color and opacity of the point in that direction, the main advantage of NeRF is that it can generate high quality, realistic synthesized views, while also finely modeling the details of the scene (such as shadows and reflections). However, its main disadvantage is the large computational overhead, since 5D integration is required once for every new view generated.
In the present invention, in S3, the workflow of NeRF is as follows:
a. input: neRF receives as input a set of 2D images, which should include multiple views of the scene, along with the camera parameters of those images (including camera position and orientation of each image), so that global information of the scene can be obtained;
b. training process: in training, neRF takes as input each point in the scene (in 3D space) and a view direction, then predicts the color and volume density of the point in that direction, which determines the degree of attenuation of the light as it passes through the point, this process can be seen as a 5D problem (point in 3D space +2D visual angle), and in training, neRF uses an integral loss function that measures the error between the predicted color and the observed color of the network, this integral process essentially simulates the propagation of the light in the scene, and by back propagation and gradient descent, the parameters of NeRF are optimized so that the predicted color is as close as possible to the observed color;
c. and (3) outputting: after training is completed, the NeRF may generate a new, synthesized view. This is achieved by forward propagating each point in the scene and a new direction of observation, and then integrating the predicted color and volume density.
In the invention, in S6, the digital twin is converted to form a data set composed of the room space and the coordinates and types of different objects in the room, and the subsequent judgment is carried out by using the data set.
And all that is not described in detail in this specification is well known to those skilled in the art.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (5)
1. An intelligent judging method for indoor household appliance interaction based on a nerf is characterized by comprising the following steps of: the method specifically comprises the following steps:
s1, scanning by a mobile phone: in the process, the user only needs to shoot a video covering all IoT devices in the room with the mobile phone;
s2, deducing camera external parameters:
(1) Feature extraction and matching: when the features are extracted from all the images, the COLMAP starts to perform feature matching, wherein feature matching refers to finding out the same feature points between two or more images, feature descriptors, a vector representing the intensity pattern of pixels around the features plays a key role at this stage, and features with the most similar (i.e. closest) descriptors can be found in different images by using, for example, the euclidean distance method, and are considered as matched feature points;
(2) Sparse reconstruction: with the matched feature points, we can create a sparse three-dimensional model by computing the position of the feature points in space and the position and pose of the camera by solving an optimization problem called "beam-method adjustment", which is basically a process that minimizes projection errors. Projection errors are the distances between the points observed on the image and the 3D points projected by the camera model. In this process, the algorithm will try to find a set of camera parameters (including the position and pose of each camera) that minimize all projection errors, as well as the position of the 3D points, this optimization problem is usually solved by iterative linearization methods, such as the gauss-newton method or the Li Wen berg-quard method. In each iteration, a parameter update is searched to further reduce projection errors, and when the optimization converges, the position and the posture of the camera and the three-dimensional coordinates of the matched feature points can be obtained;
s3, reconstructing an indoor environment by using the NeRF according to the image to obtain a NeRF model;
s4, according to the NeRF model, an image of any view angle in any direction in the room can be obtained, and the image can be obtained by the method:
(1) Position of fixed camera in room
The part of the method is the same as that of the part 2, and the position parameters of the fixed camera can be easily obtained through feature point matching and open reconstruction by using the existing image sequence and NeRF complementary rendering;
(2) The position of the home appliances at each position in the room, and two methods are comprehensively judged by the part:
a.3D object identification
i. Generating a 3D model: first, using the NeRF model, we can generate a 3D model of the room from multiple perspectives, including information on the shape, color, and location of objects, which can cover every corner in the room, so we can obtain the specific location of the appliance from this model;
object recognition: then we need to identify the home appliance in the 3D model. This can be achieved by training a deep learning model that can identify various appliances in the 3D model, e.g., we can train a 3D convolutional neural network that can process 3D data and identify the shape of the appliances;
household appliance positioning: once we identify the appliance in the 3D model, we can determine the location of the appliance, in particular we can find the center of gravity of the appliance model, which can be used as the location of the appliance, which is in 3D space, which can be used to guide the subsequent appliance control;
b, depth perception judgment of 2D:
firstly, carrying out target identification on a video sequence, determining the position of a household appliance in space, then respectively rendering a plurality of view angles facing the household appliance at different positions by using the surf for the position of each household appliance, and obtaining the space information of the household appliance according to the depth perception principle of the binocular camera:
i. target identification: first, it is necessary to identify various home appliances in the image, which can be achieved by target identification algorithms in deep learning, such as YOLO, SSD, or fast R-CNN, etc., which can identify predefined object categories in the image and give their positions in the image (typically represented by a bounding box);
spatial localization: next, the position of each home appliance in three-dimensional space needs to be determined. In order to achieve this, the ability of the NeRF model may be utilized, first, for each identified home appliance, depth information of each pixel point in its bounding box may be used as input, and corresponding three-dimensional coordinates may be obtained through the NeRF model, so that a rough position of each home appliance in a three-dimensional space may be obtained;
perspective rendering and depth perception: in order to acquire accurate three-dimensional information of the household appliance, images of the household appliance can be rendered from different visual angles by using a NeRF model, the principle is similar to that of a binocular camera or multi-view stereoscopic vision, the depth information of an object can be calculated by comparing images of the same object at different visual angles, and in the process, the NeRF model can generate images at any visual angle, so that enough visual angles can be provided for depth perception;
three-dimensional modeling: after the accurate three-dimensional position of the household appliance is obtained, a NeRF model can be further used for carrying out three-dimensional modeling on the household appliance, specifically, each part of the household appliance can be used as input, and the shape and the color of the part in the three-dimensional space can be obtained through the NeRF model, so that the three-dimensional model of the household appliance can be obtained, and the method is very useful for subsequent intelligent household control;
s5, modeling the intelligent household appliances in the room, and obtaining a digital twin;
s6, in order to ensure safety and privacy, a follow-up reasoning and control part is arranged at the edge end;
s7, the camera judges and infers the gesture of the person, and then judges according to the arrival of the position of the intelligent household appliance:
a. human body posture estimation: firstly, human body posture estimation needs to be carried out in a video stream acquired from a camera, which is an important problem in computer vision, the aim is to find key points of a human body such as a head, a shoulder, an elbow, a knee, a foot and the like in an image or a video, then determine the spatial relationship among the key points, so as to represent the posture of the human body, and a plurality of deep learning methods can be used for human body posture estimation such as OpenPose, alphaPose, and the methods firstly use a Convolutional Neural Network (CNN) to process the image, then find the key points of the human body through regression or classification tasks, and the algorithms can process the posture estimation of single person or multiple persons and can process the posture estimation of 2D or 3D;
b. posture judgment and reasoning: once the pose information of the person is obtained, we can make further judgment and reasoning, for example, we can judge the behavior of the person by analyzing the position and movement of the key points, such as walking, sitting down, lifting hands, etc., and we can infer the attention of the person by observing the face direction of the person or the gaze point of eyes, which is important for understanding the intention of the person;
c. judging according to the position of the household appliance: having knowledge of the person's posture and behavior, we also need to consider the location of the smart home, e.g. if the person is looking at the television, he may want to turn on the television; if a person is in the kitchen and looking into the refrigerator, he may want to open the refrigerator, during which we need to know the location of each appliance, which can be obtained by the earlier NeRF modeling, and then we can determine which appliance the person may want to interact with based on the person's location and posture and the location of the appliance.
2. The intelligent judging method for indoor household appliance interaction based on the surf is characterized by comprising the following steps of: in said S2 (1), feature extraction and matching, which is the first step in the workflow and is also the fundamental step in computer vision, the main goal of this stage is to find and describe special points in the image that are representative of the object or scene in the image, such as corner points, edges or other significant local features that have unique image properties so that they can be identified and matched with each other between multiple images, these feature points are often used to estimate camera motion for images and feature extractors can find and describe these features at various scales, typically using SIFT algorithms, with rotational, scale and illumination invariance.
3. The intelligent judging method for indoor household appliance interaction based on the surf is characterized by comprising the following steps of: in S3, neRF is a technique for generating new and synthesized views, suitable for complex scenes, such as indoor and outdoor environments, and its core idea is to learn a neural network, which takes as input a point in 3D space and a viewing direction, and outputs the color and opacity of the point in that direction, the main advantage of NeRF is that it can generate high-quality, realistic synthesized views, while finely modeling the details of the scene (such as shadows and reflections). However, its main disadvantage is the large computational overhead, since 5D integration is required once for every new view generated.
4. The intelligent judging method for indoor household appliance interaction based on the surf is characterized by comprising the following steps of: in the step S3, the workflow of NeRF is as follows:
a. input: neRF receives as input a set of 2D images, which should include multiple views of the scene, along with the camera parameters of those images (including camera position and orientation of each image), so that global information of the scene can be obtained;
b. training process: in training, neRF takes as input each point in the scene (in 3D space) and a view direction, then predicts the color and volume density of the point in that direction, which determines the degree of attenuation of the light as it passes through the point, this process can be seen as a 5D problem (point in 3D space +2D visual angle), and in training, neRF uses an integral loss function that measures the error between the predicted color and the observed color of the network, this integral process essentially simulates the propagation of the light in the scene, and by back propagation and gradient descent, the parameters of NeRF are optimized so that the predicted color is as close as possible to the observed color;
c. and (3) outputting: after training is completed, the NeRF may generate a new, synthesized view. This is achieved by forward propagating each point in the scene and a new direction of observation, and then integrating the predicted color and volume density.
5. The intelligent judging method for indoor household appliance interaction based on the surf is characterized by comprising the following steps of: in the step S6, the digital twin is converted to form a data set composed of the room space and the coordinates and types of different objects in the room, and the data set is used for subsequent judgment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310941429.XA CN116866103A (en) | 2023-07-28 | 2023-07-28 | Indoor household appliance interaction intelligent judgment method based on nerf |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310941429.XA CN116866103A (en) | 2023-07-28 | 2023-07-28 | Indoor household appliance interaction intelligent judgment method based on nerf |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116866103A true CN116866103A (en) | 2023-10-10 |
Family
ID=88223412
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310941429.XA Withdrawn CN116866103A (en) | 2023-07-28 | 2023-07-28 | Indoor household appliance interaction intelligent judgment method based on nerf |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116866103A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118011893A (en) * | 2024-01-09 | 2024-05-10 | 西乔科技南京有限公司 | Man-machine interaction system based on artificial intelligence |
-
2023
- 2023-07-28 CN CN202310941429.XA patent/CN116866103A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118011893A (en) * | 2024-01-09 | 2024-05-10 | 西乔科技南京有限公司 | Man-machine interaction system based on artificial intelligence |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11215465B2 (en) | Methods for simultaneous localization and mapping (SLAM) and related apparatus and systems | |
| US10482674B1 (en) | System and method for mobile augmented reality | |
| US10977818B2 (en) | Machine learning based model localization system | |
| Karsch et al. | Depth transfer: Depth extraction from video using non-parametric sampling | |
| US10311624B2 (en) | Single shot capture to animated vr avatar | |
| CN111353506B (en) | Adaptive line-of-sight estimation method and apparatus | |
| KR101650799B1 (en) | Method for the real-time-capable, computer-assisted analysis of an image sequence containing a variable pose | |
| JP3512992B2 (en) | Image processing apparatus and image processing method | |
| US20190087067A1 (en) | Navigation point selection for navigating through virtual environments | |
| US20180330184A1 (en) | Determining an architectural layout | |
| WO2020042970A1 (en) | Three-dimensional modeling method and device therefor | |
| CN108895981A (en) | A kind of method for three-dimensional measurement, device, server and storage medium | |
| US20210158593A1 (en) | Pose selection and animation of characters using video data and training techniques | |
| CN113628327A (en) | Head three-dimensional reconstruction method and equipment | |
| CN116012564B (en) | Equipment and method for intelligent fusion of three-dimensional model and live-action photo | |
| CN116866103A (en) | Indoor household appliance interaction intelligent judgment method based on nerf | |
| Schütt et al. | Semantic interaction in augmented reality environments for microsoft hololens | |
| Wu et al. | [Retracted] 3D Film Animation Image Acquisition and Feature Processing Based on the Latest Virtual Reconstruction Technology | |
| Taylor et al. | VR props: an end-to-end pipeline for transporting real objects into virtual and augmented environments | |
| CN117711066A (en) | Three-dimensional human body posture estimation method, device, equipment and medium | |
| US20210158565A1 (en) | Pose selection and animation of characters using video data and training techniques | |
| US20230088058A1 (en) | Method and apparatus for the computer-aided completion of a 3d partial model formed by points | |
| EP4275173B1 (en) | Computer-implemented reconstruction of interior rooms | |
| KR20120008211A (en) | Method for object recognition and pose estimation at robot | |
| CN115497029A (en) | Video processing method, device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20231010 |
|
| WW01 | Invention patent application withdrawn after publication |