WO2023125393A1

WO2023125393A1 - Method and device for controlling smart home appliance, and mobile terminal

Info

Publication number: WO2023125393A1
Application number: PCT/CN2022/141870
Authority: WO
Inventors: 肖雪莹; 杨东齐; 吴信新; 高铭英; 叶宏
Original assignee: 华为技术有限公司
Priority date: 2021-12-31
Filing date: 2022-12-26
Publication date: 2023-07-06
Also published as: CN116418610A

Abstract

The present application provides a method and device for controlling a smart home appliance, and a mobile terminal. In the method for controlling a smart home appliance, virtual home appliances are configured to correspond to home appliances to be controlled, so that a user does not need to select a home appliance according to text description, and can determine a home appliance to be controlled in a more intuitive manner, thereby reducing operational complexity in scenario creation processes, and improving user experience.

Description

Method, device and mobile terminal for controlling smart home equipment

technical field

The present application relates to the field of smart home technology, in particular to a method, device, and mobile terminal for controlling smart home devices.

Background technique

The clustered control of smart home is usually realized by simultaneously operating multiple smart home devices on one application program (APP). This method also supports user-defined scenes. Users can set the scene by themselves to choose the home equipment and control parameters they need. For example, when leaving home to automatically turn off the lights, turn off the air conditioner, and turn off the TV, the scene of leaving home, the scene of watching movies, etc. The user needs to select the corresponding household equipment and the corresponding control parameters according to the text information provided.

However, when there are too many smart home devices, especially when there are multiple devices of the same type, it is difficult to locate the required device and requires gradual trial and error. The control parameters must also be determined one by one, which makes the scene creation process complicated and reduces user experience.

application content

The present application provides a method, device and mobile terminal for controlling smart home equipment, so as to simplify the process of creating a smart home scene and improve user experience.

In the first aspect, the present application provides a method for controlling smart home devices, which is applied to mobile terminals (such as mobile phones, tablet computers, etc.), including: acquiring multimedia files; determining the virtual home devices to be controlled through augmented reality AR images household equipment, and determine control parameters applicable to the multimedia file, wherein the virtual household equipment corresponds to the household equipment to be controlled, and the control parameters are used to control the household equipment to be controlled. The method for controlling a smart home device corresponds the virtual home device with the home device to be controlled, so that the user does not need to select the home device according to the text description, and can determine the home device to be controlled in a more intuitive manner, reducing the scene creation process. The complexity of the operation in the system improves the user experience.

In a possible implementation manner, the acquisition of the multimedia file includes: acquisition of a picture, video or audio file locally or from a network. Acquiring multimedia files locally can improve the efficiency of determining control parameters, and acquiring multimedia files through the network can make the determined control parameters more diverse, meet more user needs, and improve the flexibility of controlling home appliances.

In a possible implementation manner, determining the home device to be controlled through the virtual home device in the AR image includes: scanning the home device to be controlled and its environment, and generating an AR image containing the virtual home device. The AR image is obtained by scanning, so that the corresponding household equipment to be controlled is added, and the operation is simple and easy to implement.

In a possible implementation manner, after the aforementioned generation of the AR image containing the virtual home device, it further includes: receiving a culling instruction input by a user, and excluding objects in the virtual home device that are culled by the culling instruction. By removing the corresponding household equipment that is no longer controlled by eliminating the virtual household equipment, the household equipment to be controlled can be changed flexibly.

In a possible implementation manner, the above determination of the control parameters applicable to the multimedia file includes: determining the control parameters applicable to the multimedia file according to the characteristics of the multimedia file and a preset mapping relationship, wherein the mapping relationship is the mapping relationship between the feature of the multimedia file and the control parameter. The control parameters of the household equipment are automatically determined through the mapping relationship between the characteristics of the multimedia file and the control parameters, so that the user does not need to manually set the control parameters, which further reduces the complexity of the operation, reduces the labor of the user, and further improves the user experience.

In a possible implementation manner, determining the control parameters applicable to the multimedia file according to the characteristics of the multimedia file and the preset mapping relationship includes: according to the preset mapping relationship and the video frame characteristics of the multimedia file and/or audio features, according to the playback timing of the multimedia file, determine the set of control parameters applicable to different video frames of the multimedia file; or

When the video frame and/or audio of the multimedia file meet the preset condition, according to the preset mapping relationship and the video frame feature and/or audio feature of the original multimedia file, determine the video frame/audio that is applicable to the preset condition or audio control parameters. For example, when the multimedia file is a video file such as a movie, the control parameters applicable to all video frames of the movie can be determined. Since a power supply has tens of thousands of video frames, the control parameters applicable to these frames also correspond to There are thousands of them, which can be regarded as a set. In this way, the application can set different scenes (sets of control parameters of different home devices) for different movies, further improving the flexibility of smart home device control. Alternatively, the present application can configure control parameters of home devices for some video frames in a movie, satisfying users' diverse viewing needs.

In a possible implementation manner, determining the control parameter sets applicable to different video frames of the multimedia file according to the play sequence of the multimedia file includes: inserting the first control parameter in the control parameter set into the second control parameter set The execution time of the parameter; wherein, the first control parameter is the control parameter corresponding to the first video frame in the multimedia file, and the second control parameter is the control parameter corresponding to the second video frame in the multimedia file; the first video frame corresponds to The control parameter is the control parameter of the hysteresis response type household equipment (such as curtains), and the playing time difference between the first video frame and the second video frame is the time length for the hysteresis response type household equipment to respond to the control instruction. The control instruction is the An execution instruction of the first control parameter. For example, the control parameter applicable to the m-th frame is used to control a household device that will respond with a lag such as a curtain, assuming that the control parameter controls the curtain from opening to fully closing, and assuming that the curtain response takes 10 seconds to fully close, then the control parameter is used as Applicable to the control parameters of the video frame (such as the nth frame) played before the mth frame 10 seconds, then start to control the curtain when the nth frame is played, and after 10 seconds, the movie will play to the mth frame and the curtain will be completely closed, so as to achieve the expected The control effect further improves the flexibility of controlling home equipment.

In a possible implementation manner, the above-mentioned household equipment to be controlled is determined through the virtual household equipment in the AR image, and the control parameters applicable to the multimedia file are determined according to the characteristics of the multimedia file and the preset mapping relationship, including : Receive the time node input by the user, receive the first operation instruction input by the user by clicking on the virtual home device, and receive the second operation instruction input by the user by selecting a control option, and determine the home device to be controlled according to the first operation instruction According to the second operation instruction, the control parameters applicable to the multimedia file at the time node are determined, providing a way for the user to manually set the control parameters, and increasing the flexibility of setting the control parameters.

In a possible implementation manner, after determining the control parameters applicable to the multimedia file according to the characteristics of the multimedia file and the preset mapping relationship, it also includes: receiving the time node input by the user, and receiving the user's click on the virtual The first operation instruction input by the home equipment, and receiving the second operation instruction input by the user by selecting the control option, determining the home equipment to be controlled according to the first operation instruction, and determining the application for the multimedia file in accordance with the second operation instruction The control parameters at this time node combine the automatically determined control parameters with the manual control parameters, which can meet the user's modification needs, and further improve the flexibility of controlling household devices.

In a possible implementation manner, before determining the home device to be controlled according to the first operation instruction, it further includes: receiving a third operation instruction input by the user, and selecting a virtual home device for the user according to the third operation instruction Create groups. Since virtual home devices correspond to physical home devices and can be controlled by being visible, creating a group for virtual home devices also creates a group for physical home devices, which helps to improve the convenience of home device control.

In a possible implementation manner, the above-mentioned determination of the household device to be controlled according to the first operation instruction includes: determining a control parameter for controlling the group according to the first operation instruction, or determining a control parameter for controlling the user in the group according to the first operation instruction. Control parameters for selected home devices. If the user clicks on the created group, it indicates that all household devices in the group are household devices to be controlled, which improves the flexibility and convenience of determining the household devices to be controlled.

In a possible implementation manner, after determining the control parameters applicable to the multimedia file at the time node according to the second operation instruction, it further includes: displaying a prompt that the control parameters conflict and whether to choose to overwrite, for example, when the user determines When the control parameters conflict with the control parameters automatically determined through the mapping relationship, it can provide the user with a choice, which further improves the flexibility of determining the control parameters.

In a possible implementation manner, after determining the home device to be controlled through the virtual home device in the AR image, and determining the control parameters applicable to the multimedia file according to the characteristics of the multimedia file and the preset mapping relationship, It also includes: receiving a preview instruction input by the user, and displaying the status of the virtual home device under the control of the control parameter. Through the preview, the user can determine the control parameters of the household equipment under the condition that the specific control effect of the control parameters is clear, and the efficiency of determining the control parameters is improved.

In a possible implementation manner, after determining the home device to be controlled through the virtual home device in the AR image, and determining the control parameters applicable to the multimedia file according to the characteristics of the multimedia file and the preset mapping relationship, It also includes: receiving a sharing command input by the user, generating a configuration file according to the sharing command, and displaying sharing options; receiving a sharing channel command input by the user by selecting the sharing option, and uploading the configuration file according to the sharing channel in the sharing channel command , enabling the user to share the determined control parameters with friends, improving the convenience of the application of the control parameters.

In a second aspect, the present application provides an apparatus for controlling smart home equipment, which is applied to a mobile terminal, and includes: an acquisition module and a determination module. The acquisition module is used to acquire multimedia files; the determination module is used to determine the home equipment to be controlled through the virtual home equipment in the augmented reality AR image, and determine the control parameters applicable to the multimedia file, wherein the virtual home equipment and the Corresponding to the household device to be controlled, the control parameter is used to control the household device to be controlled.

In a possible implementation manner, the acquisition module is configured to acquire pictures, video or audio files locally or from a network.

In a possible implementation manner, the determination module is configured to scan the home device to be controlled and its environment, and generate an AR image including the virtual home device.

In a possible implementation manner, the above-mentioned apparatus further includes: a first receiving module and a culling module; the first receiving module is configured to receive the user-input Removing instruction; the removing module is used to remove the object removed by the removing instruction in the above-mentioned virtual home equipment.

In a possible implementation manner, the determination module is configured to determine control parameters applicable to the multimedia file according to the characteristics of the multimedia file and a preset mapping relationship, wherein the mapping relationship is the characteristics of the multimedia file and the above-mentioned Control the mapping relationship between parameters.

In a possible implementation manner, the determination module is configured to determine the time sequence applicable to the multimedia file according to the video frame features and/or audio features of the multimedia file and the preset mapping relationship, according to the playing sequence of the multimedia file. A set of control parameters for different video frames; or for when the video frame and/or audio of the above multimedia file meet the preset conditions, according to the video frame feature and/or audio feature of the above multimedia file and the above preset mapping relationship, Determine the control parameters applicable to the video frame and/or audio meeting the preset condition.

In a possible implementation manner, the determination module is configured to insert the first control parameter in the control parameter set into the execution time of the second control parameter; wherein, the first control parameter is the first control parameter in the multimedia file. The control parameter corresponding to the video frame, the second control parameter is the control parameter corresponding to the second video frame in the multimedia file; the control parameter corresponding to the first video frame is the control parameter of the household device of the hysteresis response type, and the first video The play time difference between the frame and the second video frame is the time period for the lag response type household device to respond to the control instruction, and the control instruction is an execution instruction of the first control parameter.

In a possible implementation manner, the determination module includes: a receiving submodule and a determination submodule; the receiving submodule is used to receive the time node input by the user, and receive the first operation instruction input by the user by clicking on the virtual home device, And receiving a second operation instruction input by the user by selecting a control option; the determining submodule is used to determine the above-mentioned household device to be controlled according to the first operation instruction, and determine the time node applicable to the above-mentioned multimedia file according to the second operation instruction. control parameters.

In a possible implementation manner, the above device further includes: a second receiving module, configured to receive the The time node input by the user, and receiving the first operation instruction input by the user by clicking on the virtual home device, and receiving the second operation instruction input by the user by selecting a control option; the determination module is used to determine the above-mentioned to-be-controlled The household device determines, according to the second operation instruction, the control parameters applicable to the multimedia file at the time node.

In a possible implementation manner, the above-mentioned apparatus further includes: a third receiving module and a group creation module; the third receiving module is configured to, before the above-mentioned determining module determines the household device to be controlled according to the above-mentioned first operation instruction, A third operation instruction input by the user is received; the group creation module is used to create a group for the virtual household device selected by the user according to the third operation instruction.

In a possible implementation manner, the determination module is configured to determine, according to the first operation instruction, some or all of the household devices in the group as the household devices to be controlled.

In a possible implementation manner, the above-mentioned device further includes: a first display module, configured to display the control parameters after the above-mentioned determining module determines the control parameters applicable to the above-mentioned multimedia file at the above-mentioned time node according to the second operation instruction Prompt for conflicts and whether to choose to override.

In a possible implementation manner, the above device further includes: a fourth receiving module and a second display module; the fourth receiving module is used to determine the home device to be controlled through the virtual home device in the AR image in the above determining module, And after determining the control parameters applicable to the above-mentioned multimedia files, receiving a preview instruction input by the user; the second display module is used to display the state of the above-mentioned virtual home equipment under the control of the above-mentioned control parameters.

In a possible implementation manner, the above-mentioned device further includes: a fifth receiving module, a generating module, a third display module, a sixth receiving module, and an uploading module; After the virtual home device determines the home device to be controlled and the control parameters applicable to the above-mentioned multimedia files, it receives a sharing instruction input by the user; the generating module is used to generate a configuration file according to the sharing instruction; the third display module is used to Displaying sharing options; the sixth receiving module is used to receive a sharing channel instruction input by the user by selecting the sharing option; the uploading module is used to upload the configuration file according to the sharing channel in the sharing channel instruction.

In a third aspect, the present application provides a mobile terminal, including: a processor, a memory, and a computer program. The memory stores the computer program. When the processor runs the computer program, the mobile terminal executes any one of the above-mentioned first aspects. described method.

In a fourth aspect, the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when it is run on a computer, the computer is made to execute the method of any one of the first aspect.

In a fifth aspect, the present application provides a system on a chip, which implements any one of the methods in the first aspect.

In a sixth aspect, the present application provides a computer program for performing the method of any one of the first aspect when the computer program is executed by a computer.

In a possible design, all or part of the program in the sixth aspect may be stored on a storage medium packaged with the processor, or part or all may be stored on a memory not packaged with the processor.

Description of drawings

FIG. 1 is a schematic diagram of a system architecture or scenario applied in an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a mobile terminal in an embodiment of the present application;

FIG. 3 is a schematic flowchart of a method for controlling smart home devices in an embodiment of the present application;

4(a) to 4(e) are schematic diagrams of interfaces of mobile terminals in the method for controlling smart home devices in the embodiment of the present application;

5(a) to 5(c) are schematic diagrams of the interface of the mobile terminal in another method for controlling smart home devices in the embodiment of the present application;

FIG. 6(a) to FIG. 6(d) are schematic diagrams of interfaces of mobile terminals in another method for controlling smart home devices in the embodiment of the present application;

FIG. 7 is a schematic diagram of an interface for manually setting control parameters in the embodiment of the present application;

Fig. 8 is a schematic diagram of an exemplary structure of an apparatus for controlling a smart home device in an embodiment of the present application.

Detailed ways

The terms used in the embodiments of the present application are only used to explain specific embodiments of the present application, and are not intended to limit the present application.

Taking the system architecture or scenario shown in Figure 1 as an example, the mobile terminal 002 can be a smart phone, tablet computer, etc., and accesses the network 005 through long-distance wireless communication methods such as 2G, 3G, 4G, 5G, etc. to obtain media resources, such as Multimedia files can be found from the network 005. The servers in the network 005 store various media resources, such as pictures, video programs and so on. The smart home controller 003 is installed in the home, and the household devices 004 in the home are all connected to the smart home controller 003 . Household device 004 is a device directly experienced by the user, such as lamps, smart TVs, curtains, smart water heaters, and the like. The mobile terminal 002 is connected to the smart home controller 003 through the network 005 or short-distance communication methods such as zigbee, infrared, Bluetooth, NFC, ultra-wideband (UWB), WIFI, etc., and the smart home APP is installed on the mobile terminal 002. The user 001 sends control parameters to the smart home controller 003 through the network 005 by operating the smart home APP on the mobile terminal 002, and the smart home controller 003 receives the control parameters from the mobile terminal 002, and sends a corresponding response to the control home device 004 according to the control parameters. control instructions to control the state change of the household device 004.

As shown in FIG. 2 , the mobile terminal 002 includes: a processing module 201 , a camera 202 , a communication module 203 and a storage module 204 . There can be multiple or multiple communication modules 203, such as Bluetooth, 5G/4G/3G/2G modules, wifi, NFC, etc. The mobile terminal 002 connects to the network 005 through the communication module 203, and establishes a connection with the smart home controller 003. After the camera 202 is turned on, it can capture images of the physical environment including the household device 004 . The storage module 204 may store a mapping relationship between multimedia feature information and control parameters of household devices.

The processing module 201 can be connected to the network through a communication unit, search for multimedia file resources from the network, such as pictures, videos, audios, etc., or directly call the local storage 204 to read pictures, videos, and audios stored in the storage 204. The processing module 201 can also be used to analyze the selected pictures/videos to obtain multimedia features, and determine the household devices to be controlled and their control parameters according to the multimedia features. Or further, the processing module 201 may associate the household equipment and its control parameters with the selected picture/video according to the determined household equipment to be controlled and its control parameters (that is, generate a smart home scene). The processing module 201 can also be used to perform augmented reality (Augmented Reality, referred to as AR) modeling and enhanced display, combined with the physical environment image in the lens of the camera 20 and the laser radar scanner, to identify the household equipment in the physical environment, and Positional enhancements display virtual objects. User 001 does not need to try to locate the location of the household equipment one by one from the list, and does not need to know the serial number of the household equipment, so he can directly select the virtual household equipment according to the location of the physical environment, and then select the household equipment. The processing module 201 can also operate the household device corresponding to the physical location by operating the virtual object according to the AR model extracted and established. During the AR imaging process, the mobile terminal 002 can move to include household devices that were not originally within the shooting range. During the movement of the mobile terminal 002, even if a household device is out of the imaging range of the camera, as long as the AR recognition has been completed (that is, the virtual object of the household device is displayed at the corresponding position), the household device will still be included in the household device to be controlled. list. The list to be controlled includes all household devices identified by AR this time, and the control parameters are to be determined. The user 001 can also operate the virtual object in the AR image preview interface through the mobile terminal 002, such as deleting the virtual object. After the virtual object is deleted on the preview interface, the home device corresponding to the virtual object will be removed from the list to be controlled. Even if the camera captures the home device again later, the virtual object will not be regenerated. In this way, the user 001 can eliminate the household devices that he does not want to control, and only keep the household devices that he wants to control.

Wherein, the AR modeling is used to identify the distribution locations of the household devices 004 . During AR modeling, the sensors on the mobile terminal 002, such as image sensors and laser radar scanners, can scan the entire home environment to identify the locations of all household devices 004. During the AR identification process, the smart home controller 003 communicates with the household equipment 004 to obtain the logical distribution diagram of the household equipment 004, and establishes the distribution model of the household equipment 004 by combining the logical distribution diagram and physical positioning. For the specific technology of AR modeling, please refer to US20210097768A1. After AR modeling, the smart home controller 003 can locate the location of the home device 004 according to the image of the camera of the mobile terminal 002 and other sensors of the mobile terminal 002 such as a lidar scanner, thereby identifying the home device 004. During the communication process between the smart home controller 003 and each home device 004, combined with the distribution location, the smart home controller 003 can know the home device 004 corresponding to the virtual object, for example, establish an association between the virtual object and the ID of the home device 004 , to realize the operation of virtual objects, that is, the operation of household equipment. When used for the first time, the smart home controller 003 can communicate with the home devices 004 one by one, so as to establish a logical distribution map of the home devices 004 through communication. When new home equipment is added, AR modeling can be performed again to add the new home equipment.

An example of the method implemented by the mobile terminal 002 for controlling smart home devices is shown in Figure 3, including:

Step 31: The mobile terminal 002 acquires multimedia files. For example, the user 001 operates the mobile terminal 002 to give an instruction to select a multimedia file, and the mobile terminal 002 selects a picture, video or audio file from the memory 204 according to the operation instruction of the user 001, or obtains the corresponding multimedia file from the network 005.

Step 32: The mobile terminal 002 determines the home device to be controlled and the control parameters applicable to the multimedia file through the virtual home device in the AR image, wherein the virtual home device corresponds to the home device to be controlled, and the control parameter is used for controlling the household device to be controlled.

For example, after the mobile terminal 002 obtains the multimedia file, the user 001 can operate the mobile terminal 002 to turn on the camera to obtain the environmental image including the household device 004 to be controlled, and activate the AR function of the mobile terminal 002, combining the environmental image and the sensor captured by sensors such as lidar data, and present an AR image containing virtual objects and physical environment elements, such as walls, on the display screen of the mobile terminal 002. Wherein, the virtual object is a virtual household device obtained by identifying the household device 004 to be controlled. The virtual home device is consistent with the actual physical location of the corresponding home device in the lens of the camera 202 , in one-to-one correspondence, and there is no need for the user 001 to manually add the home device 004 to be controlled. Since the virtual household equipment in the AR image corresponds to the household equipment to be controlled one by one, the user 001 can determine the control parameters of the household equipment 004 to be controlled based on the virtual household equipment in the AR image, and can further control the household equipment to be controlled. According to the effect under the control of the determined control parameters, it is presented in the AR image interface for user 001 to preview.

The way for the mobile terminal 002 to determine the household equipment to be controlled and its control parameters can be the way that the user 001 uses the mobile terminal 002 to manually select the household equipment and its control parameters; or it can be: the user 001 uses the mobile terminal 002 to form an AR Determine the household equipment to be controlled by means of a method, and obtain some multimedia features by analyzing the multimedia file, and then determine the household equipment to be controlled and its control parameters by the multimedia features according to the preset mapping relationship; or it can be: firstly, the mobile terminal 002 Execute the operation instructions of user 001, automatically determine the household equipment 004 to be controlled and its control parameters according to the multimedia features and AR images, and then user 001 uses the mobile terminal 002 to make manual adjustments.

After determining the control parameter sets of the household devices 004 corresponding to all the features, the mobile terminal 002 may further save the determined control parameter sets. The set of control parameters can also be regarded as the media scene of the household device 004 determined according to the multimedia file. Compared with the prior art, the method provided in this embodiment needs to select household devices one by one and set control parameters, resulting in complicated steps and poor user experience. Through AR images, the camera is combined with AR processing to identify and locate household devices 004, And according to the characteristics of the multimedia file, the control parameters of the selected home equipment are automatically generated, or the home equipment is selected from the AR virtual object and its control parameters are determined according to the selection instruction of the user 001, or the home equipment and its parameters are determined manually and automatically. Controlling parameters simplifies the creation steps of smart home scenes, enhances usability, and improves user experience.

After the mobile terminal 002 determines the control parameters of the home equipment 004, it can send the control parameters to the smart home controller 003, and the smart home controller 003 sends corresponding control instructions to the home equipment 004 according to the received control parameters, so as to realize the control of the home equipment. Control of device 004.

Another example of a method for controlling smart home devices is shown in Figures 4(a) to 4(e). In this embodiment, the mobile terminal 002 automatically generates a scene according to a picture. The picture can be freely selected by the user, such as a picture taken in real time or a picture taken before (that is, an existing picture), or a picture obtained by searching the Internet. The smart home APP in the mobile terminal 002 can provide not only traditional custom scene creation, but also multimedia scenes, so as to determine the control parameters of home equipment according to selected multimedia files such as pictures, that is, generate scenes. Moreover, the smart home APP is integrated with AR functions. When the user uses the mobile terminal 002 to select household equipment, the household equipment can be identified through AR, so that the household equipment can be seen and controlled in the mobile terminal 002 . In the traditional custom solution, many home devices are provided in the form of a list. If the user wants to select a smart home device in a specific location, he needs to first filter the type, and then try to filter one by one in the same type of device, and then choose Home equipment, the selection of each home equipment has to do this, it is very complicated. In this embodiment, the terminal 002 uses the above AR function to perform AR processing on the scanned environment image of the home device to be controlled to obtain a controllable virtual home device, which avoids cumbersome screening operations, simplifies user operation steps, and improves user experience .

When implementing the method for controlling smart home devices, the user 001 opens the smart home APP in the mobile terminal 002, clicks the "smart" button below, and selects "Use media files to create Scene" button, the interface displayed by the mobile terminal 002 is shown in Figure 4(a). The user 001 clicks the "select media file" button in the interface, and the display screen of the mobile terminal 002 can jump to the interface shown in FIG. 4(b). By default, the interface displays the "Local Resources" page for locally stored media files. The interface also provides a "network resource" button to enable user 001 to search for pictures from the network 005, or to jump to the gallery/album interface of the mobile terminal 002 through the "local resource" button to guide user 001 to store images from the local memory 204. Select an image from Image. After the "network resources" button is clicked by the user, the mobile terminal 002 displays the search page as shown in Figure 4(c), and after the user 001 clicks the "search" button in the interface, the mobile terminal 002 displays all the network resource search results ( not shown in the figure). User 001 can select a picture by clicking the file icon in the search results. After the smart home APP receives the operation instruction from user 001, it obtains the link of the picture selected by user 001 in the background, and obtains the picture from the network through the link. Moreover, in the interface shown in FIG. 4( b ), the picture selected by the user 001 is highlighted. After the user 001 clicks the "Finish" button on the upper right of the interface, the mobile terminal 002 turns on the camera and enters the AR imaging interface. After the environmental image of the home device 004 is processed by AR, the AR image shown in Figure 4(d) is displayed on the image preview interface. image. The AR image includes physical environment elements roof 400 and wall 401 , and also includes two kinds of virtual household equipment: virtual downlight 41 and virtual spotlight 42 . Both the virtual downlight 41 and the virtual spotlight 42 are displayed as virtual objects at the physical location of the household equipment, that is, the physical downlight is located in the middle area of the roof 400, and the virtual downlight 41 displayed in the AR image is also located in the middle area of the roof 400. The middle area of the roof 400 ; the physical spotlights are located at the left and right edges of the roof 400 , and the virtual spotlights 42 displayed in the AR image are also located at the left and right edges of the roof 400 . The user 001 can operate the home appliance at the same physical location as the virtual object by operating the virtual object. For example, the user 001 adjusts the brightness of the physical downlight by adjusting the brightness of the virtual downlight 41 in the AR image.

The AR image also includes an “automatically generate scene” button 43 and a preview button 44 . After the user 001 clicks the "automatically generate scene" button 43, the mobile terminal 002 analyzes the picture selected by the user 001, and obtains at least one kind of picture characteristic information such as brightness, contrast, color information (such as color type, color percentage, etc.), label information wait. Or, the mobile terminal 002 may also start to analyze the picture after the user 001 selects the picture, or after the user 001 selects the picture and clicks the "finish" button in the above-mentioned FIG. 4(c).

The mobile terminal 002 can pre-set and store the mapping relationship between the multimedia feature information and the control parameters of the home appliance 004. After the mobile terminal 002 parses and obtains the picture feature information, it can determine the home appliance 004 to be controlled and its home appliance 004 according to the mapping relationship. Control parameters, the household device 004 to be controlled corresponds to the virtual household device. Part of the mapping relationship is shown in Table 1.

Table I

Table 1 lists some approximate control parameters for controlling household devices based on image features. During actual implementation, the mapping relationship stored in the memory 204 may include specific numerical mapping functions such as numerical functions of picture brightness and downlight brightness, numerical functions of contrast and frequency, and the like. In addition, a control parameter of a household device may be affected by various factors. When multiple factors affect the control parameter of the household device, the processor 201 determines the final control parameter of the household device according to a predetermined rule.

After the user 001 clicks the "automatically generate scene" button 43, if the mobile terminal 002 has analyzed the feature information of the picture, the mobile terminal 002 determines the household equipment to be controlled and the corresponding control parameters according to the above mapping relationship. If the picture feature information is not analyzed at this time, the picture feature information can be parsed first, and then the control parameters of the household device 004 can be determined according to the picture feature information and the above-mentioned mapping relationship table.

Optionally, after determining the control parameters of the home appliance 004 to be controlled, the mobile terminal 002 displays the result of "automatically generated scene" as "automatically generated scene" 45 as shown in FIG. 4(e).

The mobile terminal 002 can also provide a preview function. For example, the user 001 can click the preview button 44 to preview the scene effect produced by the home device 004 under the control of the control parameters. For example, after the mobile terminal 002 receives the operation command "preview" from the user 001, it controls each virtual home device in the AR image to simulate the corresponding working state according to the control parameters of the generated scene, and displays it.

The method for controlling smart home devices in this embodiment is different from the traditional smart home scene editing mode. The user 001 only needs to select a picture and use the camera of the mobile terminal 002 to select the home devices in the target area without checking the home devices one by one. The selection solves the tedious problem of locating and selecting household equipment, realizes the fully automatic operation of selecting household equipment and determining control parameters, and improves user experience. Moreover, the method of automatically generating scenes based on pictures on the mobile terminal provides a new method for determining scenes and enriches the scene functions of smart home devices.

An example of another method for controlling smart home devices is shown in Figures 5(a) to 5(c). In this embodiment, the multimedia file is a video file. Since video files are usually large, video files are usually network resources. For example, when mobile terminal 002 is connected to network 005, user 001 uses the method shown in Figure 4(b) to click search on the search interface of the smart home APP, and mobile terminal 002 executes user 001's operation instruction "search for video files" and sends a message to network 005. Send a request for searching video files, and display the search results for user 001 to choose. After user 001 selects a video file in the search results, the smart home APP obtains the network link of the video file, such as a URL address, and associates with the selected video file through the link of the video file. The mobile terminal 002 displays an interface as shown in FIG. 5( a ) to create an AR image including virtual household devices. In this embodiment, the creation process of the AR image including the virtual home device is the same as that in the previous embodiment. Similarly, the virtual household equipment in the AR image corresponds to the household equipment 004 in the lens of the camera 202 one by one, and the positions are the same. User 001 changes the position of the mobile terminal 002, scans the home devices to be controlled and performs AR processing, and creates a virtual object in the environment image interface shown in Figure 5(a), that is, adds the home devices to be controlled to the list to be controlled. User 001 clicks the "automatically generate scenes" button 43 in Figure 5(a), and the smart home APP starts to analyze along the playing sequence of the video file 500, and determines the scene in the list to be controlled according to the mapping relationship according to the features obtained through analysis. Control parameters for home appliances. Optionally, during the parsing process, the interface of the mobile terminal 002 may further display information such as "analyzing" and "parsing completed", so that the user 001 can learn about the parsing progress of the video file 500.

When the mobile terminal 002 parses the video file 500, in addition to parsing the video frame, it can also parse the audio therein. Each video frame is a picture in a picture, that is, a frame of image. Therefore, the analysis of the video frame can refer to the picture analysis method in the previous embodiment. When the mobile terminal 002 parses the audio in the video file, the audio feature information may include tag information such as music style, line keyword information/lyric keyword information, tempo information, and the like.

After the mobile terminal 002 parses and obtains the image feature information and audio feature information, it determines the control parameters of the home device corresponding to the virtual home device in the AR image according to the mapping relationship between these multimedia feature information and the control parameters of the home device 004 . Wherein, the mapping relationship between the multimedia feature information and the control parameters of the home device 004 includes the mapping relationship between the picture feature information and the control parameters of the home device 004, and the mapping relationship between the audio feature information and the control parameters of the home device 004. Mapping relations. The mapping relationship between the picture feature information and the control parameters of the household device 004 is similar to the previous embodiment, and reference can be made to Table 1 above. Since a video file contains many video frames, when determining the control parameters, the control parameters of the home device at each moment in the video file playback process can be determined according to the characteristics of all video frames, that is, for the video playback device in the home device 004 For example, during the whole process of playing video files on a TV, the smart home controller 003 controls the home devices such as curtains, lamps, and air conditioners in the home devices 004 to adjust the environment according to the video frames played at different times. In this case, after the user 001 clicks the "automatically generate scene" button 43, the mobile terminal 002 generates a control parameter set for the video file 500 of the household equipment. The set of control parameters includes the control parameters of the home devices when different video frames are displayed during the playing of the video file 500 . Alternatively, the mobile terminal 002 can also determine the control parameters of the home device 004 according to the characteristics of the video frames in the video file 500 that meet the preset conditions, that is, to play the frame that meets the preset conditions during the playback of the video file 500 by the video playback device. When the video frame is displayed, the smart home controller 003 controls the home equipment used to adjust the environment accordingly. For example, in all video frames of a movie, several video frames meet the conditions for generating scenes. For example, for the video frame and/or audio playing thunder, determine the corresponding control parameters of the home device according to its characteristics, such as brightening the light a little, while ignoring the characteristics of other video frames or audio. The mobile terminal 002 can determine the control parameters of the home equipment corresponding to each video frame after parsing the video frame. At this time, the progress displayed by the progress bar 56 is the progress of parsing plus determining the control parameters. When the mobile terminal 002 only determines the control parameters of the corresponding household equipment for the video frames that meet the preset conditions, the dotted line parts 561 and 562 on the progress bar 56 are the progress of analyzing the video frames that meet the preset conditions and determining the control parameters. That is, the video frames in the dotted line parts 561 and 562 are configured with control parameters for home devices, and the video frames in the remaining blank parts are not configured with control parameters for home devices.

When determining the control parameters of the household equipment for a certain video file, the type of the household equipment may be further considered. For example, home appliances can be classified into real-time response types and lag response types. Real-time response home devices can respond to control commands in real time to quickly achieve desired effects, such as lighting effects, music playback from speakers, and so on. Therefore, for real-time response home devices, when a video playback device plays a certain video frame of a video file, the smart home device can send corresponding control instructions to this type of home device according to the control parameters, for example, to control the color of the light , flashing, brightness, color temperature; or control playing specific music, etc.

Lag-response type household devices need a certain response time after receiving the control command to achieve the expected effect of the control parameters, such as the opening/closing of the curtains, the sea breeze simulation of the air conditioner, the smell of the fragrance of the speaker, etc. Wherein, it takes 5 to 10 seconds for the curtain to fully open to fully close. When the video file is played, it is displayed frame by frame according to the time sequence. Assuming that the control parameter corresponding to a certain video frame is to close the curtain, then when the video frame is displayed, after the smart home controller 003 sends a close command to the curtain, the curtain will be closed within 5 to 10 minutes. Seconds to fully close. It can be understood that at this time, the video frame has been displayed for 5 to 10 seconds, and the currently displayed video frame is the hundreds or even thousands of video frames after the video frame. Therefore, in order to make the video playback device play the video file At this time, the smart home controller 003 can control the home equipment to achieve the expected environmental effects, such as the brightness and darkness of the environment, the aromatherapy atmosphere, and the simulated sea breeze. The playback timing inserts it at a certain moment before the playback, so that the smart home controller 003 can send control instructions to the lagging response type home devices in advance according to the control parameters, so that before the video playback device plays a certain video frame, The home equipment of the control lagging response type changes state, and when the video playback device plays the video frame, the expected state of the control parameter is reached. For example, the video frame playback sequence of the video file 500 shown in FIG. 5( b ) is: video frame 501 , . . . , video frame 502 , . . . , video frame 503 . Wherein, the color of most of the picture of the video frame 503 is blackish, correspondingly determine a control parameter for the curtain: completely close the curtain, so as to make it easier to see the blackish video frame 503 clearly. But if when the video playback device plays the video frame 503, the smart home controller 003 sends a control command to completely close the curtain according to the control parameter, then when the curtain is completely closed, the video playback device has played the video frame 503 and may be playing Video frame 504. Since it takes 5 seconds for the curtain to be fully opened to fully closed, and the playing time interval between the video frame 502 and the video frame 503 is 5 seconds, the mobile terminal 002 can determine the control parameters of the home equipment corresponding to all the video frames. Afterwards, or after determining the control parameters of the home equipment corresponding to the video frame 503 and previous video frames, insert the control parameters of the curtains corresponding to the video frame 503 into the control parameters corresponding to the video frame 502, or insert the corresponding control parameters of the video frame 503 into The execution time of the control parameter of the curtain is modified to the playing time of the video frame 502. In this way, when the video playback device plays the video file 500, if the video frame 502 is played, the smart home controller 003 generates a control command for the home device according to the control parameters corresponding to the video frame 502, and at the same time it also generates a control command for the curtain inserted therein. Parameters generate and issue a control command “completely close” to the curtains, or simultaneously generate and transmit a control command “completely close” according to the control parameters of the curtains corresponding to the video frame 503 at the playback moment. The curtain receives and executes the "completely close" instruction when the video playback device plays the video frame 502. After 5 seconds, the curtain is completely closed, and the video playback device also plays the video frame 503 just in time, thereby achieving the expected effect of the control parameters.

For the mapping relationship between the audio feature information and the control parameters of the household device 004, please refer to Table 2.

Table II

According to the mapping table, the mobile terminal 002 determines the control parameters of the household equipment for different picture characteristic information and audio characteristic information. When the mobile terminal 002 determines that there are multiple control parameters of the household equipment at a certain moment and a conflict occurs, a control parameter is determined according to the multiple control parameters according to a preset rule, for example, a control parameter is selected from among them. In this way, the mobile terminal 002 finally determines a control parameter set according to the video file 2, and the control parameters in the control parameter set are in one-to-one correspondence with each video frame or the playing time of the video frame. Optionally, the mobile terminal 002 may further send the set of control parameters to the smart home controller 003 . When the user 001 uses the video playback device to play the video file 500, the smart home controller 003 is triggered to generate corresponding control instructions according to the control parameters in the control parameter set according to the playback sequence and send them to the corresponding home equipment, so that the user can watch the video file. In the process of 500, enjoy the corresponding environment or atmosphere created by the home equipment when playing different video frames, and watch the video file 500 with the best feeling.

Optionally, after the mobile terminal 002 determines the set of control parameters, it can further display an interface as shown in FIG. The file 500 corresponds to the set of control parameters of the household equipment.

Optionally, the mobile terminal 002 may further provide a preview function. After the user 001 clicks the "preview" button 44 in Figure 5(c), the video playback progress bar 57 is consistent with the playback timing of the video file 500, and the playback progress of the video file 500 is displayed. control effect. Assuming that the control parameters are determined by some video frames that meet the preset conditions, when the video file 500 is played to the dotted progress bars 571 and 572, the state of the virtual home device starts to change accordingly according to the control parameters. Assuming that the mobile terminal 002 has determined a set of control parameters according to the video frames played throughout the video file 500, there are no dotted line progress bars 571 and 572 on the video playback progress bar 57, and during the full playback of the video file 500, the state of the virtual home device can be The whole process is adjusted according to the set of control parameters. Optionally, when previewing, the user can also select a time point for previewing. Optionally, the preview interface may also include a media window 58 for playing video images corresponding to the progress of the video playback progress bar 57 .

The method for controlling household equipment provided in this embodiment determines the control parameters of household equipment based on video, which can be used in personalized movie viewing scenarios. According to the characteristics of each movie, the control parameters of the corresponding home equipment can be determined, which is equivalent to each power supply corresponding to several movie viewing scenes, which effectively improves the user's movie watching experience. Furthermore, for curtains and other lagging response household devices with a long execution time, by advancing their control parameters, their control instructions can be sent before the video frame is played, so that when the video frame is played, the ideal viewing angle can be achieved. shadow environment. Further optionally, only when the video frame satisfies the set conditions will the automatic generation of the scene be triggered, that is, the generation of the viewing scene will be triggered only when the corresponding home scene needs to be provided, which reduces the data processing amount of the mobile terminal and improves In order to determine the efficiency of the control parameters, the user experience is improved.

In the method for controlling home appliances provided by this embodiment, the user manually determines the control parameters of the smart home, that is, the scene, using a mobile terminal, as shown in FIG. 6( a ) to FIG. 6( d ). After the user 001 selects a video file, the mobile terminal 002 displays an AR identification interface. The interface includes a multimedia playing window 58 and a playing progress bar 57 . When the user 001 manually determines the control parameters of the home device, he can select a time node on the playback progress bar 57, and then change the position of the mobile terminal 002 so that the camera 202 can capture the home device 004 to be controlled. After the mobile terminal 002 captures the household device 004, a corresponding virtual object is generated. As shown in Figure 6(a), after the user selects a virtual home device such as a downlight, controllable options such as "light effect", "color temperature", "brightness" and "color" pop up on the interface. After user 001 selects the "light effect" option, the interface pops up secondary controllable options such as the type of light effect, such as "blinking", "jumping", "waves" and "breathing" as shown in Figure 6(b). Assuming that the user 001 selects the "jump" option, that is, the control parameters of the downlight in the home device at the playback moment of the video file are determined.

Optionally, when the user manually determines the control parameters, the mobile terminal 002 may support the creation of a group for the convenience of operating multiple household devices of the same type, so as to perform group operations on the household devices. As shown in Figure 6(c), assuming that user 001 has the same requirements for the six downlights 61, the six downlights 61 can be selected to create group 1, so as to uniformly control the six downlights 61, avoiding one-by-one control . The group of household devices can be created only once. For example, after creating group 1 at a certain time node, other time nodes can also select this group 1 to determine the control parameters of the downlight. After creating a group, when user 001 selects a home device, he can select a group or a single home device to determine the control parameters through different operations, for example, click to select a group to determine the control parameters of the group of home devices, double-click to select a single home device to determine the Control parameters for individual home devices.

Optionally, this embodiment can be used as a supplementary solution to the previous two embodiments. After the mobile terminal automatically generates a scene, that is, determines the control parameters of the home device, and before sending it to the smart home controller, the user can manually select the home device, and manually Adjust the control effect, that is, manually determine the control parameters of the household equipment. When the manual and automatic control parameters conflict, the manually determined control parameters can overwrite the original automatic control parameters, and can also retain the original automatic control parameters. For example, after the mobile terminal 002 determines the control parameters of multiple household devices for multiple time nodes of a certain video file, the user creates or modifies the control parameters of the household devices.

If the control parameters manually determined by the user conflict with the control parameters automatically determined by the mobile terminal, the mobile terminal prompts the user whether to overwrite the previously automatically determined control parameters as shown in Figure 6(d). The user can choose to overwrite the control parameters automatically determined by the mobile terminal and replace them with manually determined light effects, or keep the control parameters automatically determined by the mobile terminal 002 and give up the manually determined light effects.

The method for controlling home appliances provided in this embodiment supports users to manually select home appliances through AR images, and determine control parameters of the home appliances to further realize customization. In addition, the control parameters of household equipment are determined by combining manual and automatic methods. On the basis of automatically generated scenes, users can adjust them again, which enriches the way of scene generation and improves the user experience.

On the basis of the foregoing embodiments, another example method for controlling home devices may further include: the mobile terminal receives a user's sharing operation instruction, executes the sharing operation instruction, generates a configuration file according to the determined control parameters of the home device, and upload to the network. Wherein, the configuration file may include the category of the household equipment and the corresponding control parameters, or may further include the corresponding multimedia file or its identification such as name and ID. As shown in Figure 7, after the mobile terminal prompts "Scene has been automatically generated", the user can click the menu in the upper left corner, and then the mobile terminal pops up the operation options "Automatic scene", "Save", "Download", "Upload" and "Share "wait. After the user clicks "Share", the mobile terminal will generate a corresponding configuration file, and further, pop-up sharing methods such as displaying network platform theme store icons, instant messaging software such as QQ, WeChat icons, etc. for the user to choose, when the user clicks one of them After the icon is displayed, the mobile terminal receives the user's instruction and performs the corresponding upload operation.

Corresponding to the above uploading operation, the mobile terminal can also download the configuration file of the smart home scene from the network platform for controlling the home equipment. The smart home APP can identify the configuration file of the smart home scene, and after reading the parameters of the configuration file, generate the control parameters of the home equipment corresponding to the configuration file.

When other users receive the configuration file through the instant messaging software, the smart home APP in the mobile terminal they use can parse the configuration file. After parsing, the mobile terminal can receive the user's operation instruction to first select the application environment, and then configure the parameters of the configuration file to the household devices in the application environment in the selected application environment. The choice of application environment can be established by scanning the camera of the mobile terminal and identifying home devices through AR, or directly selecting from the stored AR environment such as the living room/bedroom/study room. The stored AR environment contains several home devices. It is pre-established by the mobile terminal according to the user's operation instructions.

When the mobile terminal configures the parameters of the configuration file to the household equipment, it can be configured according to the category of the household equipment, for example, the same type of household equipment applies the control parameters of the same type of household equipment. In the actual environment, household devices that cannot correspond to the category of the configuration file can be ignored or skipped, and only household devices of the same category that can be configured can be configured.

The method for controlling home appliances provided by this embodiment uploads the generated scene, that is, the control parameters, to the platform through the sharing operation of the mobile terminal for others to download or directly share with friends. After receiving it, others can directly apply the configuration file to generate the corresponding scene Or determine the corresponding control parameters, which improves the convenience of the application of the control parameters.

An example of an apparatus for controlling smart home devices is shown in FIG. 8 , including: an acquiring module 801 and a determining module 802 . The device may be an application program applied to a mobile terminal, and is used to implement the methods in the above embodiments. The acquisition module 801 is used to acquire multimedia files such as pictures, videos, etc.; the determination module 802 is used to determine the home equipment to be controlled through the virtual home equipment in the AR image, and determine the control parameters applicable to the multimedia file, wherein the The virtual home device corresponds to the home device to be controlled (for example, selecting a virtual home device is equivalent to selecting a physical home device), and the control parameter is used to control the home device to be controlled. Wherein, the above-mentioned obtaining module 801 can be used to obtain pictures, video or audio files from local or network. The determining module 802 may be configured to scan the home device to be controlled and its environment, and generate an AR image including the virtual home device. As a variant of this embodiment, the device for controlling smart home devices may further include: a first receiving module and a rejecting module; the first receiving module is used to generate the virtual home device containing the above-mentioned After the AR image, a culling command input by the user is received; the culling module is used to cull the object that is culled by the culling command in the virtual home device.

In the above device embodiments, the control parameters can be determined automatically or manually. When the control parameters are automatically determined, the determination module 802 can be used to determine the control parameters applicable to the above-mentioned multimedia files according to the characteristics of the above-mentioned multimedia files and the preset mapping relationship, wherein the mapping relationship is the characteristics of the above-mentioned multimedia files and the above-mentioned control parameters. For the mapping relationship between parameters, refer to the description in the foregoing method embodiments.

When the above-mentioned multimedia file is a video file, the above-mentioned determination module 802 can be used to determine the video file suitable for the above-mentioned multimedia file according to the video frame feature and/or audio feature of the above-mentioned multimedia file and the above-mentioned preset mapping relationship, according to the playing sequence of the above-mentioned multimedia file. A set of control parameters for different video frames of the file; or, the determining module 802 is configured to, when the video frame and/or audio of the multimedia file meet a preset condition, according to the video frame feature and/or audio feature of the multimedia file and the aforementioned preset The set mapping relationship determines the control parameters applicable to the video frame and/or audio meeting the preset condition. When the determined control parameters include control parameters for controlling hysteresis-response household appliances, the determination module 802 may be configured to insert the first control parameter in the set of control parameters into the execution time of the second control parameter; wherein, The first control parameter is a control parameter corresponding to the first video frame in the multimedia file, and the second control parameter is a control parameter corresponding to the second video frame in the multimedia file; the control parameter corresponding to the first video frame is a lag response The control parameter of the type of household equipment, the playing time difference between the first video frame and the second video frame is the time period for the lagging response type of household equipment to respond to the control instruction, and the control instruction is the execution instruction of the first control parameter.

When the control parameters are manually determined, the determination module 802 may include: a receiving submodule and a determination submodule. The receiving submodule is used to receive the time node input by the user, and to receive the first operation instruction input by the user by clicking on the virtual home device, and to receive the second operation instruction input by the user by selecting a control option; The first operation instruction determines the household device to be controlled, and the control parameters applicable to the multimedia file at the time node are determined according to the second operation instruction.

On the basis of the automatic determination of the control parameters by the above-mentioned device, the device for controlling the smart home equipment may further include: a second receiving module to be combined with the manual determination of the control parameters. The second receiving module is used to receive the time node input by the user after the determination module 802 determines the control parameters applicable to the multimedia file according to the characteristics of the multimedia file and the preset mapping relationship, and receive the time node input by the user, and the user clicks on the virtual home. The first operation instruction input by the device, and the second operation instruction input by the user by selecting a control option; the determination module is used to determine the above-mentioned household equipment to be controlled according to the first operation instruction, and determine the applicable home device according to the second operation instruction. The control parameters of the above multimedia file at the time node.

As another variant of the above-mentioned device embodiment, the device for controlling smart home devices may further include: a third receiving module and a group creation module; the third receiving module is used to Before determining the household equipment to be controlled, a third operation instruction input by the user is received; the group creation module is used to create a group for the virtual household equipment selected by the user according to the third operation instruction. In some possible implementation manners, the determination module 802 is configured to determine, according to the first operation instruction, some or all of the home devices in the group as the home devices to be controlled.

As another variation of the above device embodiment, the device for controlling smart home devices may further include: a first display module. The first display module is used for displaying a prompt of conflicting control parameters and whether to choose to overwrite after the determining module 802 determines the control parameters applicable to the multimedia file at the above time node according to the second operation instruction.

As another variant of the above device embodiment, the device for controlling smart home equipment may further include: a fourth receiving module and a second display module; After the home device determines the home device to be controlled and the control parameters applicable to the multimedia file, it receives a preview instruction input by the user; the second display module is used to display the status of the virtual home device under the control of the control parameters .

As another modification of the above device embodiment, the device for controlling smart home equipment may further include: a fifth receiving module, a generating module, a third display module, a sixth receiving module, and an uploading module; the fifth receiving module uses After the determining module determines the household equipment to be controlled through the virtual household equipment in the AR image, and determines the control parameters applicable to the above multimedia files, it receives a sharing instruction input by the user; the generating module is used to generate the configuration according to the sharing instruction file; the third display module is used to display sharing options; the sixth receiving module is used to receive a sharing channel instruction input by the user by selecting the sharing option; the uploading module is used to upload the shared channel according to the sharing channel instruction configuration file.

It can be understood that some or all of the steps or operations in the foregoing embodiments are only examples, and other operations or modifications of various operations may also be performed in the embodiment of the present application. In addition, various steps may be performed in different orders presented in the above embodiments, and it may not be necessary to perform all operations in the above embodiments.

An embodiment of the present application also provides an electronic device, including: one or more processors; memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one One or more computer programs include instructions, and when the instructions are executed by the device, the device executes the methods shown in the above-mentioned embodiments.

The embodiment of the present application also provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is run on a computer, the computer executes the method provided in the embodiment of the present application.

The embodiment of the present application further provides a computer program product, the computer program product includes a computer program, and when running on a computer, causes the computer to execute the method provided in the embodiment of the present application.

In the embodiments of the present application, "multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three kinds of relationships, for example, A and/or B may indicate that A exists alone, A and B exist simultaneously, or B exists alone. Among them, A and B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single, or Can be multiple.

Those of ordinary skill in the art can appreciate that each unit and algorithm steps described in the embodiments disclosed herein can be realized by a combination of electronic hardware, computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In several embodiments provided in this application, if any function is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), magnetic disk or optical disc, etc. A medium on which program code can be stored.

The foregoing is only a specific implementation of the present application. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should be covered by the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

A method for controlling smart home equipment, characterized in that it is applied to a mobile terminal, comprising:

Get multimedia files;

Determine the household equipment to be controlled through the virtual household equipment in the augmented reality AR image, and determine the control parameters applicable to the multimedia file, wherein the virtual household equipment corresponds to the household equipment to be controlled, and the control parameters It is used to control the household device to be controlled.
The method according to claim 1, wherein obtaining a multimedia file comprises:

Get pictures, video or audio files from local or network.
The method according to claim 1 or 2, wherein determining the household equipment to be controlled through the virtual household equipment in the AR image includes:

Scanning the home device to be controlled and its environment to generate the AR image including the virtual home device.
The method according to claim 3, wherein after generating the AR image containing the virtual home device, further comprising:

Receiving a cull command input by the user, and cull objects in the virtual home device that are culled by the cull command.
The method according to any one of claims 1 to 4, wherein the determining the control parameters applicable to the multimedia file includes:

Determine the control parameters applicable to the multimedia file according to the characteristics of the multimedia file and a preset mapping relationship, wherein the mapping relationship is a mapping relationship between the characteristics of the multimedia file and the control parameters.
The method according to claim 5, wherein determining the control parameters applicable to the multimedia file according to the characteristics of the multimedia file and a preset mapping relationship comprises:

According to the video frame feature and/or audio feature of the multimedia file and the preset mapping relationship, according to the play sequence of the multimedia file, determine the set of control parameters applicable to different video frames of the multimedia file;

or

When the video frame and/or audio of the multimedia file meet the preset condition, according to the video frame feature and/or audio feature of the multimedia file and the preset mapping relationship, determine the Conditional video frame and/or audio control parameters.
The method according to claim 6, wherein, according to the playing sequence of the multimedia file, determining the set of control parameters applicable to different video frames of the multimedia file includes:

Inserting the first control parameter in the set of control parameters into the execution time of the second control parameter; wherein, the first control parameter is the control parameter corresponding to the first video frame in the multimedia file, and the second control parameter is The control parameter corresponding to the second video frame in the multimedia file; the control parameter corresponding to the first video frame is a control parameter of a lagging response type household device, and the playing time difference between the first video frame and the second video frame is the time period for the household device of the hysteresis response type to respond to the control instruction, and the control instruction is an execution instruction of the first control parameter.
The method according to any one of claims 1 to 4, wherein determining the household equipment to be controlled through the virtual household equipment in the AR image, and determining the control parameters applicable to the multimedia file include:

Receive the time node input by the user, receive the first operation instruction input by the user by clicking on the virtual home device, and receive the second operation instruction input by the user by selecting a control option, and determine the home to be controlled according to the first operation instruction A device for determining a control parameter applicable to the multimedia file at the time node according to the second operation instruction.
The method according to any one of claims 5 to 7, wherein, after determining the control parameters applicable to the multimedia file according to the characteristics of the multimedia file and the preset mapping relationship, further comprising:

Receive the time node input by the user, receive the first operation instruction input by the user by clicking on the virtual home device, and receive the second operation instruction input by the user by selecting a control option, and determine the home to be controlled according to the first operation instruction A device for determining a control parameter applicable to the multimedia file at the time node according to the second operation instruction.
The method according to claim 9, characterized in that before determining the household device to be controlled according to the first operation instruction, further comprising:

A third operation instruction input by the user is received, and a group is created for the virtual home device selected by the user according to the third operation instruction.
The method according to claim 10, wherein determining the household device to be controlled according to the first operation instruction comprises:

According to the first operation instruction, it is determined that part or all of the home appliances in the group are the home appliances to be controlled.
The method according to any one of claims 9 to 11, wherein after determining the control parameters applicable to the multimedia file at the time node according to the second operation instruction, further comprising:

Displays a prompt for conflicting control parameters and whether to choose to overwrite.
The method according to any one of claims 1 to 11, characterized in that, after determining the household equipment to be controlled through the virtual household equipment in the AR image, and determining the control parameters applicable to the multimedia file, further comprising:

receiving a preview instruction input by the user, and displaying the status of the virtual home device under the control of the control parameters.
The method according to any one of claims 1 to 12, characterized in that, after determining the household equipment to be controlled through the virtual household equipment in the AR image, and determining the control parameters applicable to the multimedia file, further comprising:

Receive a sharing command input by the user, generate a configuration file according to the sharing command, and display sharing options;

receiving a sharing channel instruction input by the user by selecting the sharing option, and uploading the configuration file according to the sharing channel in the sharing channel instruction.
A device for controlling smart home equipment, characterized in that it is applied to a mobile terminal, including:

The acquisition module is used to acquire multimedia files;

A determining module, configured to determine a home device to be controlled through a virtual home device in an AR image, and determine control parameters applicable to the multimedia file, wherein the virtual home device corresponds to the home device to be controlled, and The control parameters are used to control the household equipment to be controlled.
The device according to claim 15, wherein the acquiring module is configured to: acquire pictures, video or audio files locally or from a network.
The device according to claim 15 or 16, wherein the determining module is configured to: scan the home appliance to be controlled and its environment, and generate the AR image including the virtual home appliance.
The device according to claim 17, further comprising:

A first receiving module, configured to receive a culling instruction input by a user after the determining module generates the AR image containing the virtual home device;

The culling module is used for culling the objects culled by the culling instruction in the virtual home device.
The device according to any one of claims 15 to 18, wherein the determining module is configured to: determine the control parameters applicable to the multimedia file according to the characteristics of the multimedia file and a preset mapping relationship, Wherein, the mapping relationship is a mapping relationship between the feature of the multimedia file and the control parameter.
The device according to claim 19, wherein the determination module is used for:

According to the video frame feature and/or audio feature of the multimedia file and the preset mapping relationship, according to the play sequence of the multimedia file, determine the set of control parameters applicable to different video frames of the multimedia file;

or

When the video frame and/or audio of the multimedia file meet the preset condition, according to the video frame feature and/or audio feature of the multimedia file and the preset mapping relationship, determine the Control parameters for pre-conditioned video frames and/or audio.
The device according to claim 20, wherein the determination module is used for:

Inserting the first control parameter in the set of control parameters into the execution time of the second control parameter; wherein, the first control parameter is the control parameter corresponding to the first video frame in the multimedia file, and the second control parameter is The control parameter corresponding to the second video frame in the multimedia file; the control parameter corresponding to the first video frame is a control parameter of a lagging response type household device, and the playing time difference between the first video frame and the second video frame is the time period for the household device of the hysteresis response type to respond to a control instruction, and the control instruction is an execution instruction of the first control parameter.
The device according to any one of claims 15 to 18, wherein the determining module comprises:

The receiving sub-module is used to receive the time node input by the user, and receive the first operation instruction input by the user by clicking on the virtual home device, and receive the second operation instruction input by the user by selecting a control option;

The determining submodule is configured to determine the household device to be controlled according to the first operation instruction, and determine the control parameters applicable to the multimedia file at the time node according to the second operation instruction.
The device according to any one of claims 19 to 21, further comprising:

The second receiving module is used to receive the time node input by the user after the determining module determines the control parameters applicable to the multimedia file according to the characteristics of the multimedia file and the preset mapping relationship, and receive the user's click receiving the first operation instruction input by the virtual home device, and receiving the second operation instruction input by the user by selecting a control option;

The determination module is configured to determine the household device to be controlled according to the first operation instruction, and determine the control parameters applicable to the multimedia file at the time node according to the second operation instruction.
The device according to claim 23, further comprising:

A third receiving module, configured to receive a third operating instruction input by a user before the determining module determines the household device to be controlled according to the first operating instruction;

A group creating module, configured to create a group for the virtual home device selected by the user according to the third operation instruction.
The device according to claim 24, wherein the determining module is used for:

According to the first operation instruction, it is determined that part or all of the home appliances in the group are the home appliances to be controlled.
The device according to any one of claims 23 to 25, further comprising:

The first display module is configured to, after the determination module determines the control parameters applicable to the multimedia file at the time node according to the second operation instruction, display a prompt indicating that the control parameters conflict and whether to choose to overwrite.
The device according to any one of claims 15 to 25, further comprising:

The fourth receiving module is configured to receive a preview instruction input by the user after the determination module determines the household equipment to be controlled through the virtual household equipment in the AR image and determines the control parameters applicable to the multimedia file;

The second display module is used for displaying the state of the virtual household device under the control of the control parameters.
The device according to any one of claims 15 to 26, further comprising:

The fifth receiving module is configured to receive a sharing instruction input by the user after the determining module determines the household equipment to be controlled through the virtual household equipment in the AR image and determines the control parameters applicable to the multimedia file;

A generating module, configured to generate a configuration file according to the sharing instruction;

a third display module, configured to display sharing options;

A sixth receiving module, configured to receive a sharing channel instruction input by the user by selecting the sharing option;

An upload module, configured to upload the configuration file according to the sharing path in the sharing path instruction.
A mobile terminal, characterized in that it includes: a processor, a memory, and a computer program, the memory stores the computer program, and when the processor runs the computer program, the mobile terminal executes claims 1 to 14 above any one of the methods described.
A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when it is run on a computer, the computer is made to execute the method according to any one of claims 1 to 14.
A chip system, characterized in that it executes the method described in any one of claims 1 to 14 above.