CN111314398A

CN111314398A - Equipment control method, network distribution method, system and equipment

Info

Publication number: CN111314398A
Application number: CN201811512113.4A
Authority: CN
Inventors: 郑思源; 姚海通; 林文彬
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2018-12-11
Filing date: 2018-12-11
Publication date: 2020-06-19

Abstract

The embodiment of the application provides a device control method, a network distribution method, a system and a device. The equipment control method comprises the following steps: displaying a scene image shot by a real scene, wherein the scene image contains an equipment image of at least one entity equipment; and responding to a control event triggered by a user on the scene image aiming at a device image, and executing the control operation of the entity device corresponding to the device image. According to the technical scheme provided by the embodiment of the application, a user can obtain a scene image through a place where the entity equipment is located through live-action shooting, and then can complete control over the corresponding entity equipment through a mode of operating on the scene image, so that the obtained effect is very visual and the use is convenient; especially, when a plurality of controllable entity devices are arranged in a place, the scene images can visually display the images of the entity devices, and a user can control the corresponding entity devices by performing corresponding operation on the images of the devices to be controlled, so that the control is visual and is not easy to make mistakes.

Description

Equipment control method, network distribution method, system and equipment

Technical Field

The present application relates to the field of computer technologies, and in particular, to an apparatus control method, a network distribution method, a system, and an apparatus.

Background

The Internet of Things (IoT) is a huge network formed by collecting information of objects or processes to be monitored, connected and interacted in real time through various information sensing devices and combining the information with the Internet. The purpose of the Internet of things is as follows: the connection of objects, objects and people and all objects and the network is realized, and the identification, management and control are convenient. Nowadays, devices accessing to the internet of things (internet of things devices for short) are more and more abundant and can be controlled through smart speakers or mobile phones and the like.

In the prior art, the control of the internet of things equipment is not intuitive, either through voice or through a switch on a mobile phone application.

Disclosure of Invention

The present application provides a device control method, a network distribution method, a system and a device that solve the above problems or at least partially solve the above problems.

In one embodiment of the present application, a device control method is provided. The method comprises the following steps:

displaying a scene image shot by a real scene, wherein the scene image contains an equipment image of at least one entity equipment;

and responding to a control event triggered by a user on the scene image aiming at a device image, and executing the control operation of the entity device corresponding to the device image.

In another embodiment of the present application, an internet of things system is provided. This thing networking device includes:

at least one physical device;

the client is used for displaying a scene image shot by a real scene, wherein the scene image contains an equipment image of at least one entity equipment; and responding to a control event triggered by a user on the scene image aiming at a device image, and executing the control operation of the entity device corresponding to the device image.

In yet another embodiment of the present application, a device control method is provided. The method comprises the following steps:

responding to click operation triggered by a user on a scene image, and acquiring equipment information of an equipment image selected by the user; the scene image is obtained through live-action shooting and contains an equipment image of at least one entity equipment;

monitoring control operation of a user for the equipment image when detecting that the entity equipment corresponding to the equipment information is in a networking state;

and after monitoring that the user finishes the control operation, executing the control operation on the entity equipment corresponding to the equipment image based on the operation content of the control operation.

In yet another embodiment of the present application, an internet of things system is provided. This thing networking system includes:

at least one physical device;

the client is used for responding to click operation triggered by a user on the scene image and acquiring equipment information of an equipment image selected by the user; the scene image is obtained through live-action shooting and contains an equipment image of at least one entity equipment; monitoring control operation of a user for the equipment image when detecting that the entity equipment corresponding to the equipment information is in a networking state; and after monitoring that the user finishes the control operation, executing the control operation on the entity equipment corresponding to the equipment image based on the operation content of the control operation.

on the scene image, a control piece associated with the device image of the first entity device is configured;

and responding to the touch operation of the user on the control, and displaying a control page or window so that the user can complete the control on the entity equipment through the control interface or window.

In yet another embodiment of the present application, a method for distributing a network is provided. The method comprises the following steps:

identifying the equipment image of at least one entity equipment in the scene image to obtain the equipment information of each entity equipment;

and when determining that the non-networked entity equipment exists based on the equipment information of each entity equipment, distributing the network to the non-networked entity equipment.

In yet another embodiment of the present application, a client device is provided. The client device includes: a memory, a processor, and a display, wherein,

the memory is used for storing programs;

the display coupled with the processor;

the processor, coupled with the memory, to execute the program stored in the memory to:

controlling the display to display a scene image shot by a real scene, wherein the scene image contains an equipment image of at least one entity equipment;

In yet another embodiment of the present application, a client device is provided. The client device includes a memory and a processor, wherein,

the memory is used for storing programs;

In yet another embodiment of the present application, a client device is provided. The client device includes: a memory, a processor and a display; wherein the content of the first and second substances,

the memory is used for storing programs;

the display coupled with the processor;

controlling the display to be on the scene image, and configuring a control piece associated with the device image of the first entity device;

and responding to the touch operation of the user on the control, and controlling the display to display a control page or window so that the user can complete the control on the entity equipment through the control interface or window.

the memory is used for storing programs;

the display coupled with the processor;

According to the technical scheme provided by the embodiment of the application, a user can obtain a scene image through a place where the entity equipment is located through live-action shooting, and then can complete control over the corresponding entity equipment through a mode of operating on the scene image, so that the obtained effect is very visual and the use is convenient; especially, when a plurality of controllable entity devices are arranged in a place, the scene images can visually display the images of the entity devices, and a user can control the corresponding entity devices by performing corresponding operation on the images of the devices to be controlled, so that the control is visual and is not easy to make mistakes.

According to the technical scheme, the user can shoot the place where the entity equipment is located through the live view to obtain the scene image, then the equipment information of the images of the equipment in the scene image is automatically identified, the network distribution is automatically carried out on the entity equipment which is not networked based on the identification result, the user does not need to manually carry out network distribution on each entity equipment, and the use of the user is facilitated.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is a schematic structural diagram of an internet of things system provided by an embodiment of the present application;

fig. 2 is a schematic flowchart of an apparatus control method according to an embodiment of the present application;

fig. 3 is a flowchart illustrating an IoT device control method according to an embodiment of the present application;

fig. 4 is a schematic flowchart of an apparatus control method according to another embodiment of the present application;

fig. 5 is a schematic flowchart of an apparatus control method according to another embodiment of the present application;

FIG. 6a illustrates an example of a control associated with a physical device at a location of a device image of the physical device;

FIG. 6b illustrates an example of a control 600 being displayed in proximity to a device image of a physical device;

FIG. 6c illustrates an example of displaying page elements for characterizing associations between device imagery and control controls of a physical device;

fig. 7 is a schematic flowchart of a network distribution method according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of an apparatus control device according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of an apparatus control device according to another embodiment of the present application;

fig. 10 is a schematic structural diagram of an apparatus control device according to yet another embodiment of the present application;

fig. 11 is a schematic structural diagram of a distribution network device according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of a client device according to an embodiment of the present application.

Detailed Description

Internet of things devices (IoT devices for short) are becoming more and more abundant, such as smart speakers, smart refrigerators, indoor lighting fixtures, sockets, electric cookers, and so on. After the IoT devices access a network (e.g., wifi network), the user can control the IoT devices through an Application (APP) on the mobile phone. For example, when a user opens a mobile phone APP, an identifier (such as a name) of an IoT device and a corresponding control switch thereof are displayed on a mobile phone application interface; the user can complete the control of the IoT equipment through the touch control switch. Alternatively, the user controls the IoT device, e.g., a smart speaker, through voice. The user needs to carry a wake-up word in the control voice, such as a sound box; after the smart sound box detects the awakening word, corresponding response operation (such as weather forecast broadcasting) is performed based on a voice instruction sent by the user. It can be seen that, in the prior art, when a user controls an IoT device, the user cannot see the product form of the entity IoT device on the mobile phone APP, only can see a device identifier (such as a name) of the IoT device, and cannot correspond the entity IoT device to the actual location, which is not intuitive enough.

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application. Further, in some flows described in the specification, claims, and above-described figures of the present application, a number of operations are included that occur in a particular order, which operations may be performed out of order or in parallel as they occur herein. The sequence numbers of the operations, e.g., 101, 102, etc., are used merely to distinguish between the various operations, and do not represent any order of execution per se. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor limit the types of "first" and "second" to be different.

Fig. 1 shows a schematic structural diagram of an internet of things system provided by an embodiment of the present application. As shown in fig. 1, the internet of things system includes: at least one entity device 102 and a client 101. At least one entity device 102 and the client 101 both access the network. Specifically, the client 101 is configured to display a scene image captured by a real scene, where the scene image includes an apparatus image of at least one entity apparatus; and responding to a control event triggered by a user on the scene image aiming at a device image, and executing the control operation of the entity device corresponding to the device image.

For example, as shown in fig. 1, at least one entity device 102 includes: a microwave 1021, a refrigerator 1022, a washing machine 1023, a television 1024, and the like. Each entity device 102 is configured to communicate with an access network (e.g., access point 104 shown in fig. 1) over a physical communication interface or layer (shown in fig. 1 as a wireless network interface or wired connection 103). The Wireless network interface may conform to a Wireless network protocol in the prior art, such as WiFi (Wireless Fidelity, IEEE 802.11). Fig. 1 shows only one implementation example of the access network of each device, that is, the microwave oven 1021, the refrigerator 1022, the washing machine 1023 and the television 1024 access the network through a wireless interface, and the washing machine 1023 can also access the network through a wired manner; in fact, the present embodiment is not particularly limited to whether each device accesses the network in a wireless manner or in a wired manner.

The client 101 may include any terminal device such as a smart phone, a tablet computer, a Personal Digital Assistant (PDA), and a smart wearable device (e.g., a smart watch, smart glasses, etc.).

The access point 104 may be connected to the internet via, for example, an optical communication system (such as a FiOS, Fiber optical Service Fiber Service), a cable modem, a Digital Subscriber Line (DSL) modem, or the like.

Additionally, an entity device, such as an IoT device, referred to in this embodiment refers to any object that has an addressable interface (e.g., an Internet Protocol (IP) address, a bluetooth Identifier (ID), a Near Field Communication (NFC) ID, etc.) and can communicate information to one or more other entity devices over a wired or wireless connection. The entity device may have a passive communication interface (such as a Quick Response (QR) code, a Radio Frequency Identification (RFID) tag, an NFC tag, etc.) or an active communication interface (such as a modem, transceiver, transmitter receiver, etc.). The entity devices may also have a particular set of attributes (e.g., device status or condition (such as whether the IoT device is on or off, idle or active, available for task execution or busy, etc.), cooling or heating functions, environmental monitoring or recording functions, lighting functions, sound emitting functions, etc.), which may be embedded in and/or controlled/monitored by a Central Processing Unit (CPU), microprocessor, ASIC, etc., and configured for connection to a network, such as a local area network or the internet. For example, the entity devices may include, but are not limited to: refrigerators, ovens, microwave ovens, dishwashers, washing machines, clothes dryers, air conditioners, televisions, lights, dust collectors, sprinklers, electricity meters, gas meters, and the like, so long as the devices are equipped with addressable communication interfaces for communicating with a network. The physical devices may also include cell phones, desktop computers, laptop computers, tablet computers, Personal Digital Assistants (PDAs), and the like. Accordingly, an internet of things system may be comprised of a combination of internet-accessible devices (e.g., laptop or desktop computers, cell phones, etc.) and devices that typically do not have internet connectivity (e.g., dishwashers, etc.).

According to the technical scheme provided by the embodiment, a user can obtain a scene image through a place where the entity equipment is located through live-action shooting, and then can complete control over the corresponding entity equipment through a mode of operating on the scene image, so that the obtained effect is very visual and the use is convenient; especially, when a plurality of controllable entity devices are arranged in a place, the scene images can visually display the images of the entity devices, and a user can control the corresponding entity devices by performing corresponding operation on the images of the devices to be controlled, so that the control is visual and is not easy to make mistakes.

Entity devices in the existing internet of things (IoT) are generally connected through a wifi network, and then perform data interaction with a server (such as a cloud) through a client. There is no direct interaction between the client and the entity device, but a server (such as a cloud) is included, and certainly, the local area network can be directly accessed. That is, in the method provided in this embodiment, the control of the entity device may be specifically executed by the server. Specifically, the method provided in this embodiment may further include:

the client 101 is further configured to respond to a control event triggered by a user on the scene image for an equipment image, and acquire equipment information and control content of the equipment image; and sending the device information and the control content to the server 105.

And the server 105 is configured to control the entity device corresponding to the device information according to the control content.

The server may be a server in a local area network, or a network-side server (such as a cloud).

In real-world scenarios, such as televisions, smart speakers, desk lamps, outlets, etc. in a home are spread across many corners of the home. When a user uses a camera of a device such as a mobile phone, a computer, or the like to shoot, the user cannot necessarily shoot all the images within the view range of the camera. Therefore, the client 101 in this embodiment is further configured to respond to a panoramic scanning operation of a user, and acquire a plurality of images acquired in a panoramic scanning mode; generating the scene image in a panoramic mode based on the plurality of images. Wherein, the panorama mode: when the picture with ultra-wide amplitude is shot, redundant positions are reserved behind each photo, so that a plurality of photos shot by a user continuously moving can form an ultra-wide live-action photo. Specifically, when shooting, the lens is moved according to the moving track, so that more scenes enter the camera, and then all the scenes passed by the camera can be collected into one picture, so that one picture covers more information and contents.

The following embodiments further describe a specific workflow of the client in the internet of things system provided by this embodiment.

Fig. 2 shows a flowchart of a device control method according to an embodiment of the present application. The execution subject of the method provided by this embodiment may be a client. Specifically, as shown in fig. 2, the device control method includes:

201. and displaying a scene image shot by the real scene, wherein the scene image contains an equipment image of at least one entity equipment.

202. And responding to a control event triggered by a user on the scene image aiming at a device image, and executing the control operation of the entity device corresponding to the device image.

In 201, the scene image may be acquired by an image acquisition module (e.g., a camera). For example, after monitoring that a user touches an image acquisition control on a client interactive interface, an image acquisition module of the client is called by calling an interface so as to enable the image acquisition module to be in an open state, so that the user can conveniently acquire scene images.

Of course, the scene image may be an image in a panoramic mode. For example, the method provided in this embodiment may further include the following steps:

responding to the panoramic scanning operation of a user, and acquiring a plurality of images collected in a panoramic scanning mode;

generating the scene image in a panoramic mode based on the plurality of images.

The panoramic scanning operation may specifically be: after a user touches a panoramic image acquisition control on a client-side interactive interface, displaying a panoramic scanning moving track on the client-side interface; and the user moves the client end according to the panoramic scanning moving track so as to collect the scenery entering the viewing range on the scanning moving track. It should be noted that, for a process of generating a scene image in a panoramic mode based on multiple images, reference may be made to the prior art, and this embodiment is not described herein again.

In 202, the control event may be generated after the user clicks a device image on the scene image.

In an implementation technical solution, in this embodiment, in step 202, "in response to a control event triggered by a user on the scene image for a device image, execute a control operation on a physical device corresponding to the device image", may be implemented by adopting the following steps:

2021. and responding to a control event triggered by a user on the scene image aiming at a device image, and acquiring the device information and the control content of the device image.

2022. And sending the equipment information and the control content to the server so that the server controls the entity equipment corresponding to the equipment information according to the control content.

Wherein the device information may include, but is not limited to: device identification, network status, and operating status.

The 2021 may include the steps of:

and S1, responding to the clicking operation of the user on the scene image, and acquiring the device image selected by the user.

Specifically, the step S1 may include: responding to a clicking operation of a user on the scene image, and acquiring an operation position of the clicking operation; and detecting the equipment image in the detection area with the operation position as the center on the scene image, if the equipment image is not detected, expanding the detection area outwards, and continuing to detect in the expanded detection area until the equipment image exits. In one embodiment, each device image is pre-stored locally, and during the detection process, whether the device image is detected or not can be determined by image comparison.

Alternatively, each physical device has a pattern (e.g., a two-dimensional code or a bar code) for identifying the device, and the patterns are oriented outward before the scene image is captured so that the patterns can be captured. Thus, step S1 may include: responding to a clicking operation of a user on the scene image, and acquiring an operation position of the clicking operation; and detecting a pattern meeting a preset requirement in a detection area with the operation position as the center on the scene image, wherein if the pattern meeting the requirement is detected, the pattern is the equipment image selected by the user.

And S2, identifying the equipment image to obtain the equipment information.

In an achievable technical scheme, image recognition (machine learning or deep learning and other technologies are used for carrying out object recognition on equipment images) is carried out on the equipment images to obtain equipment types or equipment names (such as a refrigerator and a television); and then, acquiring the network state, the working state and the like of the equipment according to the equipment category or the equipment name so as to obtain the equipment information.

Or, in another realizable technical scheme, the physical device is provided with patterns such as a bar code or a two-dimensional code; before shooting a scene image, a user enables the bar code and the two-dimensional code to face outwards so that the two-dimensional code can be shot; the physical device has a bar code or a two-dimensional code on the device image presented in the scene image, and the device information of the physical device can be obtained by identifying the bar code or the two-dimensional code on the device image. Specifically, the step S2 may include the following steps:

detecting a pattern meeting a preset requirement in the equipment image;

and when the pattern is detected, identifying the pattern to obtain the equipment information.

Wherein, the pattern that accords with preset requirement includes: two-dimensional codes or bar codes.

And S3, responding to the control operation of the user for the equipment image, and determining the control content.

The control operation may be an operation of a user toggling a switch control, or a selection operation of a function, or the like. For example, the intelligent electric cooker can select a porridge cooking mode, a rice cooking mode, an appointment mode and the like. After the user control operation is finished, the control content controlled by the user at this time can be obtained. For example, if the user toggles the switch control to the on state, the determined control content is the on control.

Further, before performing step 202 in this embodiment, the method provided in this embodiment may further include the following steps:

205. and detecting whether the physical devices of each device image in at least one device image in the scene image have finished the distribution network.

206. And when the entity equipment of the unfinished distribution network exists, executing distribution network operation to enable the entity equipment of the unfinished distribution network to access the network.

The implementation basis of the embodiment is as follows: the entity equipment needs to complete the distribution network, namely, the entity equipment has successfully accessed the network. Only when the entity equipment is accessed to the network, the control of the entity equipment can be realized. Therefore, before the method provided by the embodiment is implemented, the distribution network detection can be performed on the entity device presenting the device image in the scene image. Only when each entity device is ensured to complete the distribution network, the corresponding device image on the scene image can be used as a control object.

In the internet of things system, some entity devices such as an intelligent socket, an intelligent air conditioner, an intelligent air purifier and the like cannot search like devices such as a computer, a mobile phone and the like because of no human-computer interaction interface, select a specified AP and input a connection password. The above steps 205 and 206 are intended to address the networking requirements of this type of device. Further, the name (SSID) and password of the AP are notified to a network connection module (e.g., wifi module) in the entity device in some way, and then the entity device connects the specified AP according to the received SSID and password.

The distribution network operates in the following two modes:

and (3) SoftAP distribution network mode: the entity equipment needs to be in an AP mode, and the client is used for an STA mode; the client is connected with the AP of the entity equipment to form a local area network; the client sends the SSID and the password which need to be connected with the route to the entity equipment, and the entity equipment actively connects with the designated route and then completes the connection.

One-key configuration (SmartConfig) mode: a network connection module (such as a wifi module) in the entity device is in a Promiscuous mode (Promiscuous Model), and monitors all messages in the network; the client encodes the SSID and the password into the UDP message and sends the UDP message through a broadcast packet or a multicast packet. And the entity equipment receives the UDP message, decodes the UDP message to obtain the correct SSID and password, and then actively connects the route of the specified SSID to complete the connection.

Still further, the method provided by this embodiment may further include the following steps:

207. and if the operation of the distribution network fails, displaying prompt information to prompt a user to confirm whether the entity equipment with the failed operation of the distribution network is opened and works in a distribution network mode.

According to the distribution process, the equipment can be distributed only when the equipment is opened and works and in a distribution network mode (such as a WIFI distribution network mode). The purpose of displaying the prompt message is to prompt the user to see whether the entity device is not turned on and works in the distribution network mode.

The following describes a technical solution provided by the embodiment of the present application, taking two-dimensional codes visible on physical devices as an example.

After the user enters the client application, the networking state of each IoT device (i.e., the entity device in each embodiment) may be checked first, and if there is an IoT device that is not networked, all IoT devices that need to be configured with a network may enter the network configuration mode, and it is ensured that each IoT device is located in a visible area, and the two-dimensional code of each device is also visible. After the preparation is completed, as shown in fig. 3, the method for controlling the device according to this embodiment may specifically include:

301. in response to a panoramic scanning request triggered by a user, entering an app panoramic scanning mode, and scanning a scene in a certain angle (such as a scene of 90 degrees, 135 degrees, 180 degrees or 360 degrees) in an area to obtain a scene image in the panoramic mode.

302. And after scanning is finished, displaying the scene image and identifying all the two-dimensional codes in the scene image to obtain a plurality of pieces of equipment information.

303. Determining whether there is an IoT device that is not networked based on the plurality of device information; if yes, go to step 304; otherwise, step 305 is entered.

304. The non-networked IoT device is configured and performs step 305.

305. Responding to the clicking operation of a user in a scene image, and acquiring a two-dimensional code at the operation position of the clicking operation; and identifying the two-dimensional code to obtain equipment information.

306. And determining the control content in response to the control operation performed by the user for the selected equipment image.

307. And sending the equipment information and the control content to a server, and controlling the IoT equipment corresponding to the equipment information by the server according to the control content.

Another embodiment of the present application provides an internet of things system, where the architecture of the internet of things system is the same as the structural schematic diagram of the internet of things system shown in fig. 1. The difference between this embodiment and the embodiment shown in fig. 1 is: the execution functions of the clients are different. Specifically, the internet of things system provided in this embodiment, like the embodiment shown in fig. 1, includes: at least one entity device 102 and a client 101. The areas are as follows: the client 101 is configured to respond to a click operation triggered by a user on a scene image, and acquire device information of a device image selected by the user; the scene image is obtained through live-action shooting and contains an equipment image of at least one entity equipment; monitoring control operation of a user for the equipment image when detecting that the entity equipment corresponding to the equipment information is in a networking state; and after monitoring that the user finishes the control operation, executing the control operation on the entity equipment corresponding to the equipment image based on the operation content of the control operation.

Likewise, the control of the entity device can be realized by the server side. That is, the technical solution provided by this embodiment further includes:

the client 101 is further configured to monitor that the user completes the control operation, and send the operation content and the device information to the server 105;

the server 105 is configured to control the entity device corresponding to the device information according to the operation content.

In the technical solutions of the previous embodiments of this embodiment, before a user operates a scene image, a network is first configured for physical devices appearing in the scene image. In fact, it is also feasible for the user to operate and detect whether to network or not. That is, according to the technical scheme provided by this embodiment, after a user clicks an equipment image on a scene image, it is determined whether an entity device corresponding to the equipment image is in a networking state; and if the network is in the disconnected state, carrying out subsequent actions after the network is distributed. The workflow of the client can be seen in the following embodiments.

Fig. 4 is a flowchart illustrating an apparatus control method according to another embodiment of the present application. The execution subject of the method provided by this embodiment may be a client. As shown in fig. 4, the device control method includes:

401. and responding to the clicking operation triggered by the user on the scene image, and acquiring the equipment information of the equipment image selected by the user.

The scene image is obtained through live-action shooting and contains an equipment image of at least one entity equipment. In a specific implementation, the scene image may be a scene image in a panoramic mode.

402. And monitoring the control operation of the user aiming at the equipment image when detecting that the entity equipment corresponding to the equipment information is in the networking state.

403. And after monitoring that the user finishes the control operation, executing the control operation on the entity equipment corresponding to the equipment image based on the operation content of the control operation.

In an implementation technical solution, the step 401 of obtaining device information of a device image selected by a user in response to a click operation triggered by the user on a scene image may specifically be:

responding to a click operation triggered by a user on a scene image, and acquiring an equipment image at the operation position of the click operation; and identifying the equipment image at the operation position to obtain the equipment information of the equipment image selected by the user. In particular, image recognition techniques (e.g., deep learning techniques) may be used to identify the device image to identify the device type or name. Or the equipment image is provided with a two-dimensional code or a bar code, and equipment information is obtained by identifying the two-dimensional code or the bar code on the equipment image selected by the user.

In an implementation technical solution, the step 402 of monitoring a control operation performed by a user for an equipment image when detecting that an entity equipment corresponding to the equipment information is in a networking state may specifically be implemented by the following steps:

4021. and when detecting that the entity equipment corresponding to the equipment information is in a networking state, displaying a control interface or window corresponding to the equipment image.

4022. And monitoring the control operation of the user in the control interface or the window.

Only when the entity equipment is in a networking state, the user can control the entity equipment through the client. And when detecting that the entity equipment corresponding to the equipment information is in a networking state, displaying a control interface or window corresponding to the equipment image, and waiting for control operation of a user in the control interface or window. In specific implementation, the control interface or window may include: switch controls, function options, appointment time settings, etc. Specifically, the control interfaces or windows corresponding to different types of entity devices are different, and may be set based on the specific device characteristics of the entity devices, which is not specifically limited in this embodiment.

In an implementation manner, the step 403 "after monitoring that the user completes the control operation, executing the control operation on the physical device corresponding to the device image based on the operation content of the control operation" includes:

and after monitoring that the user finishes the control operation, sending the operation content and the equipment information to a server, so that the server controls entity equipment corresponding to the equipment information according to the operation content.

Further, the method provided by the embodiment of the present application may further include the following steps:

404. and when detecting that the entity equipment corresponding to the equipment information is in a network disconnection state, executing distribution network operation to enable the entity equipment corresponding to the equipment information to be accessed into a network.

405. And after the entity equipment corresponding to the equipment information is successfully accessed into the network, the equipment image is the object capable of performing control operation.

The operation of the distribution network in 404 may refer to corresponding contents in the above embodiments, and details are not described here.

Fig. 5 is a flowchart illustrating an apparatus control method according to another embodiment of the present application. Also, the execution subject of the method provided by this embodiment may be a client. As shown in fig. 5, the device control method includes:

501. and displaying a scene image shot by the real scene, wherein the scene image contains an equipment image of at least one entity equipment.

502. And on the scene image, a control piece associated with the device image of the first entity device is configured.

503. And responding to the touch operation of the user on the control, and displaying a control page or window so that the user can complete the control on the entity equipment through the control interface or window.

For reference 502, reference may be made to the related contents in the above embodiments, and details are not repeated here.

In 502, the display modes of the control may include, but are not limited to, the following:

configuring an invisible control for the first entity equipment in an implicit mode at the equipment image position of the first entity equipment; or

Displaying a control associated with the first entity device at the position of the device image of the first entity device; or

Displaying a control associated with the first physical device in the vicinity of the device image of the first physical device; or

And displaying a control equipped for the device image of the first entity device on the scene image, and displaying a page element for representing an association relationship between the device image of the first entity device and the control.

FIG. 6a illustrates an example of a control associated with a physical device at a location of a device image of the physical device; FIG. 6b shows an example of displaying a control 600 associated with a physical device in proximity to the device imagery thereof; and FIG. 6c shows an example of displaying page elements for characterizing associations between device imagery and control controls of a physical device. In the example shown in fig. 6c, the page elements used for representing the association relationship, that is, the device categories of the entity devices displayed on the control controls; the user can associate the device image in the figure with the control according to the page elements of the display device type on each control.

In an implementation solution, the step 502 of providing, on the scene image, a control associated with the device image of the first physical device may specifically include:

5021. and identifying the equipment image of the first entity equipment to obtain the equipment information of the first entity equipment.

5022. And when the first entity equipment is determined to be in the networking state based on the equipment information of the first entity, configuring a control associated with the equipment image of the first entity equipment.

5021, an image recognition technology (such as a deep neural network technology) can be used to recognize the device image to obtain a device type or name; then, according to the device type or name, the information such as the network state and the working state is obtained. Or, the device information of the entity device corresponding to the device image is obtained by identifying the two-dimensional code or the barcode in the device image.

503. and identifying the device image of the second entity device in the scene image to obtain the device information of the second entity device.

504. And when the second entity equipment is determined to be in the network disconnection state based on the equipment information of the second entity, a network distribution control associated with the equipment image of the second entity equipment is configured.

An example of a washing machine equipped with a device image display with a distribution network control 601 associated therewith is shown in fig. 6 a.

Fig. 7 shows a schematic flow chart of a network distribution method provided in an embodiment of the present application. The execution subject of the method provided by this embodiment may be a client. Specifically, as shown in fig. 7, the device control method includes:

601. and displaying a scene image shot by the real scene, wherein the scene image contains an equipment image of at least one entity equipment.

602. And identifying the equipment image of at least one entity equipment in the scene image to obtain the equipment information of each entity equipment.

603. And when determining that the non-networked entity equipment exists based on the equipment information of each entity equipment, distributing the network to the non-networked entity equipment.

For specific implementation of the above 601-603, reference may be made to corresponding contents in the above embodiments, and details are not described here.

According to the other technical scheme, the user can shoot the place where the entity equipment is located through the live view to obtain the scene image, then the equipment information of the images of the equipment in the scene image is automatically identified, the network is automatically distributed to the entity equipment which is not networked based on the identification result, the user does not need to manually distribute the network to each entity equipment, and the user can use the network conveniently

In an implementation solution, the step 602 "identifying the device image of at least one physical device in the scene image to obtain the device information of each physical device" includes:

6021. and acquiring a pattern meeting the requirements in the equipment image of each entity equipment.

6022. And identifying the patterns meeting the requirements in the equipment images of the entity equipment to obtain the equipment information of the entity equipment.

In practice, the pattern meeting the requirement may be, but is not limited to: bar codes, two-dimensional codes, etc.

After the distribution network of the entity equipment is completed through the steps so that the entity equipment is successfully accessed into the network, the images of the equipment in the scene image are the objects which can be controlled.

It should be noted that the clicking operation of the user on the scene image mentioned in the foregoing embodiments may be, for example: long press, continuous click, single click, etc.

Fig. 8 shows a schematic structural diagram of a device control apparatus according to an embodiment of the present application. As shown in the figure, the device control apparatus includes: a display module 11 and an execution module 12. The display module 11 is configured to display a scene image captured by a real scene, where the scene image includes an apparatus image of at least one entity apparatus; the execution module 12 is configured to, in response to a control event triggered by a user on the scene image for a device image, execute a control operation on a physical device corresponding to the device image.

Further, the execution module 12 is further configured to: responding to a control event triggered by a user on the scene image aiming at an equipment image, and acquiring equipment information and control content of the equipment image; and sending the equipment information and the control content to the server so that the server controls the entity equipment corresponding to the equipment information according to the control content.

Further, the device information includes at least one of the following information: device identification, network status, and operating status.

Further, the execution module 12 is further configured to: responding to the clicking operation of the user on the scene image, and acquiring an equipment image selected by the user; identifying the equipment image to obtain the equipment information; and determining the control content in response to the control operation of the user on the equipment image.

Further, the execution module 12 is further configured to: detecting a pattern meeting a preset requirement in the equipment image; and when the pattern is detected, identifying the pattern to obtain the equipment information.

The patterns meeting the preset requirements include but are not limited to: two-dimensional codes or bar codes, etc.

Further, the device control apparatus provided in this embodiment may further include: the device comprises an acquisition module and a generation module. The acquisition module is used for responding to the panoramic scanning operation of a user and acquiring a plurality of images acquired in a panoramic scanning mode; the generating module is configured to generate the scene image in a panoramic mode based on the plurality of images.

Further, the device control apparatus provided in the embodiment of the present application may further include: detection module and distribution network module. The detection module is used for detecting whether the physical devices of each device image in at least one device image in the scene image complete the distribution network; the distribution network module is used for executing distribution network operation when the entity equipment of the unfinished distribution network exists, so that the entity equipment of the unfinished distribution network is accessed to the network.

Further, the device control apparatus provided in the embodiment of the present application may further include a display module, where the display module is configured to display a prompt message to prompt a user to determine whether the entity device that fails in the operation of the distribution network is turned on and operates in the distribution network mode if the operation of the distribution network fails.

Here, it should be noted that: the device control apparatus provided in the foregoing embodiments may implement the technical solutions described in the foregoing method embodiments, and the specific implementation principle of each module or unit may refer to the corresponding content in the foregoing method embodiments, which is not described herein again.

Fig. 9 shows a schematic structural diagram of a device control apparatus according to another embodiment of the present application. As shown in fig. 9, the device control apparatus includes: the device comprises an acquisition module 21, a detection module 22, a monitoring module 23 and a control module 24. The acquiring module 21 is configured to respond to a click operation triggered by a user on a scene image, and acquire device information of a device image selected by the user; the scene image is obtained through live-action shooting and contains an equipment image of at least one entity equipment; the detection module 22 is configured to detect whether an entity device corresponding to the device information is in a networking state; the monitoring module 23 is configured to monitor a control operation performed by a user for the device image when the entity device corresponding to the device information is in a networking state; the control module 24 is configured to execute a control operation on the entity device corresponding to the device image based on the operation content of the control operation after the monitoring module monitors that the user completes the control operation.

Further, the monitoring module 23 is further configured to display a control interface or a window corresponding to the device image when the entity device corresponding to the device information is in a networking state; and monitoring the control operation of the user in the control interface or the window.

Further, the device control apparatus provided in this embodiment further includes a distribution network module. The distribution network module is further used for executing distribution network operation to enable the entity equipment corresponding to the equipment information to be accessed to the network when the entity equipment corresponding to the equipment information is in a disconnected state; and after the entity equipment corresponding to the equipment information is successfully accessed into the network, the equipment image is the object capable of performing control operation.

Further, the control module 24 is further configured to: and after the monitoring module monitors that the user completes the control operation, the operation content and the equipment information are sent to a server, so that the server controls entity equipment corresponding to the equipment information according to the operation content.

Fig. 10 is a schematic structural diagram of an apparatus control device according to still another embodiment of the present application. As shown in the figure, the device control apparatus includes: a first display module 31, a second display module 32 and a third display module 33. The first display module 31 is configured to display a scene image captured by a real scene, where the scene image includes an apparatus image of at least one entity apparatus; the second display module 32 is configured to configure a control associated with the device image of the first entity device on the scene image; the third display module 33 is configured to display a control page or window in response to a touch operation of the user on the control, so that the user completes control over the entity device through the control interface or window.

Further, the second display module 32 is further configured to:

identifying the equipment image of the first entity equipment to obtain equipment information of the first entity equipment;

and when the first entity equipment is determined to be in the networking state based on the equipment information of the first entity, configuring a control associated with the equipment image of the first entity equipment.

Further, the device control apparatus provided in this embodiment further includes: an identification module and a fourth display module. The identification module is used for identifying an equipment image of a second entity equipment in the scene image to obtain equipment information of the second entity equipment; the fourth display module is configured to configure a distribution network control associated with the device image of the second entity device when it is determined that the second entity device is in the network disconnection state based on the device information of the second entity.

Fig. 11 shows a schematic structural diagram of a distribution network device according to an embodiment of the present application. As shown in fig. 11, the distribution network apparatus includes: display module 41, identification module 42 and distribution network module 43. The display module 41 is configured to display a scene image captured by a real scene, where the scene image includes an apparatus image of at least one entity apparatus; the identification module 42 is configured to identify an apparatus image of at least one entity apparatus in the scene image, so as to obtain apparatus information of each entity apparatus; the distribution network module 43 is configured to distribute a network to the non-networked entity devices when determining that the non-networked entity devices exist based on the device information of each entity device.

According to the technical scheme provided by the embodiment, the user can shoot the place where the entity equipment is located through the live view to obtain the scene image, then the equipment information of the images of the equipment in the scene image is automatically identified, the network distribution is automatically carried out on the entity equipment which is not networked based on the identification result, the user does not need to manually carry out the network distribution on each entity equipment, and the use of the user is facilitated

Further, the identification module 42 is further configured to obtain a pattern meeting requirements in the device image of each entity device; and identifying the patterns meeting the requirements in the equipment images of the entity equipment to obtain the equipment information of the entity equipment.

Here, it should be noted that: the distribution network device provided in the above embodiments may implement the technical solutions described in the above method embodiments, and the specific implementation principle of each module or unit may refer to the corresponding content in the above method embodiments, and is not described herein again.

Fig. 12 shows a schematic structural diagram of a client device according to an embodiment of the present application. As shown in fig. 12, the client device includes: memory 51, processor 52, and display 54. Wherein the content of the first and second substances,

the memory 51 is used for storing programs;

the display 54 coupled with the processor;

the processor 52, coupled to the memory 51, is configured to execute the program stored in the memory 51, so as to:

controlling the display 54 to display a scene image captured by a real scene, wherein the scene image includes an equipment image of at least one physical equipment;

The processor 52 may also implement other functions besides the above functions when executing the program in the memory 51, which may be specifically referred to the description of the foregoing embodiments.

Further, as shown in fig. 12, the client device further includes: communication components 53, display 54, power components 55, audio components 56, and the like. Only some of the components are schematically shown in fig. 12, and the client device is not meant to include only the components shown in fig. 12.

Another embodiment of the present application provides a client device. The structure of the client device provided by this embodiment can be referred to the structure shown in fig. 12. The client device provided by the embodiment includes: a memory and a processor, wherein,

the memory is used for storing programs;

When the processor executes the program in the memory, the processor may implement other functions in addition to the above functions, which may be specifically referred to the description of the foregoing embodiments.

Further, the client device further comprises: communication components, power components, audio components, and the like.

Another embodiment of the present application provides a client device. The structure of the client device provided by this embodiment can be referred to the structure shown in fig. 12. The client device provided by the embodiment includes: a memory, a processor and a display; wherein the content of the first and second substances,

the memory is used for storing programs;

the display coupled with the processor;

the memory is used for storing programs;

the display coupled with the processor;

According to the technical scheme, the user can shoot the place where the entity equipment is located through the real scene to obtain the scene image, then the equipment information of the images of the equipment in the scene image is automatically identified, the network distribution is automatically carried out on the entity equipment which is not networked based on the identification result, the user does not need to manually carry out the network distribution on each entity equipment, and the use of the user is facilitated.

The memory in the embodiments described above may be configured to store other various data to support operations on the client device. Examples of such data include instructions for any application or method operating on the client device. The memory may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

Accordingly, the embodiments of the present application also provide a computer-readable storage medium storing a computer program, where the computer program can implement the steps or functions of the information processing method provided in the foregoing embodiments when executed by a computer.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

1. An apparatus control method characterized by comprising:

2. The method of claim 1, wherein performing a control operation on a physical device corresponding to a device movie in response to a control event triggered by a user on the scene image for the device movie comprises:

responding to a control event triggered by a user on the scene image aiming at an equipment image, and acquiring equipment information and control content of the equipment image;

and sending the equipment information and the control content to the server so that the server controls the entity equipment corresponding to the equipment information according to the control content.

3. The method of claim 2, wherein the device information comprises at least one of: device identification, network status, and operating status.

4. The method of claim 2, wherein acquiring device information and control content of a device movie in response to a control event triggered by a user on the scene image for the device movie comprises:

responding to the clicking operation of the user on the scene image, and acquiring an equipment image selected by the user;

identifying the equipment image to obtain the equipment information;

and determining the control content in response to the control operation of the user on the equipment image.

5. The method of claim 4, wherein identifying the device image to obtain the device information comprises:

detecting a pattern meeting a preset requirement in the equipment image;

6. The method of claim 5, wherein the pattern meeting the preset requirements comprises: two-dimensional codes or bar codes.

7. The method of any one of claims 1 to 6, further comprising:

8. The method according to any one of claims 1 to 6, wherein, in response to a control event triggered by a user on the scene image for a device movie, performing a control operation on a physical device corresponding to the device movie, further comprises:

detecting whether entity equipment of each equipment image in at least one equipment image in the scene image has finished the distribution network;

and when the entity equipment of the unfinished distribution network exists, executing distribution network operation to enable the entity equipment of the unfinished distribution network to access the network.

9. The method of claim 8, further comprising:

and if the operation of the distribution network fails, displaying prompt information to prompt a user to confirm whether the entity equipment with the failed operation of the distribution network is opened and works in a distribution network mode.

10. An internet of things system, comprising:

at least one physical device;

11. The system of claim 10, further comprising:

the client is further used for responding to a control event triggered by a user on the scene image aiming at an equipment image, and acquiring equipment information and control content of the equipment image; sending the equipment information and the control content to the server

And the server is used for controlling the entity equipment corresponding to the equipment information according to the control content.

12. The system of claim 10 or 11,

the client is also used for responding to the panoramic scanning operation of the user and acquiring a plurality of images collected in the panoramic scanning mode; generating the scene image in a panoramic mode based on the plurality of images.

13. An apparatus control method characterized by comprising:

14. The method according to claim 13, wherein monitoring a control operation performed by a user for the device image when detecting that the physical device corresponding to the device information is in a networking state comprises:

when detecting that the entity equipment corresponding to the equipment information is in a networking state, displaying a control interface or window corresponding to the equipment image;

and monitoring the control operation of the user in the control interface or the window.

15. The method of claim 13 or 14, further comprising:

when detecting that the entity equipment corresponding to the equipment information is in a network disconnection state, executing distribution network operation to enable the entity equipment corresponding to the equipment information to be accessed into a network;

and after the entity equipment corresponding to the equipment information is successfully accessed into the network, the equipment image is the object capable of performing control operation.

16. The method according to claim 13 or 14, wherein the monitoring that the user completes the control operation and then executing the control operation on the physical device corresponding to the device image based on the operation content of the control operation includes:

17. An internet of things system, comprising:

at least one physical device;

18. The system of claim 17, further comprising:

the client is also used for monitoring that the user finishes the control operation and then sending the operation content and the equipment information to a server;

and the server is used for controlling the entity equipment corresponding to the equipment information according to the operation content.

19. An apparatus control method characterized by comprising:

20. The method of claim 19, wherein providing a control associated with the device imagery of the first physical device on the scene image comprises:

21. The method of claim 19 or 20, wherein providing a device image of the first physical device with a control associated therewith on the scene image comprises:

22. The method of claim 19 or 20, further comprising:

identifying an equipment image of a second entity equipment in the scene image to obtain equipment information of the second entity equipment;

and when the second entity equipment is determined to be in the network disconnection state based on the equipment information of the second entity, a network distribution control associated with the equipment image of the second entity equipment is configured.

23. A method for network distribution, comprising:

24. The method of claim 23, wherein identifying the device image of at least one physical device in the scene image to obtain the device information of each physical device comprises:

acquiring a pattern meeting the requirements in the equipment image of each entity equipment;

and identifying the patterns meeting the requirements in the equipment images of the entity equipment to obtain the equipment information of the entity equipment.

25. A client device, comprising: a memory, a processor, and a display, wherein,

the memory is used for storing programs;

the display coupled with the processor;

26. A client device comprising a memory and a processor, wherein,

the memory is used for storing programs;

27. A client device, comprising: a memory, a processor and a display; wherein the content of the first and second substances,

the memory is used for storing programs;

the display coupled with the processor;

28. A client device, comprising: a memory, a processor, and a display, wherein,

the memory is used for storing programs;

the display coupled with the processor;