WO2019062210A1

WO2019062210A1 - Interaction method and device, storage device and terminal

Info

Publication number: WO2019062210A1
Application number: PCT/CN2018/091128
Authority: WO
Inventors: 黄辉; 贾巨涛; 宋德超; 冼海鹰; 何贤俊
Original assignee: 格力电器（武汉）有限公司; 珠海格力电器股份有限公司
Priority date: 2017-09-30
Filing date: 2018-06-13
Publication date: 2019-04-04
Also published as: CN107861388A

Abstract

Disclosed in the present application are an interaction method and device, a storage device and a terminal. The method comprises: obtaining an AR device model of a device entity to be interacted with; receiving touch location information of a user on the AR device model; and determining a control instruction corresponding to the touch location information, and sending the control instruction to the device entity. The solution of the present application can overcome such defects in the prior art as poor intuitiveness, inconvenience, and poor user experience, and therefore achieve beneficial effects of good intuitiveness, convenient operation, and good user experience. The defects in the prior art are overcome and the beneficial effects are achieved.

Description

Interactive method, device, storage device and terminal

Technical field

The present invention relates to the field of home appliance technology, and particularly relates to an interaction method, device, storage device and terminal, in particular to an air conditioner interaction method based on AR and wireless communication technology, a device corresponding to the method, and a storage storing the method instruction. A device, and a terminal capable of executing the method instructions.

Background technique

Air conditioning (ie air conditioner) can adjust and control the temperature, humidity, cleanliness and speed of the ambient air in the building/structure. The existing air conditioner can control the switch of the air conditioner and the temperature setting through the setting buttons on the remote controller or the air conditioner panel. However, the existing methods of interaction between people and air conditioners are mostly based on remote control panels to achieve control functions, and the operation mode is not intuitive enough and not convenient enough.

In the prior art, there are defects such as poor intuitiveness, inconvenience, and poor user experience.

Application content

The purpose of the present application is to provide an interaction method, a device, a storage device, and a terminal, so as to solve the problem that the operation mode of controlling the air conditioner through the remote control panel is not convenient in the prior art, and the operation is convenient.

The present application provides an interaction method, including: acquiring an AR device model of a device entity to be interacted; receiving touch location information of the AR device model by the user; determining a control instruction corresponding to the touch location information, and determining the Control instructions are sent to the device entity.

Optionally, acquiring an AR device model of the device entity to be interacted, comprising: receiving image information of the device entity scanned by the scanning module or the identification image thereof, and/or input by the input module; Performing AR processing on the image information to generate the AR device model; and/or extracting feature information of the device entity from the image information, and calling the corresponding information corresponding to the feature information from a pre-established information base An AR device model; and/or receiving identification information of the device entity scanned by the scanning module or its identification image, and/or input by the input module; setting the identification device and setting the AR device model according to the setting Corresponding relationship, determining the AR device model corresponding to the identification information.

Optionally, the feature information includes: an outer contour; and/or the identification information includes at least one of a two-dimensional code, a barcode, an ID code, and an RF label.

Optionally, the method further includes: receiving an execution result of the control instruction by the device entity.

Optionally, receiving an execution result of the control instruction by the device entity includes: receiving an execution result that is actively sent after the device entity executes the control instruction.

Optionally, the method further includes: when the execution result is not received, sending a query request to request the device entity to send the execution result.

Optionally, the method further includes: when the consecutively sending the set number of query messages still does not receive the response, discarding the query; and/or, when determining that the device entity does not execute the control instruction, resume sending the control instruction The previous state.

Optionally, the method further includes: determining a current running state of the device entity corresponding to the execution result; and displaying the current running state synchronization into the AR device model to implement the AR device model and the A synchronous display of the current operating state.

Optionally, the device entity includes: an air conditioner, a water heater, a refrigerator, a washing machine, an induction cooker, an electric heater, an electric oven, a rice cooker, an electric fan, a water purifier, an air purifier, a range hood, and a disinfection cabinet. And at least one of the touch position information includes: a top portion of the AR device model, a bottom portion, a position of the body, and a position other than the body in the display interface where the body is located And at least one of: the control instruction, comprising: at least one of a power on, a power off, an open state, and an operation mode.

In an aspect of the present invention, an aspect of the present application provides an interaction apparatus, including: a communication unit, configured to acquire an AR device model of a device entity to be interacted; and the communication unit is further configured to receive a user to the AR device. a touch location information of the model; a control unit configured to determine a control instruction corresponding to the touch location information, and send the control instruction to the device entity by the communication unit.

Optionally, the acquiring, by the communication unit, the AR device model of the device entity to be interacted comprises: receiving, by the scanning module, scanning the device entity or its identification image, and/or the device entity input by the input module. Image information; performing AR processing on the image information to generate the AR device model; and/or extracting feature information of the device entity from the image information, and calling the feature from a pre-established information base The AR device model corresponding to the information; and/or receiving identification information of the device entity scanned by the scanning module or the identification image thereof and/or input by the input module;

The AR device model corresponding to the identification information is determined according to a correspondence between the setting identification information and the set AR device model.

Optionally, the method further includes: the communication unit is further configured to receive an execution result of the control instruction by the device entity.

Optionally, the receiving, by the communications unit, the execution result of the control instruction by the device entity, specifically: receiving an execution result that is actively sent after the device entity executes the control instruction.

Optionally, the control unit is further configured to: when the execution result is not received, send a query request to request the device entity to send the execution result.

Optionally, the method further includes: the control unit is further configured to: when the continuous sending of the set number of query messages has not received the response, abandon the query; and/or, when determining that the device entity does not execute the control instruction At the time, the state before the transmission of the control command is resumed.

Optionally, the method further includes: the control unit is further configured to determine a current running state of the device entity corresponding to the execution result; the communication unit is further configured to dynamically display the current running state to the virtual In the AR device model, a synchronous display of the AR device model and the current running state is implemented.

According to another aspect of the present application, a storage device includes: storing, by the storage device, a plurality of instructions, where the multiple instructions are used by a processor to load and execute the interaction method described above. .

In accordance with the above method, the present application further provides a terminal, including: a processor, configured to execute a plurality of instructions; a memory, configured to store a plurality of instructions; wherein the plurality of instructions are used by the The memory is stored and loaded by the processor and performs the interaction method described above; or the interactive device described above.

The solution of the present application realizes convenient interaction between people and equipment by utilizing AR and wireless technology, and realizes convenient control of air-conditioning equipment.

Further, the solution of the present application realizes the interaction between the user and the device by adopting the AR device model, adopts the intuitive interaction mode to improve the operation convenience of the device, and changes the singularity of the traditional control mode through the remote control panel, thereby improving the user operation experience.

Further, the solution of the present application realizes the wireless communication by wireless technology (for example, wifi, bluetooth, zigbee, etc.) by using the augmented reality (AR) technology to simultaneously display the physical condition of the air conditioner and the air conditioner operating state to the screen, and realize the human being. Interact with air conditioning through augmented reality.

Further, the solution of the present application scans the air conditioner physical object or the identification map through a device with a display screen such as a mobile phone or a tablet, generates an AR air conditioning model in the display screen, realizes interaction between the user and the air conditioner, and realizes natural interaction of the air conditioning device.

Therefore, the solution of the present application generates an AR air conditioning model based on the air conditioner or its identification map, and realizes the interaction between the user and the air conditioner by controlling the AR air conditioning model; solving the operation mode of controlling the air conditioner through the remote control panel is not convenient. The problem, thus, overcomes the shortcomings of the prior art, such as poor intuitiveness, inconvenience, and poor user experience, and realizes the beneficial effects of good intuitiveness, convenient operation, and user experience.

Other features and advantages of the present application will be set forth in the description which follows.

The technical solutions of the present application are further described in detail below through the accompanying drawings and embodiments.

DRAWINGS

FIG. 1 is a schematic flowchart diagram of an embodiment of an interaction method according to the present application;

2 is a schematic flowchart of an embodiment of acquiring an AR device model according to image information in the method of the present application;

3 is a schematic flowchart of an embodiment of acquiring an AR device model according to identification information in the method of the present application;

4 is a schematic flow chart of an embodiment of a result of executing a control instruction by a receiving device entity in the method of the present application;

FIG. 5 is a schematic flowchart of an embodiment of determining whether a device entity receives a control instruction in the method of the present application;

6 is a schematic flowchart of an embodiment of performing a fault prompt when a device entity cannot receive a control command in the method of the present application;

7 is a schematic flowchart of an embodiment of synchronously displaying a current running state of a device entity to an AR device model in the method of the present application;

FIG. 8 is a schematic structural diagram of an embodiment of an interaction apparatus according to the present application; FIG.

9 is a schematic diagram of an effect of an embodiment of an AR identification map of a terminal-to-device entity according to the present application;

10 is an AR effect diagram of an embodiment of an AR identification map of a terminal-to-device entity according to the present application;

11 is an interaction flowchart of an embodiment of an apparatus for performing interaction control on a device entity by a terminal according to the present application;

12 is a schematic diagram of a touch position of a body of an embodiment of an AR device model in a terminal of the present application;

FIG. 13 is a view showing a touch position display effect of a display area other than the body of the embodiment of the AR device model in the terminal of the present application.

With reference to the accompanying drawings, the reference numerals in the embodiments of the present application are as follows:

102-communication unit; 104-control unit; 106-prompt unit.

Detailed ways

The technical solutions of the present application will be clearly and completely described in the following with reference to the specific embodiments of the present application and the corresponding drawings. It is apparent that the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

According to an embodiment of the present application, an interaction method is provided, as shown in FIG. 1 is a schematic flowchart of an embodiment of the method of the present application. The interaction method can include:

At step S110, an AR device model of the device entity to be interacted is obtained.

Optionally, the device entity may include: an air conditioner, a water heater, a refrigerator, a washing machine, an induction cooker, an electric heater, an electric oven, a rice cooker, an electric fan, a water purifier, an air purifier, a range hood, and a disinfection cabinet. At least one.

Therefore, through the AR interaction control of multiple device entities, the AR interactive control has a wide application range and good flexibility in use.

In an optional example, a schematic flowchart of an embodiment of acquiring an AR device model according to image information in the method of the present application shown in FIG. 2 may be further described, and a step of acquiring an AR device model of a device entity to be interacted in step S110 is further illustrated. process.

Step S210: Receive image information of the device entity scanned by the scanning module, or the identification image thereof, and/or input by the input module.

Step S220, performing AR processing on the image information to generate the AR device model.

For example, the user scans the air conditioner physical object or the identification map through a device with a display screen such as a mobile phone or a tablet, and generates an AR air conditioning model in the display screen (for example, see the examples shown in FIGS. 9 and 10).

For example, scanning the air conditioner physical object or the identification map of different models, correspondingly generating different AR air conditioning models, and performing human-computer interaction based on the AR air conditioning model.

For example, as shown in FIG. 11 , the air conditioning interaction method based on the AR and the wireless communication technology may be implemented by: after the mobile phone or the tablet device scans successfully, the AR air conditioning model is generated on the screen.

Therefore, the AR device model is obtained by performing AR processing according to the image information of the device entity, and the processing effect is good, and the reliability of generating the AR device model is high.

In an optional example, a schematic flowchart of an embodiment of acquiring an AR device model according to image information in the method of the present application shown in FIG. 2 may be further described, and further, in step S110, the AR device model of the device entity to be interacted is obtained. a process.

Step S230, extracting feature information (for example, external contours, functions, and the like of the device entity) of the device entity from the image information received in step S210.

Optionally, the feature information may include: an outer contour.

Thus, through various forms of feature information, it is advantageous to improve the efficiency and accuracy of the call to the AR device model.

Step S240, calling the AR device model corresponding to the feature information from a pre-established information base.

For example, the AR air conditioner model generated after the scan can be stored, and the stored AR air conditioner model is called when the air conditioner needs to be controlled. For example, when it is necessary to control the air conditioner, the AR air conditioner model corresponding to the feature information can be called by the feature information extracted after scanning.

Therefore, the feature information is extracted based on the image information of the device entity, and the AR device model corresponding to the feature information is invoked, so that the AR device model is obtained with high efficiency and convenience.

In an optional example, a schematic flowchart of an embodiment of acquiring an AR device model according to the identification information in the method of the present application shown in FIG. 3 may be further described, and further, in step S110, the AR device model of the device entity to be interacted is obtained. a process.

Step S310, receiving identification information of the device entity scanned by the scanning module, or the identification image, and/or input by the input module.

Optionally, the identification information may include at least one of a two-dimensional code, a barcode, an ID code, and an RF label.

Thus, through various forms of identification information, it is advantageous to improve the convenience and flexibility of acquiring an AR device model based on the identification information.

Step S320, determining the AR device model corresponding to the identification information according to the correspondence between the setting identification information and the setting AR device model.

Therefore, by calling the AR device model based on the identification information of the device entity, the acquisition convenience is good and the reliability is high.

At step S120, touch location information of a touch operation performed by the user on the AR device model is received.

For example, the air conditioner can be touch-operated in the display screen of the AR air conditioner model.

For example: the software reads the touch location information sensed by the phone or tablet device.

Optionally, the touch location information may include: a top of the AR device model (eg, a top position A of an AR air conditioning model), a bottom (eg, a bottom position B of an AR air conditioning model), and any position of the airframe At least one of any position other than the body in the display interface where the fuselage is located (for example, body positions C and D of the AR air conditioning model).

For example, referring to the example shown in Figures 12 and 13, touch the top logo (location A) of the virtual product in the screen.

For example, referring to the example shown in Figures 12 and 13, touch the bottom tuyere (position B) of the virtual product in the screen.

For example, referring to the example shown in Figures 12 and 13, the body temperature (position C) of the virtual product in the screen is touched.

For example, referring to the example shown in FIG. 12 and FIG. 13, the body (virtual D) of the virtual product in the screen is touched.

Thus, the flexibility and convenience of interactive control based on touch location information is facilitated by various forms of touch location information.

At step S130, a control instruction corresponding to the touch location information is determined, and the control instruction is transmitted to the device entity to implement control of the device entity by the AR device model.

For example, AR and wireless technologies can be used to realize convenient interaction between people and devices.

For example: send the read touch location information to the device via a wireless connection.

For example, wireless communication (such as wifi, Bluetooth, zigbee, etc.) enables wireless communication between people and air conditioners through augmented reality.

Therefore, the interaction between the user and the device entity is realized by the AR device model based on the device entity, and the intuitiveness is strong, the interaction convenience is good, and the user experience is good.

Optionally, the control instruction may include: power on, power off, opening (eg, air volume, temperature, etc.), and operation mode (eg, air outlet mode, cooling mode, heating mode, dehumidification mode) At least one of, etc.).

For example, referring to the example shown in Figures 12 and 13, touch the top logo of the virtual product in the screen (position A): enable power on/off.

For example, referring to the example shown in FIG. 12 and FIG. 13, the bottom air outlet (position B) of the virtual product in the screen is touched: the air outlet swing is realized.

For example, referring to the example shown in Figures 12 and 13, the body temperature (position C) of the virtual product in the touch screen is touched: temperature adjustment is achieved.

For example, referring to the example shown in Figures 12 and 13, touch the body of the virtual product in the screen (position D): provide more personalized settings on both sides of the screen.

Therefore, the convenience and flexibility of interaction between the AR device model and the device entity can be improved by using various control commands corresponding to the touch location information, and the humanization is good.

In an optional implementation manner, the method may further include: receiving an execution result of the control instruction by the device entity, to implement monitoring of execution of the control instruction by the device entity.

For example, the device end executes the received control command, and feeds back the result of the execution to the mobile phone or the tablet device through wireless communication.

Therefore, by the execution result of the control instruction triggered by the AR device model, the receiving device entity implements monitoring of the execution of the device entity based on the control instruction, thereby facilitating the implementation of the interaction between the user and the device entity based on the AR device model. Reliability and security.

In an optional example, an execution result that is actively sent after execution of the control instruction by the device entity is received. For example, the device entity can actively send the execution result after executing the instruction.

For example, the home appliance entity feeds back the execution result to the mobile phone or tablet after executing the instruction.

Optionally, the flow diagram of an embodiment of the execution result of the receiving device entity to the control instruction in the method of the present application shown in FIG. 4 may further be used to further describe the specific result of receiving the execution result of the control entity by the device entity. process.

Step S410, sending a request message requesting the device entity to return an execution result of the control instruction.

Step S420, receiving the execution result returned by the device entity based on the request message.

Therefore, by receiving the execution result by transmitting the request message, erroneous transmission or erroneous reception can be avoided, the communication security is strong, and the execution result of the acquisition is accurate and reliable.

In an optional implementation, the method further includes: determining whether the device entity receives the control instruction.

In an optional example, when the execution result is not received, a query request is sent to request the device entity to send the execution result.

For example, in a special case, when the device entity does not actively send the execution result or the result is not received by the mobile phone, the mobile phone starts to send the query request.

For example, when the mobile phone or the tablet does not receive the feedback message, the query command is sent, and the receiving end sends the execution result message.

For example, when the mobile phone or the tablet does not receive the feedback message, the query command is sent first, and after the n times of the query, the device does not execute the command, and the AR device resumes the pre-operation state.

Optionally, the flowchart of an embodiment of determining whether the device entity receives the control instruction in the method of the present application, as shown in FIG. 5, further illustrates a specific process of determining whether the device entity receives the control instruction.

Step S510: Before receiving the execution result of the control instruction by the device entity, determining whether a feedback message when the device entity receives the control instruction has been received.

Step S520, when the feedback message has been received, receiving, by the device entity, an execution result of the control instruction.

Or step S530, when the feedback message is not received, the control instruction is sent again.

Therefore, by determining whether the device entity receives the control instruction to receive the execution result or resend the control command according to the determination result, it is advantageous to improve the reliability and security of the interaction between the AR device model and the device entity.

In an optional implementation manner, the method may also include: processing when the execution result is not received.

In an optional example, the query is discarded when the consecutively sent query messages of the set number of times have not received the response; and/or, when it is determined that the device entity does not execute the control command, the control command is resumed. The previous state.

For example, if the query message is sent continuously for n times without a response, the query is abandoned, and the operation terminal is not executed, and the sender returns to the original state (pre-operation state).

Optionally, a schematic flowchart of an embodiment of performing a fault prompt when the device entity cannot receive the control command in the method of the present application shown in FIG. 6 further illustrates a specific manner of performing the fault prompt when the device entity cannot receive the control command. process.

Step S610, determining whether the feedback message has not been received within the set duration, and/or the number of times the control command is sent again reaches a set number of times.

Step S620, when the feedback message is not received within the set duration, and/or the number of times reaches a set number of times, prompting (for example, initiating a sound and light, voice, etc. prompt message) to prompt the The device entity cannot receive the control command, and/or the fault condition that the device entity cannot send the feedback message.

Therefore, by prompting the user to promptly intervene or maintain the device entity, the user may prompt the pair when the control command is still unsuccessful or the feedback message of the control instruction is not received by the device entity. The timeliness and reliability of the maintenance of the device entity, on the other hand, can prompt the user to intervene in time to enjoy the service brought by the device entity, and the humanization is good.

In an optional implementation, the process of synchronously displaying the current running state of the device entity to the AR device model may also be included.

In an optional example, the flow diagram of an embodiment of the AR device model for synchronously displaying the current running state of the device entity in the method of the present application shown in FIG. 7 is further described, and the current running state of the device entity is synchronously displayed. The specific process to the AR device model.

Step S710, determining a current running state of the device entity corresponding to the execution result.

In step S720, the current running state synchronization is virtually displayed in the AR device model to implement synchronous display of the AR device model and the current running state.

For example, the synchronization between the virtual device and the actual device is performed to ensure that the state of the air conditioner in the screen is consistent with the real air-conditioning state, thereby realizing the interaction process between the person and the product.

For example, using the augmented reality (AR) technology, the air conditioner equipment and the air conditioner operating state are simultaneously displayed on the screen to realize the natural interaction of the air conditioner.

Among them, the natural interaction is mainly reflected in the operation mode of the direct touch device, and the operation is more intuitive, and conforms to the intuitive operation habit of the user.

Therefore, by synchronizing the current running state of the device entity with the virtual device to the AR device model based on the execution result of the control command by the device entity, the synchronous operation and display between the device entity and the AR device model are realized, and the synchronization is timely and the user is The viewing is intuitive and convenient.

Through a large number of experiments and verification, the technical solution of the present embodiment is adopted, and the convenient interaction between the human and the device is realized by using the AR and the wireless technology, and the air conditioner equipment is conveniently controlled.

According to an embodiment of the present application, an interactive device corresponding to the interaction method is also provided. Referring to Figure 8, there is shown a block diagram of an embodiment of the apparatus of the present application. The interaction device may include a communication unit 102 and a control unit 104.

In an optional example, the communication unit 102 can be configured to acquire an AR device model of a device entity to be interacted with. For the specific functions and processing of the communication unit 102, refer to step S110.

The device entity may include at least: an air conditioner, a water heater, a refrigerator, a washing machine, an induction cooker, an electric heater, an electric oven, a rice cooker, an electric fan, a water purifier, an air purifier, a range hood, and a disinfection cabinet. One.

Optionally, the communication unit 102 obtains an AR device model of the device entity to be interacted, and specifically includes: a process of acquiring an AR device model of the device entity to be interacted.

In an optional specific example, the communication unit 102 may be configured to receive image information of the device entity scanned by the scanning module, or the identification image thereof, and/or input by the input module. The specific function and processing of the communication unit 102 are also referred to step S210.

In an optional specific example, the communication unit 102 may be further configured to perform AR processing on the image information to generate the AR device model. The specific functions and processing of the communication unit 102 are also referred to step S220.

For example, the user scans the air conditioner physical object or the identification map through a device with a display screen such as a mobile phone or a tablet, and generates an AR air conditioner model in the display screen (for example, see the examples shown in FIGS. 9 and 10).

Optionally, the communication unit 102 acquires an AR device model of the device entity to be interacted, and specifically includes another process of acquiring an AR device model of the device entity to be interacted.

In an optional specific example, the communication unit 102 may be further configured to extract feature information of the device entity (eg, external contours, functions, and the like of the device entity) from the image information. The specific functions and processing of the communication unit 102 are also referred to step S230.

Optionally, the feature information may include: an outer contour.

In an optional specific example, the communication unit 102 may be further configured to invoke the AR device model corresponding to the feature information from a pre-established information base. The specific functions and processing of the communication unit 102 are also referred to step S240.

Optionally, the communication unit 102 acquires an AR device model of the device entity to be interacted, and may further include: another process of acquiring an AR device model of the device entity to be interacted.

In an optional specific example, the communication unit 102 may be further configured to receive, by the scanning module, the device entity or an identification image thereof, and/or the identification information of the device entity input by the input module. The specific function and processing of the communication unit 102 are also referred to step S310.

In an optional specific example, the communication unit 102 is further configured to determine, according to the correspondence between the setting identification information and the set AR device model, the AR device model corresponding to the identification information. The specific function and processing of the communication unit 102 are also referred to step S320.

In an optional example, the communication unit 102 is further configured to receive touch location information of the AR device model by a user. For the specific functions and processing of the communication unit 102, refer to step S120.

In an optional example, the control unit 104 can be configured to determine a control instruction corresponding to the touch location information, and send the control instruction to the device entity by the communication unit 102 to achieve The AR device model controls the device entity. For the specific functions and processing of the control unit 104, refer to step S130.

In an optional implementation, the method may further include: receiving a process of the execution result of the control instruction by the device entity.

In an optional example, the communication unit 102 is further configured to receive an execution result of the control instruction by the device entity, to implement monitoring of execution of the control instruction by the device entity.

In an optional example, the communication unit 102 receives an execution result that is actively transmitted after the control instruction is executed by the device entity. For example, the device entity can actively send the execution result after executing the instruction.

Optionally, the execution result may also be obtained by sending a request message, as follows:

In an optional specific example, the communication unit 102 receives the execution result of the control instruction by the device entity, and specifically includes: a process of sending a request message and receiving an execution result.

In an optional specific example, the communication unit 102 may be further configured to send a request message requesting the device entity to return an execution result of the control instruction. The specific function and processing of the communication unit 102 are also referred to step S410.

In an optional specific example, the communication unit 102 may be further configured to receive the execution result returned by the device entity based on the request message. The specific functions and processing of the communication unit 102 are also referred to step S420.

Alternatively, it can also be performed as follows:

In an optional example, the control unit 104 may be further configured to determine, before receiving the execution result of the control instruction by the device entity, whether the device entity has received the control instruction Feedback message. The specific functions and processing of the control unit 104 are also referred to step S510.

In an optional example, the communication unit 102 is further configured to: when the feedback message has been received, receive, by the device entity, an execution result of the control instruction. The specific functions and processing of the communication unit 102 are also referred to step S520.

In an optional example, the communication unit 102 may be further configured to send the control instruction again when the feedback message is not received. The specific functions and processing of the communication unit 102 are also referred to step S530.

Therefore, by determining whether the device entity receives the control instruction to receive the execution result according to the determination result or to send the control instruction again, the reliability and security of the interaction between the AR device model and the device entity are improved.

Alternatively, you can also do it in the following way:

In an optional example, the prompting unit 106 is configured to perform a process of performing a fault prompt when the device entity cannot receive the control command.

In an optional example, the control unit 104 may be further configured to determine whether the feedback message has not been received within the set duration, and/or the number of times the control command is resent has reached a set number of times. The specific function and processing of the control unit 104 are also referred to step S610.

In an optional example, the prompting unit 106 may be configured to initiate a prompt when the feedback message is not received within the set duration, and/or the number of times reaches a set number of times (eg, initiate sound and light) And a prompt message for voice, etc., to prompt the device entity not to receive the control instruction, and/or the fault condition that the device entity cannot send the feedback message. The specific function and processing of the prompting unit 106 are also referred to step S620.

In an optional example, the control unit 104 is further configured to determine a current running state of the device entity corresponding to the execution result. The specific function and processing of the control unit 104 are also referred to step S710.

In an optional example, the communication unit 102 may be further configured to dynamically display the current running state synchronization into the AR device model to implement synchronous display of the AR device model and the current running state. . The specific functions and processing of the communication unit 102 are also referred to step S720.

For example, the synchronization between the virtual device and the actual device is performed to ensure that the state of the air conditioner in the screen is consistent with the state of the real air conditioner, thereby realizing the interaction process between the person and the product.

Since the processing and functions implemented by the apparatus in this embodiment substantially correspond to the foregoing embodiments, principles, and examples of the method shown in FIG. 1 to FIG. 7, the description of the present embodiment is not exhaustive, and reference may be made to the foregoing embodiment. The relevant descriptions in this article will not be repeated here.

Through a large number of experiments and verification, using the technical solution of the present application, the interaction between the user and the device is realized by adopting the AR device model, and the intuitive operation mode is adopted to improve the operation convenience of the device, and the singularity of the traditional control mode through the remote control panel is changed, and the improvement is improved. The user experience.

According to an embodiment of the present application, a storage device corresponding to an interaction method is also provided. The storage device may include: the storage device stores a plurality of instructions; the plurality of instructions are used by the processor to load and execute the interaction method described above.

Since the processing and functions implemented by the storage device of the present embodiment substantially correspond to the foregoing embodiments, principles, and examples of the method shown in FIG. 1 to FIG. 7, the description of the present embodiment is not exhaustive, and the foregoing implementation may be referred to. The relevant descriptions in the examples are not described here.

Through a large number of experiments and verification, using the technical solution of the present application, by using the augmented reality (AR) technology, the air conditioner equipment and the air conditioner operating state are simultaneously displayed on the screen, and realized by wireless technology (for example, wifi, bluetooth, zigbee, etc.). Wireless communication enables people to interact with air conditioners through augmented reality.

According to an embodiment of the present application, a terminal corresponding to an interaction method is also provided. The terminal may include: a processor for executing a plurality of instructions; a memory for storing a plurality of instructions; wherein the plurality of instructions are stored by the memory and loaded and executed by the processor The interaction method described above; or the interaction device described above.

In an optional example, the terminal can utilize AR and wireless technologies to implement convenient interaction between the person and the device.

Optionally, the interaction may be: the user scans the air conditioner physical object or the identification map through a device with a display screen such as a mobile phone or a tablet, and generates an AR air conditioning model in the display screen (for example, see FIG. 9 and FIG. 10). example).

Among them, AR (Augmented Reality) technology imaging, combined with virtual images and real images for user reference and use.

AR technology is a technology that calculates the position and angle of camera images in real time and adds corresponding images. The goal of this technology is to put the virtual world in the real world and interact on the screen.

Augmented Reality, a new technology that seamlessly integrates real-world information with virtual world information, and is an entity that is difficult to experience in a certain time and space of the real world (eg, visual information, Sound, taste, touch, etc.) through computer and other science and technology, simulation and then superimposition, the virtual information is applied to the real world, perceived by human senses, to achieve a sensory experience beyond reality. The real environment and virtual objects are superimposed in real time on the same picture or space.

Augmented reality technology not only displays the information of the real world, but also displays the virtual information at the same time. The two kinds of information complement each other and superimpose. In visual augmented reality, users can use the helmet display to combine the real world with computer graphics to see the real world around it.

In an optional example, the air conditioner can be touch operated in the display screen of the AR air conditioner model.

In an optional example, referring to the example shown in FIG. 11, the air conditioning interaction method based on the AR and the wireless communication technology may be implemented by: after the mobile phone or the tablet device scans successfully, the AR air conditioning model is generated on the screen; Read the touch location information sensed by the mobile phone or the tablet device, and send the command to the device terminal through the wireless connection; the device end executes the received control command, and returns the executed result to the mobile phone or the tablet device through wireless communication again. Synchronize the virtual device with the actual device to ensure that the state of the air conditioner in the screen is consistent with the actual air conditioning state, thereby realizing the interaction process between the person and the product.

In an optional example, the air conditioner can be directly controlled by the remote control interface of the mobile phone APP, but this method is limited to the remote control panel and lacks an intuitive and convenient interactive experience.

Since the processing and functions implemented by the terminal in this embodiment substantially correspond to the foregoing methods shown in FIG. 1 to FIG. 7 or the foregoing embodiments, principles, and examples of the apparatus shown in FIG. 8, the description in this embodiment is not For details, refer to the related description in the foregoing embodiment, and details are not described herein.

After a large number of tests, the technical solution of the present application is used to scan the air conditioner physical object or the identification map through a device with a display screen such as a mobile phone or a tablet, and generate an AR air conditioning model in the display screen to realize the interaction between the user and the air conditioner. Realize the natural interaction of air conditioning equipment.

In summary, those skilled in the art will readily understand that the above advantageous modes can be freely combined and superimposed without conflict.

The above description is only for the embodiments of the present application, and is not intended to limit the present application, and various changes and modifications may be made by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and scope of the present application are intended to be included within the scope of the appended claims.

Claims

An interactive method that includes:

Obtaining an AR device model of the device entity to be interacted with;

Receiving touch location information of a touch operation performed by the user on the AR device model;

A control instruction corresponding to the touch location information is determined, and the control instruction is sent to the device entity.
The method of claim 1, wherein the acquiring the AR device model of the device entity to be interacted comprises:

Receiving image information of the device entity scanned by the scanning module, or the identification image thereof, and/or input by the input module;

Performing AR processing on the image information to generate the AR device model; and/or extracting feature information of the device entity from the image information, and calling corresponding to the feature information from a pre-established information base The AR device model;

and / or,

Receiving, by the scanning module, the device entity or its identification image, and/or the identification information of the device entity input by the input module;

The AR device model corresponding to the identification information is determined according to a correspondence between the setting identification information and the set AR device model.
The method of claim 2, wherein

The feature information includes: an outer contour;

and / or,

The identification information includes at least one of a two-dimensional code, a barcode, an ID code, and an RF tag.
The method of any one of claims 1-3, wherein the method further comprises:

Receiving an execution result of the control instruction by the device entity.
The method of claim 4, wherein receiving an execution result of the control instruction by the device entity comprises:

Receiving an execution result that is actively sent after the control instruction is executed by the device entity.
The method of claim 4 or 5, wherein the method further comprises:

When the execution result is not received, a query request is sent to request the device entity to send the execution result.
The method of claim 6 wherein the method further comprises:

Aborting the query when the consecutively sent query message has not received the response; and/or,

When it is determined that the device entity does not execute the control instruction, the state before the sending of the control instruction is resumed.
The method of any one of claims 4-7, wherein the method further comprises:

Determining a current operating state of the device entity corresponding to the execution result;

The current running state synchronization is virtually displayed into the AR device model to implement synchronous display of the AR device model and the current running state.
A method according to any one of claims 1-8, wherein

The device entity includes at least one of an air conditioner, a water heater, a refrigerator, a washing machine, an induction cooker, an electric heater, an electric oven, a rice cooker, an electric fan, a water purifier, an air purifier, a range hood, and a sterilizer;

and / or,

The touch location information includes: at least one of a top of the AR device model, a bottom, a position of the airframe, and any position other than the body in the display interface where the airframe is located;

and / or,

The control instruction includes at least one of a power on, a power off, an open state, and an operation mode.
An interactive device comprising:

a communication unit, configured to acquire an AR device model of the device entity to be interacted with;

The communication unit is further configured to receive touch location information of a touch operation performed by the user on the AR device model;

And a control unit, configured to determine a control instruction corresponding to the touch location information, and send the control instruction to the device entity by using the communication unit.
The device according to claim 10, wherein the communication unit acquires an AR device model of a device entity to be interacted, including:

Receiving image information of the device entity scanned by the scanning module, or the identification image thereof, and/or input by the input module;

Performing AR processing on the image information to generate the AR device model; and/or extracting feature information of the device entity from the image information, and calling corresponding to the feature information from a pre-established information base The AR device model;

and / or,

Receiving, by the scanning module, the device entity or its identification image, and/or the identification information of the device entity input by the input module;

The AR device model corresponding to the identification information is determined according to a correspondence between the setting identification information and the set AR device model.
The apparatus according to claim 11, wherein

The feature information includes: an outer contour;

and / or,

The identification information includes at least one of a two-dimensional code, a barcode, an ID code, and an RF tag.
The device of any one of claims 10-12, wherein the device further comprises:

The communication unit is further configured to receive an execution result of the control instruction by the device entity.
The device according to claim 13, wherein the communication unit receives the execution result of the control instruction by the device entity, and specifically includes:

Receiving an execution result that is actively sent after the control instruction is executed by the device entity.
The apparatus according to claim 13 or 14, wherein

The control unit is further configured to: when the execution result is not received, send a query request to request the device entity to send the execution result.
The device according to claim 15, wherein

The control unit is further configured to abandon the query when continuously sending the set number of query messages and still not receiving the response; and/or,

When it is determined that the device entity does not execute the control instruction, the state before the sending of the control instruction is resumed.
A device according to any one of claims 13-16, wherein

The control unit is further configured to determine a current running state of the device entity corresponding to the execution result;

The communication unit is further configured to dynamically display the current running state synchronization into the AR device model to implement synchronous display of the AR device model and the current running state.
A device according to any one of claims 10-17, wherein

The device entity includes at least one of an air conditioner, a water heater, a refrigerator, a washing machine, an induction cooker, an electric heater, an electric oven, a rice cooker, an electric fan, a water purifier, an air purifier, a range hood, and a sterilizer;

and / or,

The touch location information includes: at least one of a top of the AR device model, a bottom, a position of the airframe, and any position other than the body in the display interface where the airframe is located;

and / or,

The control instruction includes at least one of a power on, a power off, an open state, and an operation mode.
A storage device, wherein the storage device stores a plurality of instructions; the plurality of instructions are used by a processor to load and execute the interaction method according to any one of claims 1-9.
A terminal, the terminal comprising:

a processor for executing a plurality of instructions;

a memory for storing a plurality of instructions;

The plurality of instructions for storing by the memory, and loading, by the processor, and performing the interaction method according to any one of claims 1-9;

or,

An interactive device according to any of claims 10-18.