WO2020063132A1 - Ar-based interactive programming system and method, and medium and intelligent device - Google Patents

Abstract

Provided are an AR-based interactive programming system and method, and a medium and an intelligent device. The method comprises: presenting an image of a real object and a virtual object in a superimposed manner; acquiring a program instruction set, wherein the program instruction set comprises one or more program instruction units; and instructing the real object to carry out an action according to the program instruction set. The present application improves programming toys in the prior art, involves interaction between an image of a real object and a virtual object, interaction between the real object and an intelligent device, and interaction between the virtual object and a user in a programming process, and is fully combined with AR technology, so that the programming toys are suitable for serving as intelligent companions for children.

Description

AR-based interactive programming system, method, medium and intelligent device Technical field

The present invention relates to the field of toys, and in particular, to an AR-based interactive programming system, method, medium, and smart device.

Background technique

Patent documents CN1267228A and CN1223158A respectively disclose programmable toys, which belong to earlier programming toys. Such a programming toy only stays offline and offline, and cannot be augmented by virtual means.

However, it is worth noting that from the perspective of technological development, patent document CN1223158A is an important research and development foundation for subsequent AR programming toys. Patent document CN1223158A provides users with a series of action buttons, which can be used to program the movements of toys by pressing different buttons successively. The priority date is 1997. At that time, home PCs were not yet popular, and domestic programming skills were mainly concentrated. In colleges and universities for adults; the patent document CN1223158A was able to provide programming toys for children in the 1990s. At the time, it was really a very fun toy that was worth showing off. The focus on children's toys has a strong basis for further improvement.

Later, with the development of computer technology, R & D personnel no longer suffer from the shortcomings of computer devices, and programming toys can more and more match ideas into practical applications. For example, patent document CN101789194A discloses a building block type learning and programming device, which includes a plurality of different types of real volume blocks, each type of block contains corresponding instruction information, and after connecting the block blocks to a single-chip microcomputer, the single-chip microcomputer makes The instruction information in the block outputs the control signal, presenting sound, light, electricity and other effects. However, the interactivity of the programming effect provided by the patent document CN101789194A is still insufficient.

Of course, when people started to pay attention to the impact of not excessive use of computer products on children's vision and social ability, another branch of technology was developed. Under this branch of technology, programming toys are kept as far away from smart technology as possible. For example, patent document CN105396295A discloses a children-oriented space-programmed robot toy, which uses a plurality of instruction labels arranged sequentially, each instruction label has several control instructions, and the robot body sequentially reads the control instructions on each instruction label. Then, the corresponding command actions indicated by the control command are executed in turn. However, anti-addiction software, the use of acceleration sensors to identify and assist in reminding the sitting posture and other technologies have been valued and developed. Therefore, a programming process such as the patent document CN105396295A departs from the technical solution of the smart device and is far from the technical field of AR programming toys.

Currently, with AR technology that has been developed in recent years, AR (Augmented Reality) technology can be used to enhance interactivity and improve it. To this end, researchers have experimented and created enhanced interaction between AR technology and toys.

Patent document CN106730899A discloses a toy control method and system, which adopts AR technology to realize synchronous movement between physical toys and virtual toys. However, the interaction process between the physical toy and the virtual toy in the patent document CN106730899A belongs to synchronous control, only the virtualized control of the physical toy and the physicalized control of the virtual toy are implemented, and the physical toy and the virtual toy are not in the same space. To interact.

Patent document CN105931289A is a more typical technical solution for interacting physical toys with virtual toys in the same space. It discloses a system and method for real models to cover virtual objects, such as realizing virtual dinosaurs to be displayed on display parts , And the effect of being blocked by the real jungle landscape patch, visually the virtual dinosaur can shuttle back and forth between the real jungle landscape patch. Conversely, the effect of another virtual object blocking the solid object can also be achieved accordingly.

In summary, AR programming toys are mainly related to "programming" and "AR". Among them, non-AR programming toys appeared earlier, and then combined with the development of AR. Following this R & D process spanning more than 20 years, it is necessary to consider how to combine AR and programming to improve the interaction effect of programming, so that this interaction provides more companionship in children's learning and growth.

Summary of the Invention

In view of the defects in the prior art, the object of the present invention is to provide an AR-based interactive programming system, method, medium, and smart device.

An AR-based interactive programming system provided according to the present invention includes:

AR module: Overlay and present images of real objects and virtual objects;

The AR-based interactive programming system further includes:

A programming module: obtaining a program instruction set, wherein the program instruction set includes one or more program instruction units;

Execution module: instructs a real object to perform an action according to the program instruction set.

Preferably, it further comprises:

Matching module: presents the relative relationship between the image of the real object and the virtual object.

Preferably, the programming module includes:

A programming unit acquisition module: acquiring a plurality of program instruction units according to the first operation input information;

Timing relationship acquisition module: acquiring timing relationships executed between the plurality of program instruction units according to the second operation input information;

Instruction set generation module: Generates the program instruction set according to the multiple program instruction units and timing relationships.

Preferably, the matching module includes any one or more of the following modules:

Matching size module: presents the size relative relationship between the image of the real object and the virtual object, and makes the size match between the image of the real object and the virtual object through size interaction; wherein the size interaction refers to: reminding the real object The size difference between the image and the virtual object, or instructing the real object to move to change the size of the image;

Matching orientation module: presents the relative relationship between the orientation of the image of the real object and the virtual object, and the orientation interaction makes the orientation of the image of the real object and the virtual object match; wherein the orientation interaction means: prompting the real object The orientation difference between the image and the virtual object, or instructs the real object to rotate to change the orientation of the image.

Preferably, the matching module further includes:

Obstacle elimination module: determine whether there is a conflict between the virtual object and the actual obstacle; if there is no conflict, confirm that there is no conflict; if there is a conflict, then:

-Prompt conflict;

-Instruct the real object to move so that the virtual object does not conflict with the real obstacle; or

-Update virtual objects so that they do not conflict with real obstacles.

Preferably, the virtual object includes a virtual venue, and the virtual venue is presented in any of the following ways:

-Presentation according to preset parameters;

-According to the relative relationship between the image of the real object and the virtual object;

-After the image of the real object matches the virtual object, it is rendered according to the real object.

Preferably, it further comprises:

Virtual interaction response module: presents a virtual interaction response of the virtual object to the image of the real object according to the action performed by the real object.

Preferably, it further comprises:

Real interaction response module: instructs the real object to perform a real interaction response according to the virtual interaction response.

Preferably, it further comprises:

Programming interface module: provides a visual programming interface; wherein, the programming interface module includes any one or more of the following modules:

-Overlay presentation module: Make the images of the real object and the virtual object overlay and present in the visual programming interface;

-Operation presentation module: a graphical program instruction unit, in the visual programming interface, presents a corresponding visual effect with the user's operation;

-An execution presentation module: the program instruction set is executed step by step to execute the program instruction unit, and the actual object is executed step by step in the visual programming interface;

-Omit presentation module: According to the user's designation of the program instruction unit, instruct the real object to directly respond to the corresponding action after the designated program instruction unit is executed, and present it in the visual programming interface.

Preferably, the programming interface module further includes any one or more of the following modules:

A first interface switching module: trigger switching from the operation presentation module to the execution presentation module according to the third operation input information;

The second interface switching module: triggers switching from the execution presentation module to the operation presentation module according to a virtual interaction response of the virtual object to the image of the real object.

An AR-based interactive programming method provided according to the present invention includes:

AR step: superimposing and presenting images of real objects and virtual objects;

The AR-based interactive programming method further includes:

Programming step: obtaining a program instruction set, wherein the program instruction set includes one or more program instruction units;

Execution step: According to the program instruction set, instruct a real object to perform an action.

Preferably, it further comprises:

Matching step: present the relative relationship between the image of the real object and the virtual object.

Preferably, the programming step includes:

Step of acquiring a programming unit: acquiring a plurality of program instruction units according to the first operation input information;

Step of acquiring timing relationships: acquiring timing relationships executed between the plurality of program instruction units according to the second operation input information;

Instruction set generation step: generating the program instruction set according to the multiple program instruction units and a timing relationship.

Preferably, the matching step includes any one or more of the following steps:

Step of matching size: presenting the relative size relationship between the image of the real object and the virtual object, and matching the size between the image of the real object and the virtual object through size interaction; wherein the size interaction refers to: prompting the real object The size difference between the image and the virtual object, or instructing the real object to move to change the size of the image;

Step of matching orientation: presenting the relative relationship between the orientation of the image of the real object and the virtual object, and the orientation interaction makes the orientation of the image of the real object and the virtual object match; wherein the orientation interaction refers to: prompting the real object The orientation difference between the image and the virtual object, or instructs the real object to rotate to change the orientation of the image.

Preferably, the matching step further includes:

Obstacle removal steps: determine whether there is a conflict between the virtual object and the actual obstacle; if there is no conflict, confirm that there is no conflict; if there is a conflict, then:

-Prompt conflict;

-Instruct the real object to move so that the virtual object does not conflict with the real obstacle; or

-Update virtual objects so that they do not conflict with real obstacles.

Preferably, the virtual object includes a virtual venue, and the virtual venue is presented in any of the following ways:

-Presentation according to preset parameters;

-According to the relative relationship between the image of the real object and the virtual object;

-After the image of the real object matches the virtual object, it is rendered according to the real object.

Preferably, it further comprises:

Virtual interaction response step: presenting a virtual interaction response of the virtual object to the image of the real object according to the action performed by the real object.

Preferably, it further comprises:

Real interaction response step: instructing the real object to perform a real interaction response according to the virtual interaction response.

Preferably, it further comprises:

Programming interface steps: Provide a visual programming interface; wherein the programming interface steps include any one or more of the following steps:

-Overlay presentation step: making the image of the real object and the virtual object overlay and present in the visual programming interface;

-Operation presentation step: making a graphical program instruction unit, in the visual programming interface, present a corresponding visual effect with the user's operation;

-Executing the presenting step: the case where the program instruction set executes the program instruction unit step by step, the real object executes the action step by step, and is displayed synchronously in the visual programming interface;

-Omit the presenting step: according to the user's designation of the program instruction unit, instruct the real object to directly respond to the corresponding action after the designated program instruction unit is executed, and present it in the visual programming interface.

Preferably, the programming interface step further includes any one or more of the following steps:

First interface switching step: trigger switching from the operation presentation step to the execution presentation step according to the third operation input information;

The second interface switching step: trigger switching from the execution presentation step to the operation presentation step according to a virtual interaction response of the virtual object to the image of the real object.

According to the present invention, there is provided a computer-readable storage medium storing a computer program, and the computer program implements the steps of the foregoing method when executed by a processor.

An intelligent device provided according to the present invention includes the above-mentioned AR-based interactive programming system, or the above-mentioned computer-readable storage medium storing a computer program.

Compared with the prior art, the present invention has the following beneficial effects:

The invention improves the programming toy in the prior art, and involves the interaction between the image of the real object and the virtual object, the interaction between the real object and the smart device, and the interaction between the virtual object and the user during the programming process. Fully combined with AR technology, making programming toys suitable for children's intelligent companionship.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, objects, and advantages of the present invention will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

FIG. 1 is a schematic diagram of an adjustment interface of a relative relationship between a size of an image of a real object and a size of a virtual object.

FIG. 2 is a schematic diagram of an adjustment interface for the relative relationship between the orientation of an image of a real object and a virtual object.

FIG. 3 is a schematic diagram of an interface in which an image of a real object and a grid road in a virtual object are superimposed.

Figure 4 is a schematic diagram of a visual programming interface.

FIG. 5 is a schematic diagram of an interface for executing a first program instruction in a visual programming interface.

FIG. 6 is an interface schematic diagram of executing the second step program instruction in the visual programming interface.

FIG. 7 is a flowchart of method steps in a specific scenario embodiment.

FIG. 8 is a schematic diagram of a structural framework in a specific scenario embodiment.

The figure shows:

Real object image 100

Size comparison object 201

Arrow icon 202

Virtual venue 203

detailed description

The present invention is described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, several changes and improvements can be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

Companionship is an indispensable emotional link in the growth process of children, including parent-to-child companionship, and toy-to-child companionship. More importantly, parents and children can have the opportunity to play games and learn together through toys to achieve More advanced smart companions, and programming toys are one of the best carriers. To this end, the inventors carried out technical improvements under the vision and recognition of intelligent companionship, and carried out graphic expressions with the assistance of patent engineers, patent agents, and patent attorneys, in order to obtain patent rights and enable subsequent development of more Many products accompany children at home. The technical solutions and specific application scenarios of the present application are described below.

An AR-based interactive programming system provided according to the present invention includes:

AR module: Overlay and present the image of the real object and the virtual object; the image of the real object is obtained by capturing the real picture from the camera device, such as the camera of a smart phone. The real picture includes the entire picture collected by the camera device. Contains real objects such as stairs, sofas, building blocks, robots, carpets, murals, food, pets, and people. The graphics corresponding to the recognition target in the entire screen are selected as the real object images. In a preferred example, a robot that can perform actions is used as the recognition. The target, the image of the robot constitutes the image of the real object; in other preferred examples, people or animals can be used as recognition targets. As long as the person or animal can understand the instructions, the corresponding actions can also be performed after receiving the instructions. At this time, the instruction is preferably a voice instruction, and a sub-optimal instruction may also be a graphic instruction, an odor instruction, or the like, which is suitable for a situation in which a person with a sensory impairment or an animal needs perceptual assistance.

Programming interface module: Provides a visual programming interface; the visual programming interface is mainly composed of one or more sub-interfaces, and multiple sub-interfaces can be switched, displayed, or displayed side-by-side; when displayed side by side, visually the sub-interfaces Corresponds to a display area in the visual programming interface; at least one sub-interface is used as a presentation space, and at the same time, the image of the real object and the virtual object are presented, and the image of the real object and the virtual object are superimposed and presented. Only the image and virtual object of the real object are presented. Of course, in the variation, in addition to the image and virtual object of the real object, the sub-interface can also present the real picture of the real object at the same time. Carpets and other things around real objects. Among them, the virtual object can cover a part of the real picture in which the real object is located, and the image of the real object can cover the virtual object. Those skilled in the art can refer to the blocking technology of the virtual dinosaur and the real jungle landscape plug-in board in patent document CN105931289A The overlay is implemented in an overlay manner, which will not be repeated here.

Programming module: obtaining a program instruction set, wherein the program instruction set includes one or more program instruction units; wherein the type of the program instruction unit may be a logic type instruction or an execution type instruction; the execution type The instruction instructs the real object to perform an action.

Execution module: instructs a real object to perform an action according to the program instruction set; wherein the action may be various motions, such as translation, rotation, beating, and deformation; the action may also be sound, light, and electrical effects, For example, vocalization, light emission, color change, temperature change, phase change. In the embodiment where the real object is a human, an animal, or an artificial intelligence, the action may be an action that can be achieved by this type of real object, such as laughing, bending , Picking up items, turning on household appliances, etc.

Matching module: presents the relative relationship between the image of the real object and the virtual object. Since the image of the real object and the virtual object need to be superimposed in the same space, and the image of the real object and the virtual object are required to interact, it is necessary to match the two with parameters such as size. For example, the real object is a robot and the virtual object is a grid. The influence of the robot needs to move between different grids. If the robot is too far away from the camera at this time, the robot needs to move a long distance to move from a grid. The next grid is not suitable for game effects, so it is necessary to present the relative relationship between the image of the real object and the virtual object before the virtual interactive response, the real interactive response, and the execution action, and match the relative relationship. Adjustment.

Virtual interaction response module: presents a virtual interaction response of the virtual object to the image of the real object according to the action performed by the real object; for example, the virtual interaction response may be eating a virtual cake, brightening the grid, and saving a virtual animal. Way, you can remind the similarities and differences between the programming results and the expected structure.

Real interaction response module: According to the virtual interaction response, instruct the real object to perform a real interaction response; the real interaction response may be that the instructed robot makes a circle in place, shakes, changes expression, and emits different sounds.

The invention also provides a smart device, which includes the AR-based interactive programming system. The smart device may be a smart phone, a tablet computer, a smart watch, smart glasses, a projection device, a VR helmet, and other devices. Taking a smart phone as an example, the image of a real object is captured by a camera of the smart phone. The screen of the smart phone overlays and displays the image of the real object and the virtual object, and presents the programming interface module, and receives the user's visual programming interface through the touch screen. In the operation, the short-range wireless communication module of the smart phone or the WIFI network is used to send instructions to the real object, the voice control instruction is played through the smart phone's sound module, and the light control instruction is issued through the smart phone's lighting.

In the following, through further preferred examples, the AR-based interactive programming system and smart device provided by the present invention are specifically described.

The programming module includes:

Programming unit acquisition module: Acquire multiple program instruction units according to the first operation input information; display graphical program instruction units in a visual programming interface, such as graphically shaped as a jigsaw puzzle; users use the program instruction units Performing the first operation input information to obtain a plurality of program instruction units, wherein the first operation input information includes input information generated by clicking, voice control selection, gesture control, eye selection, and peripheral selection. Peripherals can be mouse, electronic pen and other external devices.

Timing relationship acquisition module: Acquire a timing relationship between the multiple program instruction units according to the second operation input information. The second operation input information is used to set the execution timing between the obtained multiple program instruction units. It is a single execution or a cyclic execution, and the timing relationship is mainly determined by the second operation input information and / or the logic of the program instruction unit itself. The second operation input information may be a user dragging a program instruction unit after selecting it on the touch screen, or setting an execution sequence number of each program instruction unit. In a preferred example, according to the function of the touch screen, the first operation input information and the second operation input information may be generated by the same operation of the user, for example, the user edits a graphical program instruction unit from visual editing Drag one area of the interface to another area.

Instruction set generation module: Generates the program instruction set according to the multiple program instruction units and timing relationships. The visual effect of the program instruction set on the visual editing interface may be the orderly arrangement of multiple graphical program instruction units at different positions, and the code in the program instruction set is preferably not presented visually.

The matching module includes any one or more of the following modules:

Matching size module: presents the size relative relationship between the image of the real object and the virtual object, and makes the size match between the image of the real object and the virtual object through size interaction; wherein the size interaction refers to: reminding the real object The size difference between the image and the virtual object, or instructing a real object to move to change the size of the image; the virtual object includes a size comparison object between the shape of the size comparison object and the image of the real object Match, for example, both are circular, as shown in Figure 1, or both are circular and square, respectively. If the size contrast object is exactly enveloped, basically includes or contains images of real objects, then both are considered The sizes are matched, otherwise, the sizes are considered not to match, so that the relative relationship between the sizes is realized, reflecting the size difference. In a variation, when the size difference between the image of the real object and the virtual object is large and does not match, the real object may be instructed to move to match the size relationship; if the image of the real object is too small, it indicates that the real object is close to the camera ; If the image of the real object is too large, it indicates that the real object is far from the camera; or, if it is recognized that the real object does not all enter the screen, it can instruct the real object to all enter the camera's frame acquisition angle.

Matching orientation module: presents the relative relationship between the orientation of the image of the real object and the virtual object, and the orientation interaction makes the orientation of the image of the real object and the virtual object match; wherein the orientation interaction means: prompting the real object The orientation difference between the image and the virtual object, or instructs the real object to rotate to change the orientation of the image; the virtual object includes a contrast-oriented object. After the real object is identified, the orientation of the real object can be identified, and Icons such as arrows indicate that, as shown in FIG. 2, for example, the orientation of a toy car can be set to the front direction of the toy car; the visually programming interface presents both the orientation-oriented object and the orientation icon of the real object at the same time, so that The user may automatically recognize the difference in orientation between the two. After obtaining the difference in orientation, the user can operate to change the orientation of the virtual object or instruct the real object to rotate to reduce or eliminate the difference in orientation.

Obstacle elimination module: determine whether the virtual object conflicts with the real obstacle. When virtual objects are functional for programming, conflicts between real obstacles and virtual objects may occur. For example, if an image of a virtual object robot that is a real object needs to move along a virtual road, the virtual road cannot overlap with a real obstacle. As another example, the extension of a virtual road is overlapped by a wall or a sofa. If there is no conflict, confirm that there is no conflict; if there is a conflict, then: prompt a conflict; instruct the real object to move so that the virtual object does not conflict with the real obstacle; or, update the virtual object so that the virtual object does not conflict with the real obstacle .

The virtual object includes a virtual venue. For example, virtual sites are grids, roads, bridges, watercourses, stadiums, cities, grasslands, mountains and rivers, tracks, and so on. The virtual venue is presented in any of the following ways: based on preset parameters; based on the relative relationship between the image of the real object and the virtual object; and after the image of the real object matches the virtual object, it is based on the real object. The presenting according to the preset parameters may specifically generate and present the virtual venue according to the preset parameters without considering the relative relationship between the image of the real object and the virtual object. The presentation is based on the relative relationship between the image of the real object and the virtual object. Specifically, the virtual object is correspondingly presented according to the image of the real object; for example, if the image of the real object is large, the grid in the virtual venue is also The corresponding area is larger; if the image of the real object is smaller, the grid in the virtual site is also smaller. In a preferred example, the motion parameters of the real object are protected. For example, the step distance of the real object of the robot is constant. When the grid size in the virtual field is adapted to the relative relationship, the real object of the robot moves one step, which is almost the same. Or just move from one grid to another. After the image of the real object matches the virtual object, the real object is presented; specifically, the position of the real object is recognized in real time, indicating that the action of the real object is adapted to the virtual venue. For example, instruct the real object to move so that the real object moves exactly from one grid to another.

The programming interface module includes any one or more of the following modules:

Overlay presentation module: Make the image of the real object and the virtual object overlay and present in the visual programming interface; for example, as shown in FIG. 1. Operation presentation module: Make a graphical program instruction unit, in the visual programming interface, present a corresponding visual effect with the user's operation; for example, if the user drags the program instruction unit, the program instruction unit is being dragged It is highlighted during the movement and moves to the end position of the drag track. The execution presentation module: the program instruction set is executed step by step to execute the program instruction unit and the real object is executed step by step in the visual programming interface, and displayed synchronously; as shown in FIG. 5 and FIG. 6. Omit presentation module: According to the user's designation of the program instruction unit, instruct the real object to directly respond to the corresponding action after the designated program instruction unit is executed, and present it in the visual programming interface. For example, if there are 5 consecutive program instruction units that turn 90 degrees, if the user clicks on the debug and executes the fifth program instruction unit that turns 90 degrees, the real object will only turn 90 degrees, instead of 360 degrees + 90 degrees, Save time in programming proofreading.

The programming interface module further includes any one or more of the following modules:

First interface switching module: trigger switching from the operation presentation module to the execution presentation module according to the third operation input information; for example, switching from FIG. 4 to the interface shown in FIG. 5. The second interface switching module: trigger switching from the execution presentation module to the operation presentation module according to a virtual interaction response of the virtual object to the image of the real object; for example, switching from FIG. 5 to the interface shown in FIG. 4.

In the following, the present invention will be described in more detail with reference to the accompanying drawings of the description through preferred examples of specific application scenarios.

As shown in FIG. 1, the real object includes a robot, and FIG. 1 shows an image 100 of the real object. The robot has a spherical structure, and wheels are installed at the bottom, which can move on its own, including translation and rotation. The user turns on the mobile phone, and continuously shoots the robot with the mobile phone. FIG. 1 is a schematic diagram showing a realistic picture on the screen of the mobile phone.

The steps shown in combination with FIG. 7. First, the relative distance relationship between the robot and the mobile phone needs to be adjusted, that is, the virtual object, especially the distance relationship between the virtual field and the robot. The virtual field is not shown in FIG. 1, but is shown in FIG. 3. This is because the virtual field needs to be generated according to the orientation of the robot. The size comparison object 201 in FIG. 1 is similar to the size and size of the robot image, and basically forms an envelope. It is considered that the distance between the robot and the mobile phone is appropriate at this time.

Then, as shown in FIG. 2, an arrow icon 202 is displayed on the mobile phone interface, and the user rotates the arrow icon so that the arrow icon is aligned with the front direction of the robot.

Through the operations of FIG. 1 and FIG. 2, the relative relationship between the robot and the mobile phone has been properly determined. On this basis, as shown in FIG. 3, a virtual venue 203 is generated. The user needs to look at the grid roads in the virtual field and move forward along the grid road one by one. Of course, this requires the user to obtain the instruction program instruction set programmatically, and use the program instruction set to instruct the robot to move along the grid road.

Figure 4 shows the visual programming interface. The left area of the visual programming interface is a graphical program instruction unit. There are four types of program instruction units, which can instruct the robot to perform actions: forward one frame, backward One division, 90 degrees to the right, 90 degrees to the left. The right area of the visual programming interface is the program instruction units that the user has selected and sorted, that is, the six program instruction units in the right area mainly constitute the program instruction set. According to the program instruction set movement, the robot will follow the road from one end of the grid road to the other.

If you want to debug the program during the programming process, you can enter the visual programming interface shown in Figure 5 and Figure 6, so that the program instruction set can execute the program instruction unit step by step, and the real object step by step actions. , In the visual programming interface, synchronous display. For example, in Figure 5, a solid dot is marked on the graphic of the program instruction unit in the first step, which indicates that the program instruction unit is being executed. Accordingly, the robot starts from the first grid on the network road to the second. The Internet advances one space. After that, as shown in FIG. 6, the solid dots are marked on the graphic of the program instruction unit in the second step, which indicates that the program instruction unit is being executed. Accordingly, the robot starts from the second grid of the network road to the The three networks advance one grid.

The following describes an AR-based interactive programming method provided by the present invention. Those skilled in the art can implement the AR-based interactive programming system by referring to the steps and procedures in the AR-based interactive programming. That is, the AR-based interactive programming method can be understood as a preferred example of the AR-based interactive programming system.

An AR-based interactive programming method provided according to the present invention includes:

AR step: superimposing and presenting images of real objects and virtual objects;

The AR-based interactive programming method further includes:

Programming step: obtaining a program instruction set, wherein the program instruction set includes one or more program instruction units;

Execution step: According to the program instruction set, instruct a real object to perform an action.

Preferably, it further comprises:

Matching step: present the relative relationship between the image of the real object and the virtual object.

Preferably, the programming step includes:

Step of acquiring a programming unit: acquiring a plurality of program instruction units according to the first operation input information;

Step of acquiring timing relationships: acquiring timing relationships executed between the plurality of program instruction units according to the second operation input information;

Instruction set generation step: generating the program instruction set according to the multiple program instruction units and a timing relationship.

Preferably, the matching step includes any one or more of the following steps:

Step of matching size: presenting the relative size relationship between the image of the real object and the virtual object, and matching the size between the image of the real object and the virtual object through size interaction; wherein the size interaction refers to: prompting the real object The size difference between the image and the virtual object, or instructing the real object to move to change the size of the image;

Step of matching orientation: presenting the relative relationship between the orientation of the image of the real object and the virtual object, and the orientation interaction makes the orientation of the image of the real object and the virtual object match; wherein the orientation interaction refers to: prompting the real object The orientation difference between the image and the virtual object, or instructs the real object to rotate to change the orientation of the image.

Preferably, the matching step further includes:

Obstacle removal steps: determine whether there is a conflict between the virtual object and the actual obstacle; if there is no conflict, confirm that there is no conflict; if there is a conflict, then:

-Prompt conflict;

-Instruct the real object to move so that the virtual object does not conflict with the real obstacle; or

-Update virtual objects so that they do not conflict with real obstacles.

Preferably, the virtual object includes a virtual venue, and the virtual venue is presented in any of the following ways:

-Presentation according to preset parameters;

-According to the relative relationship between the image of the real object and the virtual object;

-After the image of the real object matches the virtual object, it is rendered according to the real object.

Preferably, it further comprises:

Virtual interaction response step: presenting a virtual interaction response of the virtual object to the image of the real object according to the action performed by the real object.

Preferably, it further comprises:

Real interaction response step: instructing the real object to perform a real interaction response according to the virtual interaction response.

Preferably, it further comprises:

Programming interface steps: Provide a visual programming interface; wherein the programming interface steps include any one or more of the following steps:

-Overlay presentation step: making the image of the real object and the virtual object overlay and present in the visual programming interface;

-Operation presentation step: making a graphical program instruction unit, in the visual programming interface, present a corresponding visual effect with the user's operation;

-Executing the presenting step: the case where the program instruction set executes the program instruction unit step by step, the real object executes the action step by step, and is displayed synchronously in the visual programming interface;

-Omit the presenting step: according to the user's designation of the program instruction unit, instruct the real object to directly respond to the corresponding action after the designated program instruction unit is executed, and present it in the visual programming interface.

Preferably, the programming interface step further includes any one or more of the following steps:

First interface switching step: trigger switching from the operation presentation step to the execution presentation step according to the third operation input information;

The second interface switching step: trigger switching from the execution presentation step to the operation presentation step according to a virtual interaction response of the virtual object to the image of the real object.

According to the present invention, there is provided a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, the steps of the AR-based interactive programming method are implemented. Examples are chips, memories, and optical discs. Especially servers that support APP stores. A smart device provided according to the present invention includes the computer-readable storage medium storing the computer program.

Those skilled in the art know that, in addition to implementing the system, device and each module provided by the present invention in a pure computer-readable program code manner, the system, device and each module provided by the present invention can be made by logically programming the system module. The same programs are implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system, device, and its various modules provided by the present invention can be considered as a hardware component, and the modules included in it for implementing various programs can also be considered as the structure within the hardware component; Modules for implementing various functions are considered to be both software programs that implement the system and structures within hardware components.

The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the case of no conflict, the embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other.

Claims (22)

1. An AR-based interactive programming system is characterized in that it includes:

   AR module: Overlay and present images of real objects and virtual objects;

   The AR-based interactive programming system further includes:

   A programming module: obtaining a program instruction set, wherein the program instruction set includes one or more program instruction units;

   Execution module: instructs a real object to perform an action according to the program instruction set.
2. The AR-based interactive programming system according to claim 1, further comprising:

   Matching module: presents the relative relationship between the image of the real object and the virtual object.
3. The AR-based interactive programming system according to claim 1, wherein the programming module comprises:

   A programming unit acquisition module: acquiring a plurality of program instruction units according to the first operation input information;

   Timing relationship acquisition module: acquiring timing relationships executed between the plurality of program instruction units according to the second operation input information;

   Instruction set generation module: Generates the program instruction set according to the multiple program instruction units and timing relationships.
4. The AR-based interactive programming system according to claim 2, wherein the matching module comprises any one or more of the following modules:

   Matching size module: presents the size relative relationship between the image of the real object and the virtual object, and makes the size match between the image of the real object and the virtual object through size interaction; wherein the size interaction refers to: reminding the real object The size difference between the image and the virtual object, or instructing the real object to move to change the size of the image;

   Matching orientation module: presents the relative relationship between the orientation of the image of the real object and the virtual object, and the orientation interaction makes the orientation of the image of the real object and the virtual object match; wherein the orientation interaction means: prompting the real object The orientation difference between the image and the virtual object, or instructs the real object to rotate to change the orientation of the image.
5. The AR-based interactive programming system according to claim 4, wherein the matching module further comprises:

   Obstacle elimination module: determine whether there is a conflict between the virtual object and the actual obstacle; if there is no conflict, confirm that there is no conflict; if there is a conflict, then:

   -Prompt conflict;

   -Instruct the real object to move so that the virtual object does not conflict with the real obstacle; or

   -Update virtual objects so that they do not conflict with real obstacles.
6. The AR-based interactive programming system according to claim 1, wherein the virtual object comprises a virtual venue, and the virtual venue is presented in any of the following ways:

   -Presentation according to preset parameters;

   -According to the relative relationship between the image of the real object and the virtual object;

   -After the image of the real object matches the virtual object, it is rendered according to the real object.
7. The AR-based interactive programming system according to claim 1, further comprising:

   Virtual interaction response module: presents a virtual interaction response of the virtual object to the image of the real object according to the action performed by the real object.
8. The AR-based interactive programming system according to claim 7, further comprising:

   Real interaction response module: instructs the real object to perform a real interaction response according to the virtual interaction response.
9. The AR-based interactive programming system according to claim 1, further comprising:

   Programming interface module: provides a visual programming interface; wherein, the programming interface module includes any one or more of the following modules:

   -Overlay presentation module: Make the images of the real object and the virtual object overlay and present in the visual programming interface;

   -Operation presentation module: a graphical program instruction unit, in the visual programming interface, presents a corresponding visual effect with the user's operation;

   -An execution presentation module: the program instruction set is executed step by step to execute the program instruction unit, and the actual object is executed step by step in the visual programming interface;

   -Omit presentation module: According to the user's designation of the program instruction unit, instruct the real object to directly respond to the corresponding action after the designated program instruction unit is executed, and present it in the visual programming interface.
10. The AR-based interactive programming system according to claim 9, wherein the programming interface module further comprises any one or any of the following modules:

    A first interface switching module: trigger switching from the operation presentation module to the execution presentation module according to the third operation input information;

    The second interface switching module: triggers switching from the execution presentation module to the operation presentation module according to a virtual interaction response of the virtual object to the image of the real object.
11. A method for interactive programming based on AR, comprising:

    AR step: superimposing and presenting images of real objects and virtual objects;

    The AR-based interactive programming method further includes:

    Programming step: obtaining a program instruction set, wherein the program instruction set includes one or more program instruction units;

    Execution step: According to the program instruction set, instruct a real object to perform an action.
12. The AR-based interactive programming method according to claim 1, further comprising:

    Matching step: present the relative relationship between the image of the real object and the virtual object.
13. The AR-based interactive programming method according to claim 11, wherein the programming step comprises:

    Step of acquiring a programming unit: acquiring a plurality of program instruction units according to the first operation input information;

    Step of acquiring timing relationships: acquiring timing relationships executed between the plurality of program instruction units according to the second operation input information;

    Instruction set generation step: generating the program instruction set according to the multiple program instruction units and a timing relationship.
14. The AR-based interactive programming method according to claim 12, wherein the matching step comprises any one or more of the following steps:

    Step of matching size: presenting the relative size relationship between the image of the real object and the virtual object, and matching the size between the image of the real object and the virtual object through size interaction; wherein the size interaction refers to: prompting the real object The size difference between the image and the virtual object, or instructing the real object to move to change the size of the image;

    Step of matching orientation: presenting the relative relationship between the orientation of the image of the real object and the virtual object, and the orientation interaction makes the orientation of the image of the real object and the virtual object match; wherein the orientation interaction refers to: prompting the real object The orientation difference between the image and the virtual object, or instructs the real object to rotate to change the orientation of the image.
15. The AR-based interactive programming method according to claim 14, wherein the matching step further comprises:

    Obstacle removal steps: determine whether there is a conflict between the virtual object and the actual obstacle; if there is no conflict, confirm that there is no conflict; if there is a conflict, then:

    -Prompt conflict;

    -Instruct the real object to move so that the virtual object does not conflict with the real obstacle; or

    -Update virtual objects so that they do not conflict with real obstacles.
16. The AR-based interactive programming method according to claim 11, wherein the virtual object comprises a virtual venue, and the virtual venue is presented in any of the following ways:

    -Presentation according to preset parameters;

    -According to the relative relationship between the image of the real object and the virtual object;

    -After the image of the real object matches the virtual object, it is rendered according to the real object.
17. The AR-based interactive programming method according to claim 11, further comprising:

    Virtual interaction response step: presenting a virtual interaction response of the virtual object to the image of the real object according to the action performed by the real object.
18. The AR-based interactive programming method according to claim 17, further comprising:

    Real interaction response step: instructing the real object to perform a real interaction response according to the virtual interaction response.
19. The AR-based interactive programming method according to claim 11, further comprising:

    Programming interface steps: Provide a visual programming interface; wherein the programming interface steps include any one or more of the following steps:

    -Overlay presentation step: making the image of the real object and the virtual object overlay and present in the visual programming interface;

    -Operation presentation step: making a graphical program instruction unit, in the visual programming interface, present a corresponding visual effect with the user's operation;

    -Executing the presenting step: the case where the program instruction set executes the program instruction unit step by step, the real object executes the action step by step, and is displayed synchronously in the visual programming interface;

    -Omit the presenting step: according to the user's designation of the program instruction unit, instruct the real object to directly respond to the corresponding action after the designated program instruction unit is executed, and present it in the visual programming interface.
20. The AR-based interactive programming method according to claim 19, wherein the programming interface step further comprises any one or more of the following steps:

    First interface switching step: trigger switching from the operation presentation step to the execution presentation step according to the third operation input information;

    The second interface switching step: trigger switching from the execution presentation step to the operation presentation step according to a virtual interaction response of the virtual object to the image of the real object.
21. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 11 to 20 are implemented.
22. An intelligent device, comprising the AR-based interactive programming system according to any one of claims 1 to 10, or the computer-readable storage medium storing a computer program according to claim 21.

Images