KR102437242B1 - Method, computer program and system for controlling and programming hardware by using a conversational interface - Google Patents

Abstract

The present invention relates to a physical computing method, computer program, and system for controlling and programming hardware through conversation, and more particularly, to a method of voice conversation between a user and a system, including designing, testing, and debugging of hardware. It relates to controls and physical computing methods, computer programs and systems that enable programming of such controls.
According to the present invention, when a general-purpose prototype circuit is manufactured, additional information on a user's request for circuit assembly is provided or the assembly step can be interactively performed, and the debugging process through testing of the assembled circuit is also By making it possible to perform interactively, it enables easy and accurate fabrication of prototype circuits. In addition, by using the system of the present invention, when a user writes a program code for a circuit test, etc., the user can write and execute the program code in real time through an interactive method, thereby enabling easy and fast debugging.

Description

Physical computing method, computer program and system for controlling and programming hardware through conversation

The present invention relates to a physical computing method, computer program, and system for controlling and programming hardware through conversation, and more particularly, to a method of voice conversation between a user and a system, including designing, testing, and debugging of hardware. It relates to controls and physical computing methods, computer programs and systems that enable programming of such controls.

Physical computing is a complex, yet tightly coupled, task, such as programming and assembling circuits. Previous studies clearly show that this combination is a major cause of mistakes that result in failures while taking up a significant portion of the debugging time in prototyping for beginners. Conventional studies have suggested a system that simplifies such prototyping or introduces a new debugging function, but such a system limits the tasks that a user can perform or is too complicated for beginners.

Fraser Anderson, Tovi Grossman, and George Fitzmaurice. 2017. Trigger-Action-Circuits: Leveraging Generative Design to Enable Novices to Design and Build Circuitry. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST ’17). ACM, New York, NY, USA, 331-.342. https://doi.org/10.1145/3126594.3126637

The present invention was devised to solve such a problem, and when manufacturing a general-purpose prototype circuit, it provides additional information about a user's request for circuit assembly or enables the assembly step to be interactively performed, and the primary assembly The purpose of this is to enable easy and accurate prototype circuit fabrication by making it possible to interactively perform the debugging process through circuit testing.

In addition, by using the system of the present invention, when a user writes a program code for a circuit test, etc., another purpose is to enable easy and fast debugging by enabling the user to write and execute such program code in real time through a conversational method. have.

In order to achieve the above object, a method for controlling the execution of commands by a user's voice by a command execution control apparatus of a physical computing system that performs hardware control and programming through a conversation according to the present invention, (a) a user command periodically accessing the server or, if a command file server exists separately, periodically accessing the command file server to check whether there is a change due to the update of the command file; (b) when there is a change in the command file, importing new command data added to the command file; (c) transmitting image data for displaying a status according to the execution of the command data to the display monitor; (d1) providing a graphical interface for setting new parts according to a user's request; and (d2) according to the user's input of setting contents and clicking of the save button, a definition file (hereinafter referred to as 'part file') including the setting contents of the part is newly stored or the Definitions are added and updated and stored.

delete

delete

delete

delete

When the command of the command data newly added in step (b) is a command for generating a state change in a circuit implemented on a breadboard by using a command execution device, after step (b), (b11) converting the command data into command execution code in a format executed by the command execution device; and, (b12) transmitting the command execution code to the command execution device.

delete

After the step (b), the method may further include (b21) updating and storing the command data in a log file.

when a user's request input is received, (e1) storing the entire program being operated in the command execution control device as one executable code file in the command execution control device; and, (e2) uploading the stored executable code file to a command execution device.

According to another aspect of the present invention, in order to control execution of commands by a user's voice, a computer program operating in a command execution control device of a physical computing system that performs hardware control and programming through a conversation is stored in non-transitory storage. It is stored in the medium, and the processor (a) periodically accesses the user command server or, if the command file server exists separately, periodically accesses the command file server to check whether there is a change due to the command file update. to do; (b) when there is a change in the command file, importing command data newly added to the command file; (c) displaying image data for displaying a status according to the execution of the command data on a display monitor transmitting to a monitor; (d1) providing a graphical interface for setting new parts according to a user's request; and (d2) according to the user's input of setting contents and clicking of the save button, a definition file (hereinafter referred to as 'part file') including the setting contents of the part is newly stored or the Contains instructions that cause a step to be executed where the definition is added and updated.

When the command of the command data newly added in step (b) is a command for generating a state change in a circuit implemented on a breadboard by using a command execution device, after step (b), (b11) converting the command data into command execution code in a format executed by the command execution device; and, (b12) transmitting the command execution code to the command execution device.

According to another aspect of the present invention, an apparatus for controlling execution of a command by a user's voice includes: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions stored in the at least one memory are periodically accessed by the at least one processor (a) to a user command server. or, if the command file server exists separately, periodically accessing the command file server to check whether there is a change due to the command file update; (b) when there is a change in the command file, importing new command data added to the command file; (c) transmitting image data for displaying a status according to the execution of the command data to the display monitor; (d1) providing a graphical interface for setting new parts according to a user's request; and (d2) according to the user's input of setting contents and clicking of the save button, a definition file (hereinafter referred to as 'part file') including the setting contents of the part is newly stored or the Definitions are added to cause the update-saved steps to be executed.

When the command of the command data newly added in step (b) is a command for generating a state change in a circuit implemented on a breadboard by using a command execution device, after step (b), (b11) converting the command data into command execution code in a format executed by the command execution device; and (b12) transmitting the command execution code to the command execution device.

According to another aspect of the present invention, a physical computing system that performs hardware control and programming through conversation receives a command converted from a user's voice command into text from a voice input/output device, and receives the command from the received command. a user command server that generates command data; Changes in the command file including the command of the command data are periodically checked, and when there is a change in the command file, command data newly added to the command file is imported, and a status according to the execution of the command data is displayed. Transmitting image data to be displayed on the monitor to the display monitor, providing a graphic interface for setting new parts according to the user's request, including the setting contents of the parts according to the user's input of setting contents and clicking the save button a command execution control device in which a definition file (hereinafter referred to as a 'part file') is newly stored, or a definition of the part is added to an existing part file and updated and stored; and a display monitor configured to display the image data received from the command execution control device.

The user command server further includes a function of generating or updating the command data as a command file and storing it; or a command file server that generates or updates the command data as a command file and stores it, wherein the command execution control device periodically checks for changes in the command file. In the case of including a function of generating or updating and saving a command file, the command file of the user command server is periodically checked, and when the command file server exists, the command file of the command file server can be checked periodically. have.

The command execution control device is configured to execute the command data in the command execution device when a command of the newly added command data is a command for generating a state change in a circuit implemented on a breadboard by using the command execution device. The method may further include a function of converting the converted command execution code into a command execution code of a corresponding format, and transmitting the converted command execution code to the command execution device.

According to another aspect of the present invention, a method in which a physical computing system (hereinafter referred to as 'system') that performs hardware control and programming through the conversation performs hardware control and programming through the conversation comprises: (a ) receiving a command converted from a user's voice command into text from a voice input/output device, and generating command data from the received command; (b) generating or updating the command data as a command file; (c) when there is a change in the command file, sending command data newly added to the command file, and transmitting image data for displaying a status according to the execution of the command data to the display monitor ; (d) displaying the image data received on the display monitor; (d1) providing a graphical interface for setting new parts according to a user's request; and (d2) according to the user's input of setting contents and clicking of the save button, a definition file (hereinafter referred to as 'part file') including the setting contents of the part is newly stored or the Definitions are added and updated and stored.

When the command of the command data newly added in step (c) is a command for generating a state change using a command execution device in a circuit implemented on a breadboard, after step (b), (c11) converting the command data into command execution code in a format executed by the command execution device; and (c12) transmitting the command execution code to the command execution device.

According to the present invention, when a general-purpose prototype circuit is manufactured, additional information on a user's request for circuit assembly is provided or the assembly step can be interactively performed, and the debugging process through testing of the assembled circuit is also By making it possible to perform interactively, it has the effect of enabling easy and accurate fabrication of prototype circuits.

In addition, by using the system of the present invention, when a user writes a program code for circuit testing, etc., such program code can be written and executed in real time through a conversational method, thereby enabling easy and fast debugging. .

1 is a diagram showing an actual appearance of a physical computing system that performs hardware control and programming through conversation, according to the present invention;
2 is a diagram illustrating an overall network configuration of a physical computing system that performs hardware control and programming through conversation, according to the present invention;
3 is a diagram showing a sequence diagram for performing hardware control and programming through conversation, according to the present invention;
4 is a diagram illustrating a graphic interface displayed through a display monitor in the process of controlling and programming hardware through conversation.
5 is a diagram illustrating an embodiment of a command used in a process of controlling and programming hardware through a conversation;
6 is a diagram illustrating another embodiment of a command used in a process of controlling and programming hardware through a conversation;
7 is a diagram illustrating another embodiment of a command used in a process of controlling and programming hardware through conversation;
8 is a diagram showing another embodiment of a command used in a process of controlling and programming hardware through conversation;
Fig. 9 is a diagram showing an embodiment of a graphical interface for defining the operation of a new component to be added as customization;
Figure 10 shows six embodiments of a circuit prototype that can be formed through a physical computing system that performs control and programming of hardware through dialogue, of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Prior to this, the terms or words used in the present specification and claims should not be construed as being limited to conventional or dictionary meanings, and the inventor should properly understand the concept of the term in order to best describe his invention. Based on the principle that it can be defined, it should be interpreted as meaning and concept consistent with the technical idea of the present invention. Therefore, the configuration shown in the embodiments and drawings described in the present specification is merely the most preferred embodiment of the present invention and does not represent all of the technical spirit of the present invention, so at the time of the present application, various It should be understood that there may be equivalents and variations.

FIG. 1 is a diagram illustrating an actual state of a physical computing system 1000 that performs hardware control and programming through conversation, according to the present invention, and FIG. 2 is a diagram illustrating hardware control and programming through conversation according to the present invention. It is a diagram showing the overall network configuration of the physical computing system 1000 for performing It is a diagram illustrating a graphic interface displayed through a display monitor in the process of hardware control and programming through Functions provided in the process of implementing and testing a prototype of

Hereinafter, the entire 'physical computing system 1000 performing hardware control and programming through conversation' shown in FIG. 2 will be simply referred to as 'system 1000'.

The user 10 initiates physical computing through the system 1000 through voice (S301). The voice input/output device 100 recognizes the voice of the user 10, converts it into text data, and then transmits it to the user command server 200, and also converts the conversation text received from the user command server 200 into voice. It serves to output to the user 10 .

Thereafter, the conversation process S302 between the user 10 and the user command server 200 is a process in which the user completes the command desired to be executed by the system 1000 through the conversation. Such a dialogue sequence will be described later in detail with reference to FIGS. 5 to 8 . The user command server 200 may be a kind of cloud server located on a network. As shown in FIG. 3 , an application implementing the functions performed by the user command server 200 is uploaded and installed to the user command server 200 , and the user 10 is connected to the user command server 200 through the Internet. ), the sequence of FIG. 3 may be performed by accessing the corresponding application.

An embodiment of the conversation process ( S301 , S302 ) between the user 10 and the user command server 200 will be described with reference to FIG. 5 . That is, when the user starts a voice command "Write pin 3", the command converted into text data is transmitted to the user command server 200 through the voice input/output device 100 (S301). The user command server 200 recognizes that 'Pin Value' is missing from 'Pin Value' and 'Pin Number', which are parameters required for the 'Write' command of FIG. value?" is output, and the voice input/output device 100 receives it, converts it into a voice, and outputs it to the user 10 (S302). Accordingly, the user commands an appropriate value to be applied to pin 3, and the user command server 200 receives it. In a conversation between the user 10 and the user command server 200, the interface by the voice input/output device 100 is always performed as described above, so the process of performing the interface of the voice input/output device 100 will be omitted for convenience in the following description. to explain

The user command server 200 generates the 'Write' command including all parameters as 'command data' when both 'Pin Value' and 'Pin Number', which are parameters for the 'Write' command of FIG. 5 are received as described above. (S303), and transmits the command to the file server 300 (S304). The 'command data' is data obtained by program-coding a command commanded by a user, and an embodiment thereof is as follows.

The command data may be text data representing such a program code, but may also be data in another format having the meaning of the above program code. The command file server 300 generates or updates such command data as a 'command file' and stores it (S305).

In the above description, the user command server 200 and the command file server 300 are configured as separate servers, but the functions implemented by the user command server 200 and the command file server 300 may be implemented in one server. .

The command execution control device 400 (refer to FIGS. 1 to 3 ) periodically accesses the command file server 300 connected through the network, and checks whether there is a change due to an update or the like in the command file stored in the command file server 300 . is detected (S306), and if there is a change, a command newly added to the command file is read (S307). In this case, the command execution control device 400 may read and store the added command in the format of 'command data' generated by the user command server 200 .

The command execution control device 400 is connected to the command execution device 600 as shown in FIG. 1 , and the command execution device 600 is connected with a wire or the like according to a circuit configured on the breadboard 700 . For example, when the command execution control device 400 reads a command to apply a predetermined value to pin 3 through the “Write pin 3” command from the command file server 300 , the command execution control device 400 transmits the command It is sent to the execution device 600, converted into executable code that can be directly executed by the command execution device 600 (S308) and sent to the command execution device 600 (S311). That is, when the command execution device 600 receives such an execution code, the command execution device 600 applies a corresponding value to pin 3 of the command execution device 600 according to the command execution code ( S312), the corresponding value is transmitted by the wire connecting pin 3 and the breadboard 700 accordingly, and the corresponding value is transmitted to the circuit of the breadboard 700 .

On the other hand, the command execution control device 400 stores the new command data S307 read from the command file server 300 , and displays a result of executing the command, that is, a graphic image, to the connected display monitor 500 . and transmits (S309), and the display monitor 500 displays it on the screen. 4 shows an embodiment in which the graphic image is displayed on the screen of the display monitor 500 as described above.

4(b) is an enlarged view of the display monitor 500 in FIG. 4(a), and FIG. 4(c) is an enlarged view of the breadboard 700 in FIG. 4(a). to be. '51' means a value applied to the corresponding pin, and '52' means log data for commands executed so far. '53' indicates that '54' on the breadboard 700 is displayed on the screen. Reference numerals '55' to '57' denote examples of components included in the breadboard 700, '55' denotes an LED, '56' denotes a photoresistor, and '57' denotes a switch.

The command execution control device 400 stores images of the command execution device 600 , the breadboard 700 , and each component, and also knows the state of each pin according to the command executed so far, so it is imaged By sending one data to the display monitor 500, it is displayed on the display monitor 500 in the form shown in FIG. 4(a).

5 is a diagram illustrating an embodiment of a command used in a process of controlling and programming hardware through conversation.

The 'Write' command has been described above, and the other commands of FIG. 5 are also performed in a similar manner to the 'Write' command. When the user speaks a "Read" command, the user command server 200 asks "analog or digital?" about the pin type, to clarify the command, and "What is the pin number?" is asked.

In FIG. 5 , the 'Pulse' command is a signal for applying a pulse signal to a specific pin at a certain frequency. Here, as shown, parameters such as 'Pin Number', 'Delay', and 'Count' are required. For the sake of clarity, you will be asked questions about such parameters to the user. 'Delay' means how long after the pulse will be applied, and 'Count' means how many times the pulse will be generated.

In FIG. 5, 'Pass Data' refers to a signal that allows an output signal of a specific pin to virtually enter an input signal of another specific pin, and the user can issue a command such as "Pass data from pin 1 to pin 11". can

6 is a diagram illustrating another embodiment of a command used in a process of controlling and programming hardware through conversation.

Unlike FIG. 5, FIG. 6 shows a command in which branching of several stages is supported, and there are an 'if' statement and a 'do-after' command.

An example of the 'if' statement is a command such as "If pin 5 is high, then write pin 3," issued by the user 10 .

The 'do-after' command is a command from the user 10 to perform a specific action after a specific predetermined time interval (delay).

On the other hand, in a conversation for command completion between the user 10 and the user command server 200, the user 10 does not necessarily follow a set phrase when speaking a command, and can speak in a natural language. have. Also, the user 10 does not need to memorize all the options in the command.

In this case, the user command server 200 receives the words of the user 10 .

i) autonomously infer the exact meaning of the command, or

ii) by asking the user 10 again,

You can figure out the exact meaning of the command. This method is equally applied to all the conversation processes of FIGS. 5 to 8 .

For example, if the user 10 initiates the command "If pin 5 is high, then write pin 3," the user command server 200 may indicate that 'high' indicates a digital state. Therefore, it can be inferred that the 5th pin is a digital pin.

However, since the "write pin 3" part is ambiguous, the user command server 200 asks "What value?" A question for clarifying the command may be asked to the user 10 . In response to this, as the user 10 answers with a digital value (high or low) or a numeric value from 0 to 255, the user command server 200 sends the digital value or analog value (PWM, pulse) to pin 3 width modulation) is applied.

Following the same logic, a user 10 who wants to read a value from analog pin 1 says a complete command such as "Read value from analog pin 1", "Read from analog pin 1", or "Read from pin 1" Incomplete commands such as 1” and “Read” can be said. In the case of such an incomplete command, as described above, the user command server 200 sends “Analog or digital?” or "What is the pin number?" The command is completed according to the answer of the user 10 by asking a question to the user 10 , for example.

7 is a diagram illustrating another embodiment of a command used in a process of controlling and programming hardware through conversation.

'Pin Type', 'Pin Number', 'Pin Value', 'Delay', 'Count', 'Number', etc. shown in FIG. 7 are parameters that have already appeared in FIGS. 5 and 6, and in FIG. 7, such parameters This is an example showing in detail what process is performed when supplementing with a question. For example, in FIG. 7 , when the pin type is not mentioned in the command spoken by the user 10 , the user command server 200 may ask the user "Analog or digital?" If "Analog" or "Digital" is answered, the user command server 200 receiving it can complete the command.

In FIG. 7 , 'comparison' shows a question of the user command server 200 for clarifying an inequality sign between objects to be compared, and an example of an inequality sign to which the user 10 can answer.

The 'component' of FIG. 7 will be described below with reference to FIG. 8 .

8 is a diagram illustrating another embodiment of a command used in a process of controlling and programming hardware through conversation.

In FIG. 8( a ), 'Check', 'Test', and the command check the state of a component existing on the circuit of the breadboard 700 connected to the command execution device 600 , or the corresponding component. This is a command to perform a test on . In this case, when the user says only the "Check" or "Test" command, the user command server 200 asks the user 10 what the component is, and the user asks the user 10 what is the component as shown in FIG. 7 . one of them, the user command server 200 completes the command according to the part name received in this way and transmits the corresponding command data to the command file server 300 . Such a part can be added by the user 10 , which will be described later with reference to FIG. 9 .

The 'How to use' command is a command in which the user command server 200 explains how to use a specific component when the user does not know how to connect it to the circuit. This command also has to include parameters for parts, just like 'Check', 'Test', and commands.

FIG. 8( b ) is an embodiment of phrases that more flexibly connect a conversation between the user 10 and the user command server 200 . That is, when the user 10 says "Wait", the user command server 200 answers "Ok, I will wait", in the same manner as waiting for the user's next command.

Hereinafter, the system 1000 of the present invention provides functions provided by the user 10 in the process of implementing and testing a prototype of a circuit by using various commands described with reference to FIGS. 5 to 8. This will be summarized and explained.

1. Programming through conversation

The user 10 prepares parts such as an anemometer sensor, a toggle switch, a resistor, an LED, and a jumper wire, and gives a voice command to the system 1000 of the present invention. The user connects the resistor and the LED to the breadboard 700 as a wire, and connects the anode of the LED to the digital pin 11 (D11) of the command execution device 600 board, and then says “I want to write high” to pin 11," the LED turns on.

The user 10 connects a photoresistor to the analog pin 1 (A1) of the command execution device 600 board and says "Hmm... try to read from pin 1," and the system 1000 's user command server 200 asks whether the pin is an analog pin or a digital pin, and is clarified by the user's 10 answer.

After confirming that the pin is analog, the user 10 covers a part of the sensor with his hand, and the numerical value displayed on the screen next to the A1 pin of the command execution device 600 board graphically displayed on the display monitor 500 check in real time. Finally, the user uses the value read from the sensor to adjust the brightness of the LED, just say "Pass data from pin 1 to pin 11." Now, the user 10 can adjust the brightness of the LED by partially covering the photoresist cell by hand. Afterwards the user 10 says "If A1 is greater than 120, then pulse D10 every 200 milliseconds." You can test other LEDs with the pulse command by saying

The system 1000 may use a command as shown in FIGS. 5 to 8 , and may read signals of 10 pins of the command execution device 600 board and apply a signal to the corresponding pins. Passing data from an input pin to an output pin, pulsing a pin, resetting a pin, branching a command, or performing a certain action ( action) can be scheduled to occur. All commands take effect immediately, and the user can simultaneously check the status of all pins in real time through the hardware on the breadboard 700 and the graphic interface displayed on the display monitor 500 .

The voice and sentence recognition capabilities of the system 1000 of the present invention, as described above, allow the user 10 to speak naturally without the need to memorize exact sentences, and the user command server 200 of the system 1000, Even if the user 10's command is inaccurate or contains dialogue phrases not directly related to the command (eg, "Hmm", "Uh", "Try to", "I want", etc.), the exact meaning of the command is autonomously determined. can be inferred. Also, even in the case of an unclear or incomplete command as described above, the user command server 200 may complete the command by supplementing all additional parameters of the command by asking the user 10 a question.

2. Error detection and inference through context

After the user 10 connects the anode of the LED to pin 12 (a digital pin without an analog output function), the writing of the analog signal PWM can be tested with the LED. In this case, when the user 10 says "I want to write the value 200.", the user command server 200 of the system 1000 may answer "What is the pin number?" When the user 10 says "Pin 12.", the user command server 200 recognizes an error in the user's answer, and answers as "This is not possible. I wrote the digital value high on digital pin 12 instead." can do. In this case, after realizing that the user 10 made a mistake and resetting the current state, the user 10 may execute the above command again on the D10 pin supporting the analog output.

In the command execution device 600 board as an embodiment, limited pins of analog input pins (A0~A5) and analog (PWM) output pins (~D3, ~D5, ~D6, ~D9, ~D10, and ~D11) Only a subset functions as analog input pins and analog output pins. Any pin can perform digital read/write operation, but as mentioned above, only designated analog pins can perform analog output (PWM) or analog input operation.

Specifically, the output operation maps a value from 0 to 255 to a duty cycle of 0 to 100%, while the input operation digitally converts a voltage ranging from 0 to 5V into a number from 0 to 1023. do. The user command server 200 of the system 1000 of the present invention may detect that a user has accidentally attempted an analog operation on a digital pin. In this case, the user may be notified and corrected.

The user command server 200 may prevent a logical error by notifying the user 10 about a case in which pins may be used incorrectly. For example, when using a numeric value for a digital pin, or using a 'Pass data' command between different types of pins.

Furthermore, the user command server 200 can autonomously infer the correct option when there is no risk of ambiguity. For example, when the user says "If pin 5 is high, then write pin 10", the user command server 200 may determine that pin 5 is a digital pin because 'high' represents a digital state.

However, when the user 10 says "Write pin 10", the user command server 200 may request the user 10 to clarify. As the user 10 replies with a digital value (high or low) or a numeric value (a number from 0 to 255), the user command server 200 writes a digital value or analog value (PWM) to pin 10. can (writing)

3. Providing information to users who are developing the circuit

If the user uses the anemometer sensor for the first time, the user 10 may not know how to connect it, and thus may ask the system 1000 such as “How do I use an anemometer?”. This corresponds to the 'How to use' command as described above, and is a command that informs how to connect on the breadboard 700 with respect to a specific component. In this case, the system 1000 graphically shows how to connect the wind speed sensor component to the breadboard 700 and the command execution device 600 using a wire on the display monitor 500 step by step, or the final wire connected The ring status can be displayed and shown at once. Also, the system 1000 may explain such a connection process to the user 10 through voice output.

The process in this case is also performed according to the sequence of FIG. 3 . That is, when the user command server 200 explains the connection of the wind speed sensor step by step after recognizing the question of the user 10 "How do I use an anemometer?" (S301, S302), each step corresponds to The description is generated as command data step by step (S303), and the command data is sent to the command file server 300 (S304). The command file server 300 updates the command data corresponding to the step in the command file (S305), and the command execution control device 400 periodically accesses the command file of the command file server 300 to check whether a change has occurred. During detection, if a change is detected (S306), the command data added from the corresponding command file is read (S307). As described above, the command file server 300 may be configured as a separate server, but the function of the command file server 300 may be configured to be performed by the user command server 200 . That is, the user command server 200 may also perform the role of the command file server 300 .

As in the case described in FIG. 3 , if a signal is applied or a signal value is read in relation to pins connected to the command execution device 600 and the breadboard 700 , since the command execution device 600 must be controlled, the command execution device It is necessary to generate a command execution code to be transmitted to the 600 ( S308 ), but as in the case of the above 'How to use' command, a connection between the command execution device 600 and the breadboard 700 to the user 10 (wiring) If the steps are described, it is not necessary to generate the command execution code, and thus, the process of transmitting the command execution code to the command execution device 600 is also unnecessary. Only the command execution control device 400 transmits the graphic image displayed on the screen of the display monitor 500 of the corresponding wiring state to the display monitor 500 (S309), and the display monitor 500 displays it on the screen (S310). ) will do. As shown on the screen of the display monitor 500 as described above, the user 10 may perform wiring. The wiring state displayed on the screen of the display monitor 500 in FIG. 4 may be a wiring type shown by the system 1000 according to the 'how to use' command. In addition, the system 1000 may explain the connection process step-by-step to the user 10 through voice output.

The final step in wiring is to connect the toggle switch to provide power. As in the previous step, the user 10 may perform the task by asking how to use the switch to the system 1000 .

Thereafter, the user command server 200 of the system 1000 infers that the above-described connection operation of the wind sensor is completed, and asks the user 10, "Would you like to test the anemometer first?" By asking the question, it can be suggested that the user 10 check whether the completed board operates properly. The user 10 answers "Yes" and can also tell the system 1000 which pin the wind sensor is connected to. The user command server 200 may cause the user 10 to blow wind with the wind sensor, and may show the value detected by the wind sensor on the display monitor 500 . This process may be accomplished by reading the wind sensor detection value from the command execution device 600 , the command execution control device 400 receiving it, and sending it to the display monitor 500 .

System 1000 may also ask user 10 if the device is operating. If it does not work, the system 1000 may suggest a list of troubleshooting actions, and if the user 10 is satisfied with the result, the test may stop and move on to the next task.

On the other hand, system 1000 may allow user 10 to gather information and test any component of the circuit. System 1000 maintains a list of predefined parts and assembly instructions. This is stored in the command execution control device 400 . However, the user 10 can define its own additional parts and assembly instructions, dialogs, operating modes (input/output, analog/digital), and troubleshooting suggestions. The system 1000 does not require a user to write a new program code for such a change, and enables the formation through a graphical interface that reads and updates a database of parts.

The test may be initiated by the user 10 directly speaking a test/check command, or may be initiated spontaneously by the system 1000 according to the current context. For example, if the user 10 asks for instructions for a new component, the system 1000 reminds the user 10 of the work done so far and suggests a test. The system 1000 then attempts to write or read any value to the pin of the part under test, and asks the user 10 if it works, thereby confirming that there are no implementation problems with the module being tested, so that the rest of the Do not affect the program.

4. Complete Circuit Prototype

When the prototype of the circuit is finally completed, the programming logic is put into operation in the system 1000 .

When the intensity of the lamp configured on the circuit is changed according to ambient light, the user 10 may turn off the lamp by blowing wind with the wind sensor (switch off).

The user 10 converts the entire program currently operating in the system 1000, that is, the command execution control device 400, into one executable code (written in C/C++ as an embodiment) file as a command execution control device 400 can be stored in Thereafter, the user 10 can upload the code file to the command execution device 600 , and thus can directly use the lamp circuit on the breadboard 700 connected to the command execution device 600 .

The system 1000 provides the user 10 with an option to convert the resulting interaction into an executable code file that is executed on the command execution device 600 . The user can manually upload the executable code to the command execution device 600, or modify the code by adding function blocks, copy/paste parts of the code, and add them to the library if necessary. This feature allows the user 10 to reuse or reference auto-generated code as a template.

FIG. 9 is a diagram illustrating an embodiment of a graphic interface for defining an operation of a new component to be added as customization.

The app for driving such a graphic interface may be installed in the command execution control device 400 , or may be installed in a separate PC (not shown) connected to the display monitor 500 . That is, the graphic interface as shown in FIG. 9 may be driven by the command execution control device 400 or a separate PC.

As an example, system 1000 may support 12 components of different types (input/output, analog/digital) and different numbers of pins and different interfaces. For example, LED, RGB LED, hall effect sensor, tilt sensor, piezo speaker, flex sensor, anemometer, temperature sensor, servo motor, A pushbutton, a potentiometer, or a photoresistor.

The list 905 may be customized by the user 10 , and a new component may be set by a graphical interface. That is, the system 1000 does not need to write new code for this, just using the graphical interface that appears on the display monitor 500 of FIG. 9 makes this possible. Through this interface, the user selects a file (hereinafter referred to as a 'part file') in which a definition including the setting contents of the new part is to be stored (901). In this part file, settings of previously set parts (shown in list 905) are stored, and the part file is stored in the command execution control device 400 .

In addition, when the input of setting items in the box of FIG. 9 is completed and the save button 907 is pressed, the component file of the command execution control device 400 is also updated and saved, and this file is also stored in the user command server 200 on the cloud. are uploaded

In addition, an image for the corresponding part and its wiring detailed state is selected (902), and the image may be a file stored in a separate PC, etc., completing input of settings in the box of FIG. ) is pressed, the selected image file is also mapped to the corresponding part in the command execution control device 400 and stored.

A new part is created by name definition 903 . When the definition is completed as the operation definition through the check box (input/output, digital/analog) 904 and the dialog definition 906, the definition is 'ADD' and uploaded to the parts list 905, and then ' By 'SAVE', the settings of the corresponding part are stored in the part file of the command execution control device 400 , and the corresponding image file is also stored in the command execution control device 400 . In particular, as described above, the part file is uploaded to the user command server 200 , where the dialog proceeds according to a dialogue 906 stored in the part file.

In the dialogue 906 , an answer sentence when the user command server 200 provides a guide for the corresponding part after recognizing the user's question is defined. In particular, when the user 10 asks a question by the 'how to use' command for the part, or when the user 10 makes a 'test' command for the part, It is used when the user command server 200 guides a debugging process.

10 is a diagram illustrating six embodiments of a circuit prototype that can be formed through a physical computing system that performs control and programming of hardware through dialogue, of the present invention.

100: voice input/output device
200: user command (command) output server
300: command file server
400: command execution control device
500: display monitor
600: command execution device
700: breadboard (breadboard)
1000: a physical computing system that performs hardware control and programming through dialogue

Claims (19)

delete delete delete delete delete A method for controlling the execution of a command by a user's voice by a command execution control device of a physical computing system that performs hardware control and programming through conversation, comprising:
(a) periodically accessing the user command server or, if the command file server exists separately, periodically accessing the command file server to check whether there is a change due to an update of the command file;
(b) when there is a change in the command file, importing new command data added to the command file;
(c) transmitting image data for displaying a status according to the execution of the command data to the display monitor;
(d1) providing a graphical interface for setting new parts according to a user's request; and,
(d2) According to the user's input of setting contents and clicking the save button, a definition file (hereinafter referred to as 'part file') including the setting contents of the part is newly saved, or the definition of the part is stored in an existing part file. Steps to be added and updated
A method for controlling execution of a user command of a command execution control device, comprising: 7. The method of claim 6,
When the command of the command data newly added in step (b) is a command for generating a state change using a command execution device in a circuit implemented on a breadboard, after step (b),
(b11) converting the command data into command execution code in a format executed by the command execution device; and,
(b12) transmitting the command execution code to the command execution device
User command execution control method of the command execution control apparatus, characterized in that it further comprises. 7. The method of claim 6,
After step (b),
(b21) updating and storing the command data in a log file
User command execution control method of the command execution control apparatus, characterized in that it further comprises. delete 7. The method of claim 6,
When a user's request input is received,
(e1) storing the entire program being operated in the command execution control device as a single executable code file in the command execution control device; and,
(e2) uploading the stored executable code file to a command execution device
User command execution control method of the command execution control apparatus, characterized in that it further comprises. A computer program operating in a command execution control device of a physical computing system that performs hardware control and programming through conversation in order to control execution of commands by a user's voice, comprising:
It is stored in a non-transitory storage medium, and by the processor,
(a) periodically accessing the user command server or, if the command file server exists separately, periodically accessing the command file server to check whether there is a change due to an update of the command file;
(b) when there is a change in the command file, importing new command data added to the command file;
(c) transmitting image data for displaying a status according to the execution of the command data to the display monitor;
(d1) providing a graphical interface for setting new parts according to a user's request; and,
(d2) According to the user's input of setting contents and clicking the save button, a definition file (hereinafter referred to as 'part file') including the setting contents of the part is newly saved, or the definition of the part is stored in an existing part file. Steps to be added and updated
A computer program operating in a command execution control device of a physical computing system that performs hardware control and programming through conversation in order to control execution of a command by a user's voice, including a command to be executed. 12. The method of claim 11,
When the command of the command data newly added in step (b) is a command for generating a state change using a command execution device in a circuit implemented on a breadboard, after step (b),
(b11) converting the command data into command execution code in a format executed by the command execution device; and,
(b12) transmitting the command execution code to the command execution device
A computer program stored in a non-transitory storage medium, characterized in that it further comprises instructions to be executed. An apparatus for controlling execution of a command by a user's voice, comprising:
at least one processor; and
at least one memory for storing computer-executable instructions;
The computer-executable instructions stored in the at least one memory are executed by the at least one processor,
(a) periodically accessing the user command server or, if the command file server exists separately, periodically accessing the command file server to check whether there is a change due to an update of the command file;
(b) when there is a change in the command file, importing new command data added to the command file;
(c) transmitting image data for displaying a status according to the execution of the command data to the display monitor;
(d1) providing a graphical interface for setting new parts according to a user's request; and,
(d2) According to the user's input of setting contents and clicking the save button, a definition file (hereinafter referred to as 'part file') including the setting contents of the part is newly saved, or the definition of the part is stored in an existing part file. Steps to be added and updated
An apparatus for controlling the execution of a command by a user's voice, which causes the command to be executed. 14. The method of claim 13,
When the command of the command data newly added in step (b) is a command for generating a state change using a command execution device in a circuit implemented on a breadboard, after step (b),
(b11) converting the command data into command execution code in a format executed by the command execution device; and,
(b12) transmitting the command execution code to the command execution device
An apparatus for controlling the execution of a command by a user's voice, characterized in that it further executes. A physical computing system that performs hardware control and programming through dialogue, comprising:
a user command server that receives a command converted from a user's voice command into text from a voice input/output device, and generates command data from the received command;
Changes in the command file including the command of the command data are periodically checked, and when there is a change in the command file, command data newly added to the command file is imported, and a status according to the execution of the command data is displayed. Transmitting image data to be displayed on the monitor to the display monitor, providing a graphic interface for setting new parts according to the user's request, including the setting contents of the parts according to the user's input of setting contents and clicking the save button a command execution control device in which a definition file (hereinafter referred to as a 'part file') is newly stored, or a definition of the part is added to an existing part file and updated and stored; and,
A display monitor for displaying image data received from the command execution control device
A physical computing system that performs control and programming of hardware through dialogue, including a. 16. The method of claim 15,
The user command server further includes a function of generating or updating the command data as a command file and storing it; or
Further comprising a command file server for generating or updating the command data as a command file and storing,
The method in which the command execution control device periodically checks for changes in the command file,
When the user command server includes a function of generating or updating the command data as a command file and storing the command data, the command file of the user command server is periodically checked, and if the command file server exists, the command file server to periodically check the command file of
A physical computing system that performs hardware control and programming through dialogue, characterized in that 17. The method of claim 16,
The command execution control device,
When the command of the newly added command data is a command that generates a state change in a circuit implemented on a breadboard by using a command execution device,
A function of converting the command data into a command execution code in a format executed by the command execution device, and transmitting the converted command execution code to the command execution device
A physical computing system for performing hardware control and programming through conversation, characterized in that it further comprises a. A method in which a physical computing system (hereinafter referred to as 'system') that performs hardware control and programming through dialogue of claim 16 performs hardware control and programming through dialogue,
(a) receiving a command converted from a user's voice command into text from a voice input/output device, and generating command data from the received command;
(b) generating or updating the command data as a command file;
(c) when there is a change in the command file, sending command data newly added to the command file, and transmitting image data for displaying a status according to the execution of the command data to the display monitor ;
(d) displaying the image data received on the display monitor;
(d1) providing a graphical interface for setting new parts according to a user's request; and,
(d2) According to the user's input of setting contents and clicking the save button, a definition file (hereinafter referred to as 'part file') including the setting contents of the part is newly saved, or the definition of the part is stored in an existing part file. Steps to be added and updated
A method of performing hardware control and programming through a dialogue comprising a. 19. The method of claim 18,
When the command of the command data newly added in step (c) is a command for generating a state change using a command execution device in a circuit implemented on a breadboard, after step (b),
(c11) converting the command data into command execution code in a format executed by the command execution device; and,
(c12) transmitting the command execution code to the command execution device
Method for performing control and programming of hardware through a conversation, characterized in that it further comprises.

Images