WO2020040771A1

WO2020040771A1 - Visual input command recognition

Info

Publication number: WO2020040771A1
Application number: PCT/US2018/047727
Authority: WO
Inventors: Mary G. Baker; Eric FAGGIN
Original assignee: Hewlett-Packard Development Company, L.P.
Priority date: 2018-08-23
Filing date: 2018-08-23
Publication date: 2020-02-27

Abstract

In one example in accordance with the present disclosure, an electronic device is described. The electronic device includes a capture system. The capture system receives authentication information to execute a command on the electronic device and following verification of the authentication information, captures a command symbol disposed on a physical medium input. A command recognition device of the electronic device interprets the command symbol into an executable command and a controller of the electronic device executes the command.

Description

VISUAL INPUT COMMAND RECOGNITION

BACKGROUND

[0001] Electronic devices undoubtedly play a role in many people’s day- to-day lives and computers are relied on by many users. Components of those computers such as printers and scanners are also used daily to receive and output information to and from a user. The use of such devices, as well as the fields in which they are used, is greatly expanding within society. For example, network-connected thermostats, security systems, and other home automation equipment have increased in use over the years.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The accompanying drawings illustrate various examples of the principles described herein and are part of the specification. The illustrated examples are given merely for illustration, and do not limit the scope of the claims.

[0003] Fig. 1 is a block diagram of an electronic device with visual input command recognition, according to an example of the principles described herein.

[0004] Fig. 2 is a flowchart of a method for visual input command recognition, according to an example of the principles described herein.

[0005] Fig. 3 is a block diagram of an electronic device with visual input command recognition, according to another example of the principles described herein. [0006] Figs. 4A-4D depict an example of using a visual input command recognition, according to an example of the principles described herein.

[0007] Fig. 5 is a flowchart of a method for visual input command recognition, according to another example of the principles described herein.

[0008] Throughout the drawings, identical reference numbers designate similar, but not necessarily identical, elements. The figures are not necessarily to scale, and the size of some parts may be exaggerated to more clearly illustrate the example shown. Moreover, the drawings provide examples and/or implementations consistent with the description; however, the description is not limited to the examples and/or implementations provided in the drawings.

DETAILED DESCRIPTION

[0009] While such electronic devices have clearly advanced society in a number of ways, in some cases advances to their operation may provide additional advantages to societies in which they are used. For example, electronic devices operate based on input commands. For example, a user, via a keyboard or a mouse, may instruct a computing device to execute some operation, such as printing a document. As another example, a user may use a mobile device with a network connection to issue a command to a network- connected device such as a thermostat. However, in some examples, the unavailability of input devices may preclude some use of electronic devices.

[0010] For example, in some countries there may be little network connectivity in classrooms or schools where students and even teachers do not have computers. However, there may be a printer/scanner in the classroom or school to be used for printing, copying, and/or scanning. These printers, copiers, and scanners may be rather simplistic without touch displays or enough buttons to receive input. Accordingly, these printers/scanners while having potential of advancing education by printing out textual subject matter and scanning in test answers, etc., are rendered ineffective without an interface through which such devices can be operated. [0011] As another example, smart devices can be used to control aspects of daily life such as setting a thermostat, security system, or other home automation device. However, such input-constrained (that is no keyboard, mouse, or large display) rely on network connectivity for their control, which poses the question of how they will be controlled when there is no network connectivity. During these periods of time, additional input mechanisms would facilitate interaction with the electronic devices.

[0012] Accordingly, the present specification describes a way of interacting with such devices which may difficult to interface with under certain circumstances. Such an interaction allows a user to direct or configure the device notwithstanding an incomplete input interface. According to the present specification a capture system, such as a scanner or camera is used to receive commands. The electronic device can then interpret a digital representation of the command to configure itself, produce diagnostics, or to direct the behavior or functionality of the electronic device.

[0013] For example, regarding an educational system in a developing country, a printer may output desired material responsive to a scanned in document that includes commands to print the desired material. In the example of a network-connected device, an optical capture system such as a camera may be included in the network-connected device. This camera recognizes paper-based or other graphical input. Accordingly, commands on the paper- based input can be scanned and used to alter the operation of the network- connected device.

[0014] In other words, the electronic device of the present specification relies on paper-based or other visual input to configure the electronic device thus allowing in-the-moment, non-gestural, intuitive visual instruction of the electronic device. In some examples, iterative interaction with a printing device can allow for the selective printing and scanning of different content.

[0015] Accordingly, the present specification describes an electronic device. The electronic device includes a capture system to: 1 ) receive authentication information to execute a command on the electronic device and 2) following verification of the authentication information, capture a command symbol disposed on a physical medium input. A command recognition device of the electronic device interprets the command symbol into an executable command and a controller of the electronic device executes the command.

[0016] The present specification also describes a method. According to the method, a physical medium input is optically captured. A first command symbol is extracted from a digital representation of the physical medium input and a physical medium output is generated, which physical medium output is dependent upon a received first command symbol. A second command symbol, which is input by a user on the physical medium output, is optically captured.

An operation of the electronic device is initiated based on the extracted second command symbol, the second command symbol being extracted from a digital representation of the physical medium output.

[0017] In another example, an electronic device includes the capture system and command recognition device. In this example, the electronic device includes a printer to generate a physical medium output dependent upon the visual command symbol and a print controller to control the printer.

[0018] In summary, using such an electronic device 1 ) provides for alternate mechanisms for user input; 2) provides user input when user input devices may not be available; 3) provides user input for smart devices even when faced with a lack of network connectivity; and 4) provides for user input when other mechanisms of user input may be ineffective or impractical.

However, the devices disclosed herein may address other matters and deficiencies in a number of technical areas.

[0019] T urning now to the figures, Fig. 1 is a block diagram of an electronic device (100) with visual input command recognition, according to an example of the principles described herein. The electronic device (100) may be any computing device that receives input from a user. Examples of electronic devices (100) include a computer, a scanner, a printer, a copy machine, a thermostat, a home security system, a door lock, and a door bell to name a few. While specific reference is made to a few types of electronic devices (100), other electronic devices (100) may be implemented in accordance with the principles described herein. In some examples, the electronic device (100) may not include certain input devices such as a mouse, keyboard, and a touch screen interface. For example, a printer and/or a scanner may not be able to receive input from a mouse or a keyboard and may not have a touch screen interface through which input is received.

[0020] The electronic device (100) includes a capture system (102) to optically capture a visual user input. The capture system (102) may take many forms. For example, the capture system (102) may include a scanner. For example, a printer electronic device (100) may include components such as an optical scanner to create digital representations of physical media. Another example of such a capture system (102) is a camera. For example, a home security system may include a camera to perform facial recognition or otherwise capture events occurring within the premises that the security system is meant to protect. Other types of capture systems (102) may be used in accordance with the principles described herein.

[0021] The capture system (102) of the electronic device (100) optically captures various pieces of information. For example, the capture system (102) may capture a command symbol that is received as a visual input. The visual input may be of a variety of forms including a physical symbol on a physical medium input. For example, a user may hold up a sheet of paper in front of a thermostat. The sheet of paper may include a command such as“set to 72°.” The camera capture system (102) may capture this command as a visual input.

[0022] In another example, a user may write a command on a sheet of paper that is inserted into a scanner capture system (102). For example, in an educational setting, the user may write a command on a sheet of paper such as, “print 5 copies of chapter 5.” From this captured command, a controller (106) of the electronic device (100) executes a command.

[0023] In some examples, the capture system (102) also receives authentication information to execute the command. For example, it may be desirable to restrict individuals who can use such a visual input command recognition system. As a particular example, it may be desirable that just a teacher in an educational system has the ability to interact with the electronic device (100). Accordingly, the capture system (102) may receive authentication information to control access to the electronic device (100) command execution.

[0024] In this regard, the capture system (102) may include additional components for capturing the authentication information. For example, the capture system (102) may include a facial recognition device to receive facial authentication information. In another example, the capture system (102) may include a biometric input device, such as a fingerprint scanner, to receive biometric authentication information. In yet another example, the capture system (102) may include a keyboard to receive passcode authentication information.

[0025] In yet other examples, the capture system (102) may include the same camera and/or scanner described above and the authentication information may be captured as an authentication symbol formed on a physical medium input. For example, a user, in addition to providing the command to the scanner and/or camera, may provide a security code or password that upon capture by the capture system (102), allows execution of the command triggered by the command symbol.

[0026] In some examples, the capture system (102) is not activated to capture a command symbol until the authentication information is verified. For example, a user may first input a sheet of media into a scanner that includes the passcode“2345.” Following verification of this passcode, a user may then input another sheet of media that includes the command symbol to trigger execution of a corresponding command. Such an authentication-based system ensures security of the electronic device (100) by ensuring that unauthorized users cannot influence the behavior of the device.

[0027] The capture system (102) in some examples also captures a command trigger symbol. The command trigger symbol indicates to the electronic device (100) that a command symbol is forthcoming. For example, any number of visual elements may pass by a camera or may be processed by a scanner. The use of a command trigger symbol ensures that inadvertent commands are not executed when triggered by a visual element that is not a command. For example, a document including the text“Chapter 5” may be input into a scanner. It may be intended that this document is to be scanned and saved to a memory device, and not used to trigger execution of a particular command. However, without such a command trigger symbol, the electronic device (100) may misinterpret this as a command and execute some action, such as printing chapter 5. Accordingly, the use of a command trigger symbol prior to, or simultaneous with, a command symbol could avoid such inadvertent operations on the electronic device (100).

[0028] The electronic device (100) also includes a command recognition device (104) to interpret the command symbol into an executable command.

For example, the command recognition device (104) may perform an optical character recognition operation to convert images of typed, handwritten, or printed text into machine-encoded text. The machine-encoded text, when interpreted can serve as a command to the electronic device (100). For example, as described above, the text“print chapter 5”, when converted to machine-encoded text, could cause the electronic device (100) to retrieve a file named“Chapter 5” and print a copy of that file.

[0029] In another example, such command recognition may include extracting information from a form. For example, a printed document may include data entry fields that a user may fill in. The command recognition device (104) can ascertain which fields have been entered in and execute a particular command. For example, a form may include a list of subjects for a school class and checkmarks by each. An instructor may check a box next to the subject of “History.” Upon receipt of the filled out form, the command recognition device (104) may associate the filled-in box with a command to retrieve a history section of a text book.

[0030] The electronic device (100) also includes a controller (106) to execute the command. The controller (106) may include a processor, and memory to store instructions executable by the processor. Referring to the above example, the command recognition device (104) may transmit the executable command to retrieve the“History” file from memory resources and to print the“History” file. Accordingly, the controller (106) retrieves the file and initializes a print operation to print the“History” document. [0031] In an example where the electronic device (100) is a home security system, a written command of“set alarm” may be received by a camera of the home security system. The command recognition device (104) interprets, or converts, the hand written text into a machine-encoded text that can be used by the controller (106) to activate the home security system to detect intruders.

[0032] As illustrated, an operation executed by the controller (106) can take many forms. For example, a hardware component or a hardware setting of the electronic device (100) may be initialized. As a specific example, the electronic device (100) may be a digital camera with various functionalities including a flash and a variable speed shutter. In this example, a user may hold up a piece of paper to the digital camera with instructions. The camera could take a picture of the instructions and respond accordingly, for example, by turning on the flash or changing the shutter speed.

[0033] In another example, the electronic device (100) may be a thermostat and a user may desire to change the temperature setting such that a heating system is engaged. In this example, a user may write a password as well as a textual instruction to“set temperature to 74°” on a sheet of paper and hold it in front of a capture system (102) such as a camera. In this example, the capture system (102) receives the passcode which is verified by the thermostat, for example, by comparing it to a database of verified passcodes. Following verification, the captured digital representation is passed to a command recognition device (104) which may extract the handwritten text“set

temperature to 74°” and convert it to a machine-encoded instruction to alter the operation of the thermostat. A controller (106) of the thermostat then alters the hardware operation/settings to allow the thermostat to control the heating system to raise the temperature to the desired setting.

[0034] Thus, the present disclosure describes an electronic device (100) that can receive alternate forms of input and adjust operation of the electronic device (100) based on that input. Specifically, the present disclosure describes the interaction with electronic devices (100) that lack interfaces through which such commands may be otherwise received. That is, while some input- constrained devices include some limited input devices, such as a small display on a scanner, such limited input devices may not accommodate the reception of complex inputs such as to print out certain material. Accordingly, the electronic device (100) of the present specification provides for such complex outputs, notwithstanding limited capability to receive input.

[0035] Fig. 2 is a flowchart of a method (200) for visual input command recognition, according to an example of the principles described herein.

According to the method (200), a physical medium input may be optically captured (block 201 ). That is, a capture system (Fig. 1 , 102) includes a device to optically capture a physical medium such as paper. The paper, or other physical medium may include a command symbol, such as alphanumeric characters, other characters and/or graphical representations that represent an instruction to execute a command. The command symbol that is included on the physical medium input is then extracted (block 202) from a digital representation of the physical medium input. That is, the capture system (Fig.1 , 102) generates a digital representation of the physical medium input, and the command recognition device (Fig. 1 , 104) extracts from that digital

representation, the command symbols included thereon. As described above, extraction (block 202) of the first command symbol can take many forms including performing optical character recognition on the digital representation of the physical medium input and/or extracting information from fields on the digital representation of the physical medium input. While particular reference is made to particular operations to extract (block 202) the command symbol, other operations may be executed as well.

[0036] With the command symbol extracted (block 202), a physical medium output is then generated (block 203) dependent upon the first command symbol. That is, a user may include a first command symbol that specifically identifies certain content to be physically output. Accordingly, a printer of the electronic device (Fig. 1 , 100) may generate (block 203) a physical medium output consistent with the content that is mapped to the first command symbol.

[0037] As described above, the physical medium output may include a prompt for additional command symbols. For example, a plurality of checkboxes selectable by a user may allow a user to select additional physical medium outputs. Accordingly, in this example, the electronic device (Fig. 1 ,

100) may optically capture (block 204) a second command symbol. Specifically, the capture system (Fig. 1 , 102) may capture the physical medium output on which a user has input a second command symbol, and the command recognition device (Fig. 1 , 106) can extract from a digital representation of the physical medium output, the second command symbol. The controller (Fig. 1 , 108) can then initiate (block 205) an operation based on the second command symbol. In the specific example of printing educational materials from a textbook the operation may include printing a subsequent index of a hierarchy, or may print subject matter related to a selected topic.

[0038] Fig. 3 is a block diagram of an electronic device (100) with visual input command recognition, according to another example of the principles described herein. In this example, the electronic device (100) includes the capture system (102) and command recognition device (104) as described above in connection with Fig. 1.

[0039] In this example, the electronic device (100) further includes a printer (310) to generate a physical medium output dependent upon the visual command symbol. That is, the command symbol that is received on the physical medium input captured by the capture system (Fig. 1 , 102) may map to a command to generate a physical medium output. In this example, the printer (310) generates the physical medium output and the print controller (308) controls the printer (310) in the generation of the physical medium output.

[0040] The printer (310) may also output other content. For example, the printer (310) may generate an authentication symbol on the physical medium output. That is, as described above the electronic device (100) may preclude execution of a command, or may preclude capture of a command symbol, unless an authentication has been verified. In those examples, a passcode may act as an authentication symbol that would be used to verify the rights of a user to execute a command. In some examples, the printer (310) may generate the authentication symbol, which may be complex. Allowing the printer (310) to generate the authentication symbol may provide for further security as a user- generated authentication symbol may be prone to counterfeiting, whereas an authentication symbol generated by the printer (310) may be complex and more difficult to maliciously replicate. Moreover, in some examples, the electronic device (100) may vary the authentication symbol over time such that it cannot be replicated and entered into the system as easily. In this example, just those media with the variable authentication symbol would be recognized by the electronic device (100).

[0041] In some examples, the printer (310) generates a prompt for a subsequent visual command symbol. That is, a user may, via a scanning capture system (102), insert a first command symbol in the form of a checked box on a form. A physical medium output generated by the printer (310) may include another form with boxes to be checked by a user. Accordingly, responding to such a prompt, the user can check another box such that an interaction with the electronic device (100) to access material from the memory of the electronic device (100) is facilitated. Figs. 4A-4D and 5 depict such an iterative interaction with the electronic device (100).

[0042] Figs. 4A-4D depict an example of using visual command recognition, according to an example of the principles described herein. In this example, the electronic device (100) is a printer/scanner device. Note that as depicted in Fig. 4A, the printer/scanner electronic device (100) does not include a mouse, keyboard, or touchscreen device. In this example, the electronic device (100) may output content (412) as indicated by the arrow. In the specific example depicted in Figs. 4A-4D, the content (412) is an index to a textbook as depicted in Fig. 4B.

[0043] In some examples, the electronic device (100) includes a port (414) through which the content (412) to be output is received. For example, a teacher at a school may receive course material stored on a remote storage device such as a universal serial bus (USB) memory device. Such a device is inserted into the electronic device (100) and facilitates the generation of content (412). In this example, the user relies on output media to interact with the electronic device (100) to tell it which content (412) to be output. [0044] As depicted in Fig. 4B, in some examples the printer (Fig. 3, 310) may generate a visual prompt for a subsequent command symbol. In the example depicted in Fig. 4B, the printer (Fig. 3, 310) generates the form with checkboxes. A user mark in the box is a command symbol that is interpreted by the electronic device (100). Fig. 4B also clearly depicts one form of command symbol extraction, that being from a field on a form. In this example, the user, by selecting the checkbox next to“Physics,” has entered a symbol that may be extracted from a data field by the command recognition device (Fig. 1 , 104).

[0045] Following entry of the command symbol, a user inserts the paper into the capture system (Fig. 1 , 102) of the electronic device (Fig. 1 , 100) which in this example is a scanner as depicted in Fig. 4C. Upon receipt of the command symbol, a command may be executed. As a specific example, the electronic device (100) may output second content (416) which is different than the first content (412). For example, while the first content (412) depicted in Fig. 4B indicates school subjects, the second content (416) may be for sections related to a selected subject. Accordingly, the electronic device (100) with visual input command recognition may allow for the hierarchical access to information. Accordingly, at some point, a user makes a selection and content related to that subject is printed out. That is, the second content (416) may be another index, or may be subject matter related to“Physics.” Such an operation of selecting components within the hierarchy continues until a user has received materials they desire.

[0046] As described herein, the electronic device (100) may be used in an educational setting. In these cases, the electronic device (100) may include a grading device to evaluate a user examination. In this example, the electronic device (100) may generate a graded output of the user examination. For example, when homework or exams are complete, the teacher or a student can insert the work into the scanner as depicted in Fig. 4C. The completed work is then graded against an answer key which may be stored on the external storage device. In some examples, the same sheets can then be over-printed with the corrections for students and instructors to go over. [0047] Fig. 5 is a flowchart of a method (500) for visual input command recognition, according to another example of the principles described herein. According to the method (500), authentication information for a command is received (block 501 ). Specifically, to prevent unauthorized use of the electronic device (Fig. 1 , 100) with the visual input command recognition system, authentication information may first be received (block 501 ) and verified. As described above, the authentication information may come in many forms including a passcode or visual code presented on the physical medium on which a command symbol is transcribed. Following verification of the authentication information, other operations may be performed.

[0048] In one example, a command trigger symbol is captured (block 502). That is as described above, to increase security against malicious or other users from gaining unauthorized access to the electronic device (Fig. 1 , 100), a command trigger symbol may be implemented. A command trigger symbol may be a visual cue to the electronic device (Fig. 1 , 100) that the contents of the physical medium being presented, or that the contents of the immediately subsequent physical medium, contain a command symbol for extraction and interpretation. In this fashion, the capture system (Fig. 1 , 102), in addition to capturing commands, can capture other content, such as non- command symbols. For example, a scanner of an electronic device (Fig. 1 ,

100) may be used to capture commands for generating physical medium output and may be used to simply generate digital representations of a document. These two operations may be distinguished based on a command trigger symbol disposed on physical media that contain commands to be captured.

That is, the command trigger symbol identifies command symbols as relating to executable commands.

[0049] Following verification of the authentication information, a physical medium is optically captured (block 503) and a first command symbol extracted (block 504). In some examples, these may be performed as described above in connection with Fig. 2.

[0050] According to the method (500), a physical medium output is then generated (block 505). As described above, the physical output may include a prompt for additional command symbols. Accordingly, in this example, the electronic device (Fig. 1 , 100) may optically capture (block 506) a second command symbol input onto the physical medium output and can then initiate (block 507) an operation based on the second command symbol. These operations may be performed as described above in connection with Fig. 2.

[0051] Thus, the method (500) allows for input of commands to a printing device via the printing device itself thus providing an additional mechanism for input/commanding the electronic device (Fig. 1 , 100).

[0052] In summary, using such an electronic device 1 ) provides for alternate mechanisms for user input; 2) provides user input when user input devices may not be available; 3) provides user input for smart devices even when faced with a lack of network connectivity; and 4) provides for user input when other mechanisms of user input may be ineffective or impractical.

Claims

CLAIMS What is claimed is:

1. An electronic device, comprising:

a capture system to:

receive authentication information to execute a command on the electronic device; and

following verification of the authentication information, capture a command symbol disposed on a physical medium input;

a command recognition device to interpret the command symbol into an executable command; and

a controller to execute the command.

2. The electronic device of claim 1 , wherein the electronic device is independent of a mouse, keyboard, and touch screen interface.

3. The electronic device of claim 1 , wherein the capture system comprises at least one of:

a scanner; and

a camera.

4. The electronic device of claim 1 , wherein the capture system receives the authentication information as an authentication symbol formed on a physical medium.

5. The electronic device of claim 1 , wherein the capture system further captures a command trigger symbol indicating the command symbol as relating to an executable command.

6. A method comprising:

optically capturing a physical medium input;

extracting a first command symbol from a digital representation of the physical medium input;

generating a physical medium output dependent upon a received first command symbol;

optically capturing a second command symbol input by a user on the physical medium output; and

initiating an operation on the electronic device based on an extracted second command symbol from a digital representation of the physical medium output.

7. The method of claim 6, further comprising capturing a command trigger symbol identifying the first command symbol as relating to an executable command.

8. The method of claim 6, wherein extracting a first command symbol from the physical medium input comprises at least one of:

performing optical character recognition on the digital representation of the physical medium input; and

extracting information from fields on the digital representation of the physical medium input.

9. The method of claim 6, wherein initiating a first operation on the electronic device comprises initializing at least one of a hardware component and a hardware setting of the electronic device.

10. The method of claim 6, further comprising receiving authentication information for a command to be executed on an electronic device; and

wherein optically capturing the physical medium input is performed following verification of the authentication information.

1 1. An electronic device, comprising:

a capture system to:

following verification of the authentication information, capture a visual command symbol formed on a physical medium input;

a command recognition device to interpret the visual command symbol into an executable command;

a printer to generate a physical medium output dependent upon the visual command symbol; and

a print controller to control the printer.

12. The electronic device of claim 1 1 , wherein the printer is to generate an authentication symbol on the physical medium output.

13. The electronic device of claim 1 1 , wherein the printer generates a prompt for a subsequent visual command symbol.

14. The electronic device of claim 1 1 , further comprising a grading device to evaluate a user examination, wherein the printer generates a graded output of the user examination.

15. The electronic device of claim 1 1 , wherein the capture system comprises at least one of:

a facial recognition device to receive a facial authentication information; a biometric input device to receive biometric authentication information; and

a keyboard to receive passcode authentication information.