JP6197920B2 - Data processing apparatus and program - Google Patents

Description

The present invention relates to a data processing apparatus and a program.

  Conventionally, when a command (instruction information) is input to a data processing apparatus such as a personal computer, as a technique for simplifying the command input, a technique of using a paper sheet as a command input medium, that is, printing on a paper sheet The camera is used to shoot how the user designates one of the multiple indices (marks) with his fingertips, etc., and by analyzing the captured image, it is determined which index has been instructed. There is a technique in which a command assigned to an index is input (see Patent Document 1).

JP2012-14606

  However, the technology of the above-mentioned patent document is a guidance system that recognizes a user-designated mark on a paper medium brochure (paper sheet) and displays information content corresponding to the mark on the floor surface. When the number of contents that can be displayed by the association increases, the number of marks on the paper medium also increases, which makes the whole complicated, and there is a risk that the operability may be impaired.

An object of the present invention is to make it possible to select a display option displayed on a display screen by an instruction on a predetermined medium.

Claim 1, among a plurality of display choices displayed on the front示画surface, the position indicator with respect to the display position of an arbitrary display choices the display choices the position indicator as a selection instruction for displaying choices of the display position and selection processing means for performing a selection process corresponding to, in the medium in which a plurality of marks are arranged corresponding to the plurality of display choices when the position of any of the mark is indicated by the user, it is designated comprising discriminating means for discriminating mark position, wherein the selection processing means, by referring to the storage means for storing correspondence between the plurality of marks and a plurality of display choices are disposed on the medium, the display option corresponding to the determined marked by identified by the discriminating means, an indication of the mark on the medium, the display position of the display options corresponding to the mark And position indication to replace instruction to, performs the selection processing corresponding to the display choices characterized and this.

According to the present invention, it is possible to select a display option displayed on the display screen by an instruction on a predetermined medium.

The block diagram which showed the video conference system (communication system). The figure for demonstrating the document camera 3 by the side of the data processor 1 which comprises a video conference system. The figure for demonstrating the paper sheet 7 as a command input medium. The figure for demonstrating the mark recognition method, when the pointing operation which the user points at the mark 8 which the user desires is performed using the paper sheet. 1 is a block diagram showing a basic configuration of a data processing apparatus with a built-in camera (for example, a notebook personal computer) 1. FIG. The figure for demonstrating the structure file X of the screen displayed during application execution. The figure for showing the contents of the setting of the internal table provided in the configuration file X specifically, and explaining the contents of the mark table Y automatically generated based on the setting contents of the internal table. The figure which illustrated the application screen. The figure for demonstrating the sheet | seat management table Z. FIG. The flowchart for demonstrating operation | movement (characteristic operation | movement of this embodiment) of the data processing apparatus 1 started execution according to selection of the application of starting object. 11 is a flowchart following the operation of FIG. 12 is a flowchart following the operation of FIG. The figure for demonstrating the other example of the mark recognition method, when the user points the desired mark 8 using the paper sheet 7 as a modification of this embodiment.

Hereinafter, embodiments will be described with reference to FIGS.
FIG. 1 is a block diagram showing a video conference system (communication system).
The video conference system is a communication system that allows remote conference participants (speakers) to make a voice call with a partner while viewing the other party's face images (for example, moving images) and document images on their terminal screens. . In this video conference system, as a data processing device 1 with a built-in camera, for example, a portable terminal device such as a desktop or laptop personal computer or a tablet terminal is connected via a public communication network (Internet, wireless communication network) 2. This is a remote conference system capable of high-speed bidirectional communication between the data processing devices 1.

  A document camera 3 that captures conference materials, a headset 4 that integrates a microphone and headphones, and a printing device 5 are wired or wirelessly connected to the data processing device 1. In this case, all or some of the conference participants (speakers) S1, S2, S3,... Are provided with the document processing camera 3 and the printing device 5 in addition to the data processing device 1 and the headset 4. In the illustrated example, a case where the document camera 3 and the printing device 5 are provided on the speakers S1 and S2 side is illustrated, and one of the speakers S1, S2, S3,... (Speaker S2). Is the chairman (mediator).

  The data processing device 1 compresses its own face image data acquired from its own built-in camera (not shown in FIG. 1), document image data acquired from the document camera 3, and its own audio data acquired from the headset 4. When the face image data, material image data, and voice data of the other party of the remote conference are received from the other data processing apparatus 1 and acquired, the image data and voice data are decompressed. It is processed and displayed on its own terminal screen, the headset 4 is output as voice, or the document image data is transmitted to the printing device 5 for printing.

FIG. 2 is a diagram for explaining the document camera 3 that is wired or wirelessly connected to the data processing apparatus 1 side.
The document camera 3 is a stand-type digital camera that photographs a conference material 6 and the like placed on a desk or the like from above (substantially directly above). An image is formed on an image sensor (CCD, CMOS, etc.), so that still images and moving images can be taken with high definition. The document camera 3 functions as a camera that captures the conference material 6 and also functions as a camera that captures the command input paper sheet 7 placed within the photographing range. The document camera 3 has an image data processing unit including a central processing unit, an image processing unit, and the like in addition to an image data generation unit including a photographing lens and an image sensor. May be provided on the data processing apparatus 1 side.

  The paper sheet 7 is a paper sheet on which indices (marks) 8 representing a plurality of types of instruction information (commands) that can be received by the application for each application (for example, a conference) are printed in an array. For example, it is issued from the printing device 5 for every meeting). In a state where the optimal paper sheet 7 for the conference is placed within the photographing range of the document camera 3, the user (conference participant) can use the fingertip to put a desired index (mark) 8 on the paper sheet 7. When a pointing operation is performed, the document camera 3 captures an image of the user pointing at the mark 8. The data processing apparatus 1 acquires the captured image from the document camera 3 and analyzes the captured image to determine which mark 8 has been designated by the user, and the command assigned to the mark 8 is determined. I try to input. The pointing operation may be performed by bringing the fingertip into contact with the mark 8 or approaching the mark 8, and the approaching condition does not matter as long as it is possible to recognize which mark 8 is designated by the user. Thus, the paper sheet 7 is used when the pointing operation to the mark 8 and the operation on the data processing apparatus 1 side (operation according to the type of the mark 8) are linked.

FIG. 3 is a diagram for explaining the paper sheet 7.
The paper sheet 7 is a belt-like paper sheet on which a plurality of types of marks 8 are arranged and printed. In the illustrated example, a case where rectangular marks 8 are arranged and printed in a 2 × 4 matrix is illustrated. On the upper right end side of the paper sheet 7, barcode information 9 indicating sheet identification information (sheet ID) is printed together with the sheet name “conference (1)”. The character string on each mark 8 clearly indicates the name of the command to be entered. For example, from the mark 7 on the left side of the upper row in the figure, “Material (1)”, “Material (2)”, “Speaker” (S1) ”and“ Speaker (S2) ”character strings (command names) are printed, and“ Speaker (S3) ”,“ Speaker ”,“ Approve ”,“ Print ”are displayed from the left mark 7 at the bottom. The character string (command name) is printed.

  Here, the mark 8 of “material (1)” and “material (2)” is a mark designated by the user when requesting transmission of the material data. The "speaker (S1)", "speaker (S2)", and "speaker (S3)" marks 8 are marks designated by the user when requesting transmission of a speaker's video (face image). . The “speak” mark 8 is a mark designated by the user when requesting mediation from the chairman when he / she speaks. The “agree” mark 8 is a mark designated by the user when telling the speaker that he / she agrees, such as when the content of the other party's speech is good. In this way, the various marks 8 printed on the paper sheet 7 and the operations corresponding to the types of the marks 8 are associated one-to-one. The various marks 8 on the paper sheet 7 are different for each application and are associated with different operations for each application.

FIG. 4 is a diagram for explaining a mark recognition method when the user performs a pointing operation of pointing the desired mark 8 using the paper sheet 7.
When the document camera 3 shoots the user pointing at the mark 8 on the paper sheet 7, the data processing apparatus 1 determines which mark 8 is instructed based on the captured image as follows. To discriminate (recognize). That is, the entire rectangular mark 8 on the paper sheet 7 is constituted by the red rectangular frame (red frame). The document camera 3 captures only red by the color filter function.

  The data processing apparatus 1 extracts a red frame image having a size closest to the rectangular frame of the mark 8 from the captured images acquired from the document camera 3. In the illustrated example, the case where the red frame image of the mark 7 of the material (1) and the material (2) is extracted is illustrated, and the red frame is missing from the red frame image by the user's pointing (finger A red frame image having a portion cut off at () is specified, and the red frame image is recognized as the mark 8 pointed by the user. In the illustrated example, the mark 8 of the material (2) among the marks 7 of the material (1) and the material (2) is recognized as the mark 8 designated by the user by the pointing operation. This makes it possible to obtain the document (2) and display it on its own terminal screen.

FIG. 5 is a block diagram showing a basic configuration of a data processing apparatus with a built-in camera (for example, a notebook personal computer) 1.
The CPU 11 that forms the core of the data processing device 1 is a central processing unit that operates by supplying power from the power supply unit 12 and controls the overall operation of the data processing device 1 according to various programs in the storage unit 13. . The storage unit 13 stores various programs and information, and a program area in which programs for realizing the present embodiment, various applications, and the like are stored according to the operation procedure illustrated in FIGS. 10 to 12. In addition to 13a, it has a data area 13b for storing various data required by the data processing apparatus 1 and a work area 13c for temporary storage. The data area 13b includes a configuration file X and a mark table, which will be described later. Y, sheet management table Z, and the like are stored.

  The communication unit 14 is connected to a broadband public communication network (for example, optical communication connection) during operation of an electronic mail function, an Internet connection function, or the like. The key operation unit 15 includes various push button type keys and inputs data and commands. A pointing device (mouse) 16 selects and designates icons and buttons on the screen and inputs data and commands. Here, in the present embodiment, as a means for inputting a command that can be accepted by an application that is currently being executed, the display screen is not limited to the command input using the paper sheet 7 as described above, but is displayed using the mouse 16. It is also possible to input a similar command by designating the upper icon or button as a point, or to input a similar command by key operation of the key operation unit 15.

  The display unit 17 displays a work screen or the like on a high-definition liquid crystal display or an organic EL (Electro Luminescence) display having screens with different aspect ratios. The built-in camera 18 is a digital camera that captures facing users (conference participants). Although not shown, a subject image from the photographing lens is formed on an image sensor (CCD, CMOS, etc.) to form a still image, It is possible to shoot moving images with high definition. The external connection unit 19 has a USB (Universal Serial Bus) connector, and can be connected to a document camera 3, a headset 4, a printing device 5, and the like that captures conference materials and the like.

FIG. 6 is a diagram for explaining a configuration file X of a screen displayed during application execution.
The configuration file X is for application screen layout, and is a file in which the screen layout configuration is set, such as where to place icons and what size to place them on the screen displayed when the application is executed. is there. Such a configuration file X is provided for each screen of an application. When a plurality of screens are used in one application, a plurality of configuration files X are provided corresponding to the number of screens. As described above, when a plurality of configuration files X are provided in one application, the paper sheet 7 is prepared for each configuration file X. In other words, the paper sheet 7 is prepared for each application as described above. Furthermore, when a plurality of configuration files X are provided in one application, the paper sheet 7 is prepared for each configuration file X. Yes.

  In the illustrated example, “configuration file (1) conference (1)”, “configuration file (2) conference (2)”, “configuration file (3) conference” are used for screen layouts of various applications A, B, and C. (3) ”is provided. The “configuration file (1) meeting (1)” is set with the internal table (11) and the internal table (12), and the “configuration file (2) meeting (2)” is set with the internal table (21), The internal table (22) and the internal table (23) are set, and the internal table (31) and the internal table (32) are set in the “configuration file (3) meeting (3)”.

FIG. 7 is a diagram for specifically showing the setting contents of the internal table provided in the configuration file X and for explaining the contents of the mark table Y automatically generated based on the setting contents of the internal table. .
The illustrated example shows the internal table (11) and internal table (12) of “configuration file (1) meeting (1)” as internal tables of the configuration file X. The internal table (11) is “command”, “ Each item includes “mouse instruction screen coordinate position” and “key”. “Command” indicates an icon displayed on the application screen and a command assigned to the icon. In the illustrated example, “speak”, “agree”, “print”,... Are set. Shows the case.

  The “mouse instruction screen coordinate position” indicates the coordinate position of an icon (“speech”, “agree”, “print”,...) When an icon is selected by a mouse operation. FIG. 8 is a diagram showing a display example of icons on the application screen. In the illustrated application screen, face images of a plurality of opponents are arranged and displayed, and materials (1) and (2) are arranged and displayed as various materials, and are further displayed at predetermined positions on the application screen. The state in which a plurality of icons are displayed is illustrated. The “mouse instruction screen coordinate position” indicates the display coordinate position of the icon on the application screen. “Key” indicates a key operation instead of selecting an icon by a mouse operation, and “F1”, “F2”, “F3”,... Corresponding to the icons “speech”, “agree”, “print”. Each key is assigned and set.

  The internal table (12) has items of “command”, “mouse instruction screen coordinate position”, and “key”, and “command” includes “document (1)”, “document (2)”, “ “Material (3)”, “Sender (S1)”, “Sender (S2)”, “Sender (S3)”,... Are set. The mark table Y is automatically generated based on the setting contents of the internal table (11) and the internal table (12). The mark table Y is an information sheet for storing and managing, for each application, a paper sheet 7 in which a plurality of types of commands that can be received during execution of an application and a plurality of types of marks 8 that represent the commands are associated with each other. Each item includes “command” and “mark”. In the mark table Y, a sheet ID for identifying each paper sheet 7 is stored.

  In the illustrated example, the mark table Y corresponding to the internal table (11) includes “M1”, “M2” as “marks” corresponding to “commands” of “speak”, “agree”, “print”,. ”,“ M3 ”,... Are set, and the mark table Y corresponding to the internal table (12) includes“ material (1) ”,“ material (2) ”,“ material (3) ”,“ speaker ( “M11”, “M12”, “M13”, “M14”, “M15” are “marks” corresponding to “commands” of “S1)”, “speaker (S2)”, and “speaker (S3)”. , “M16” is set. Furthermore, “ID1” is set in the mark table Y as “sheet ID” for identifying it.

FIG. 9 is a diagram for explaining the sheet management table Z.
The sheet management table Z is a table for storing and managing information related to the mark table Y every time the mark table Y is automatically generated. The items “configuration file ID”, “internal table ID”, and “sheet ID” are stored in the sheet management table Z. Have. Thus, the sheet management table Z stores and manages the sheet ID that associates each of the paper sheets 7 with each of the applications as link information. In the illustrated example, the “configuration file ID” is “application A (conference ( 1)) ”,“ Internal table ID ”is set to“ (11) ”,“ Sheet ID ”is set to“ ID1 ”, and“ Configuration file ID ”is set to“ Application A (Conference (1)) ”,“ Internal “Table ID” is set to “(12)”, “Sheet ID” is set to “ID2”, “Configuration File ID” is set to “Application A (Conference (2))”, and “Internal Table ID” is set to “(21)”. ", Where the sheet management table Z having" ID3 "as the" sheet ID "is set.

  Next, the operation concept of the data processing operation 1 in the present embodiment will be described with reference to the flowcharts shown in FIGS. Here, each function described in these flowcharts is stored in the form of a readable program code, and operations according to the program code are sequentially executed. Further, it is possible to sequentially execute the operation according to the above-described program code transmitted via a transmission medium such as a network. In other words, in addition to the recording medium, an operation unique to the present embodiment can be executed using a program / data supplied externally via a transmission medium. 10 to 12 are flowcharts showing an outline of the operation of the characteristic part of the present embodiment in the entire operation of the data processing apparatus 1. When the flow of FIGS. Return to the main flow (not shown) of the overall operation.

10 to 12 are flowcharts for explaining the operation (characteristic operation of the present embodiment) of the data processing apparatus 1 that is started in response to the selection of the application to be activated.
First, the CPU 11 of the data processing apparatus 1 starts an application selected and designated by a user operation or the like (step S1 in FIG. 10), and then displays the default screen on the display unit 17 (step S2). A default setting is made on the internal table of the configuration file X based on the display content of the default screen (step S3). In this state, it is checked whether or not a screen configuration change setting is instructed by a user operation (step S4). Here, when the user sets the screen layout of the selected application for, for example, the meeting (1), the screen layout is appropriately changed and set for the meeting (1) before the meeting is held. To change the screen configuration.

  Now, when a screen configuration change setting is instructed by a user operation (YES in step S4), processing for arbitrarily changing the screen layout in accordance with the user operation, such as addition or deletion of an icon or button, is performed (step S5). . When the screen layout is changed by this, it is checked whether or not the user operation is instructed to register the changed screen layout as the configuration file X (step S6). If there is no registration instruction (NO in step S6), the process returns to step S4 described above. If there is a registration instruction (YES in step S6), icons and buttons on the application screen are extracted (step S6). S7) A process of setting an internal table is performed based on the extracted icon and button (step S8). In this case, for each extracted icon and button, the command assigned to that icon and button is acquired, and in addition to setting for each item of “command” and “mouse instruction screen coordinate position”, a specific icon The “key” item for inputting the icon assigned to the button and the command assigned to the button is set.

  Next, it is checked whether or not the end of the screen configuration change setting is instructed by the user operation (step S9), and the process returns to the above step S4 until the end of the change setting is instructed (NO in step S9). Is repeated (steps S4 to S9), but when the end of change setting is instructed (YES in step S9), the configuration file X including the internal table set as described above is stored and the configuration file ID is also stored. Is generated and added to the configuration file X (step S10). Next, it is checked whether the creation of the mark table (information sheet) Y is instructed by the user operation (step S11). If there is no such creation instruction (NO in step S11), the flow of FIGS. .

  If there is an instruction to create the mark table Y (YES in step S11), a mark 8 corresponding to each icon extracted in step S7 is generated (step S12), and the mark table Y is based on the mark 8. Is automatically generated (step S13). In this case, for each extracted icon and button, a mark table Y in which the command assigned to the icon and button is associated with the mark 8 generated as described above is automatically generated. Then, a sheet ID is generated and stored in the mark table Y (step S14). When the mark table Y is generated in this way, a record having items of “configuration file ID”, “internal table ID”, and “sheet ID” as information related to the sheet is generated and additionally registered in the sheet management table Z ( Step S15).

  Then, the “sheet ID” is converted into the barcode information 9, and each “mark” in the mark table Y together with the barcode information 9 is transmitted from the external connection unit 19 to the printing apparatus 5 to be printed out (step). S16). In this case, on the printing apparatus 5 side, barcode information is printed at a predetermined position in accordance with a predetermined printing form, and “marks” in the mark table Y are sequentially arranged and printed. For example, “mark M1” and “mark M2” are arranged and printed. In this case, “mark” is printed as a command name on “mark M1”, and “agree” is printed as a command name on “mark M2”. .

  As a result, a paper sheet 7 is automatically issued. This paper sheet 7 is generated for each application (and for each screen (configuration file X) displayed during execution of the application). That is, when there are a plurality of screens displayed during execution of an application, icons and buttons (commands) are acquired for each application screen, and a plurality of screens are acquired based on the icons and buttons acquired for the plurality of screens. A seed mark is generated, a mark table Y is generated for each of a plurality of screens, and a paper sheet 7 is printed and issued for each of the plurality of screens. When such printing is finished, the flow of FIGS.

  On the other hand, if the change setting of the screen configuration is not instructed (NO in step S4), the flow proceeds to the flow of FIG. 11 and it is checked whether the reading of the configuration file X is instructed by the user operation (step S17). Here, when the conference is held, the user instructs the user to read the configuration file X set in advance for the conference. When reading of the configuration file X is instructed (YES in step S17), a list of each configuration file X is displayed with reference to the sheet management table Z (step S18). When an arbitrary configuration file X is selected and designated by the user operation from the list screen (step S19), the internal table is read from the selected and designated configuration file X with reference to the sheet management table Z (step S19). S20), an application screen corresponding to the setting content of the internal table is displayed on the display unit 17 (step S21).

  In a state where the desired conference application screen is displayed in this way, a photographed image is acquired from the document camera 3 (step S22), and the photographed image is analyzed to analyze any one of the sheets on the paper sheet 7. It is checked whether the mark 8 is designated by the pointing operation by the user (step S23). Here, if the mark 8 is not designated by the pointing operation (NO in step S23), it is checked whether any key operation or mouse operation has been performed (step S24), and either key operation or mouse operation is performed. If not (NO in step S24), the process returns to step S22 described above.

  When a key operation or a mouse operation is detected (YES in step S24), the key is associated with the operation key depending on which key is operated and which icon or button position is operated by the mouse. Or a command associated with each of the position (icon or button) operated by the mouse is read from the internal table (step S25), and the execution of the operation corresponding to the command is performed on the currently executing application. An instruction is given (step S26).

  It is checked whether or not it is necessary to shift to another internal table by executing the operation of such an application (step S27). When the shift to another internal table is unnecessary (NO in step S27), the process returns to the above-described step S22. When the operation is waited, but it is necessary to move to another internal table (YES in step S27), the other internal table is designated as a reading target (step S28), and then the process returns to step S20 described above. The setting contents of the table are read, and the above operation is repeated thereafter.

  If any mark 8 on the paper sheet 7 is designated by pointing operation by analyzing the captured image from the document camera 3 (YES in step S23), the flow proceeds to the flow of FIG. By analyzing, the barcode information on the paper sheet 7 is read and converted into a sheet ID (step S29). Then, the “sheet ID” associated with the internal table of the configuration file X currently being executed is read from the sheet management table Z and compared with the sheet ID read from the paper sheet 7 to check whether they match ( Step S30). That is, it is checked whether the paper sheet 7 placed in front of the document camera 3 is a paper sheet 7 corresponding to the currently executed application screen.

  If a mismatch between the two is detected (NO in step S30), it is not a paper sheet 7 corresponding to the application screen currently being executed, so a priority flag (not shown) set in advance is referred to and the paper is It is checked whether the sheet ID read from the sheet 7 has priority over the sheet ID read from the sheet management table Z (step S31). That is, it is checked whether the paper sheet 7 placed in front of the document camera 3 has priority over the application screen currently being executed. If no priority is set (NO in step S30), that is, the sheet management table Z In the case where priority is given to the sheet ID read out from, a message for requesting replacement of the paper sheet 7 is displayed (step S36), and then the process returns to step S22 in FIG.

  If priority is given to the sheet ID read from the paper sheet 7 (YES in step S31), the sheet management table Z is referred to based on the sheet ID and the configuration file X associated therewith is read (step S31). S32), it is checked whether it is the same as the application currently being executed (step S33). That is, the sheet IDs do not match, but it is checked whether the applications are the same. If they are the same application (YES in step S33), for example, the sheet ID read from the paper sheet 7 is “ID1”, the sheet management table Z If the sheet ID read from “ID2” is “ID2”, the application is “A”, and the application is the same. Therefore, the process proceeds to the next step S34, the internal table is read from the configuration file X, and the layout is displayed (step S35). If the two applications are not the same (NO in step S33), the application corresponding to the sheet ID read from the paper sheet 7 is activated (step S34), and then the internal table is read from the configuration file X and displayed in layout. (Step S35).

  If the sheet ID read from the paper sheet 7 matches the sheet ID read from the sheet management table Z (YES in step S30), the user can analyze the captured image of the paper sheet 7 by the document camera 3 to analyze the user. The mark 8 on the paper sheet 7 is recognized (step S37), and by referring to the mark table Y based on the mark 8 (step S38), a command corresponding to the mark 8 is read (step S38). ) And instructing the currently executing application to execute the operation corresponding to the command (step S39). Thereafter, the process proceeds to step S27 in FIG.

  As described above, in the present embodiment, the data processing apparatus 1 is a paper sheet on which each index (mark) 8 corresponding to a plurality of types of instruction information (commands) that can be received by an application that is currently being executed is arranged and printed. 7, when the user points to an arbitrary mark 8, the mark 8 is discriminated and a command corresponding to the mark 8 is selected by referring to the mark table Y based on the mark 8. Accordingly, the application that is currently being executed is made to perform the operation according to the above, so that the arrangement of the marks 8 on the paper sheet 7 can be made an application unit. By simply pointing at the mark 8, the operation assigned to the mark 8 can be executed, further improving the operability. It is possible to become. In other words, the entire screen during application execution (during a meeting) is complicated and difficult to see as shown in FIG. 8, and it is necessary to operate at an accurate position. However, since only a pointing operation of pointing an arbitrary mark 8 on the paper sheet 7 in application units is required, it is possible to expect improvement in operability, and it is possible to concentrate on the conference.

  The CPU 11 identifies an application corresponding to the sheet ID by referring to the sheet management table Z based on the barcode information (sheet ID) acquired from the paper sheet 7, and when the application is currently being executed, Since an arbitrary mark 8 designated by the user on the sheet 7 is discriminated, an application can be specified from the paper sheet 7, and a mark using the paper sheet 7 when the application is currently being executed. The designation can be validated, and the correspondence between the paper sheet 7 and the application can be maintained.

  When the application specified from the paper sheet 7 is not currently being executed, the fact is notified by displaying a message. Therefore, the user can exchange the paper sheet 7 and the application by exchanging the paper sheet 7. It is possible to maintain the correspondence relationship.

  Since the application specified from the paper sheet 7 is not currently being executed, the application is started, so that the user maintains the correspondence between the paper sheet 7 and the application without performing any special operation. It becomes possible to do.

  As a means for inputting a command that can be accepted by an application that is currently being executed, the command input is not limited to the command input using the paper sheet 7, but the command is input by pointing the icon on the display screen using the mouse 16. Therefore, the user who is accustomed to the operation can select the input means, which improves usability.

  As a means for inputting a command that can be accepted by an application that is currently being executed, a command can be input by a key operation as well as a command input using a paper sheet 7. The input means can be selected, which improves usability.

  The mark table Y includes, for each configuration file X in one application, a plurality of types of commands that can be accepted when the configuration file X is executed by the application, and a plurality of types of marks 8 that represent the commands. Since the storage management is performed in association with each other, the configuration of the screen layout displayed when the application is executed can be associated with a plurality of types of marks 8.

  Since a plurality of types of commands that can be accepted during execution of the application are acquired, and the mark 8 representing each command is generated and stored in the mark table Y, the mark table Y can be automatically generated. Become.

  When the document camera 3 captures a state in which an arbitrary mark 8 is designated by the user on the paper sheet 7, the data processing apparatus 1 determines which mark 8 is designated by analyzing the captured image. As a result, the user can perform application processing only by pointing in the direction of the mark 8.

  A plurality of types of commands that can be accepted during the execution of the application are acquired, a plurality of types of marks 8 representing each command are generated, and the plurality of types of commands and a plurality of types of marks are associated with each other to create a mark table Y Since the generated plural types of marks are output in an array, the user can simply assign an arbitrary mark 8 on the automatically generated paper sheet 7 and assign it to the mark 8. Therefore, it is possible to expect a further improvement in operability.

  The plurality of types of commands that can be accepted during the execution of the application can be commands assigned to icons and buttons on the display screen that can be designated by operating the mouse 16, so The means for inputting a command that can be accepted by the application is not limited to the command input using the paper sheet 7, but the command input using the mouse 16 is also possible. And can be expected to improve usability.

  Since the plurality of types of commands that can be accepted during the execution of the application can be commands assigned to the keys on the key operation unit 15, means for inputting commands that can be accepted by the currently running application As described above, not only command input using the paper sheet 7 but also command input by key operation is possible, and a user who is accustomed to the operation can select an input means, and is expected to improve usability. It becomes possible.

  A plurality of types of commands that can be accepted for each of a plurality of screens (configuration file X) displayed during application execution are acquired, and a plurality of types of marks are generated based on the commands acquired for each of the plurality of screens. Multiple types of commands and multiple types of marks are associated with each other and stored in the mark table Y, and the generated multiple types of marks are output in an array for each of the multiple screens. The paper sheet 7 can be automatically generated for each of a plurality of screens displayed therein.

  In the above-described embodiment, when a pointing operation is performed by pointing the user to the desired mark 8 using the paper sheet 7, each mark is recognized as shown in FIG. The red frame image having a portion where the red frame is missing (cut with a finger) by pointing out of 8 is identified, and the red frame image is recognized as the mark 8 pointed by the user. When there are a plurality of red frame images that are partially missing, one of the plurality of red frame images may be selected as follows.

  For example, as shown in FIG. 13, when marks 8 are arranged and printed over 3 rows × 4 columns on a paper sheet 7, a plurality of marks 8 (filled mark portions in the drawing) by pointing, that is, In the illustrated example, the mark 8 in the third column from the left in the second row from the top and a part of the mark 8 in the third row, the third column, and the fourth column are missing (cut with a finger). In such a case, among the plurality of marks 8 that are partially missing, for example, the uppermost mark 8, in the illustrated example, the mark 8 in the second row and the third column is specified, and the mark 8 is pointed by the user. The mark 8 may be recognized. Depending on the pointing direction, the leftmost mark 8 or the rightmost mark 8 may be recognized as the pointed mark 8.

  In the above-described embodiment, the paper sheet 7 on which a plurality of types of marks 8 are arranged and printed is printed and issued. However, a plurality of types of marks 8 may be arranged and displayed on the display unit 17. If a plurality of types of marks 8 are arranged and displayed in this way, the printing device 5 becomes unnecessary.

  In the above-described embodiment, the state in which the user is pointing the mark 8 on the paper sheet 7 is photographed by the document camera 3, but a touch display unit (not shown) in which a touch panel is arranged on the display panel is provided. In the state where a plurality of types of marks 8 are arranged and displayed on the touch display unit, the touched mark may be recognized when the user touches any mark display position. If a plurality of kinds of marks 8 are arranged and displayed in this way, an imaging device such as the printing device 5 and a document camera becomes unnecessary.

  In the above-described embodiment, the mark 8 is shown as a plurality of types of indices expressing the command. However, the mark 8 is not limited to the mark, and may be a character string, a symbol, or the like. For example, when the number of indices is small, the top / bottom / left / right positions / regions may be a plurality of kinds of indices.

  Moreover, although the case where it applied to the video conference system was illustrated in embodiment mentioned above, it may be a communication system other than a video conference, Furthermore, not only a system but a data processing apparatus single-piece | unit may be sufficient. Further, the data processing device may be a portable terminal device such as a desktop or notebook personal computer or a tablet terminal, or may be a mobile phone such as a smartphone, an electronic game, a music player, or a digital camera. . Also,

  Further, the “apparatus” and “unit” shown in the above-described embodiments may be separated into a plurality of cases by function, and are not limited to a single case. In addition, each step described in the above-described flowchart is not limited to time-series processing, and a plurality of steps may be processed in parallel or separately.

The embodiment of the present invention has been described above. However, the present invention is not limited to this, and includes the invention described in the claims and the equivalent scope thereof.
Hereinafter, the invention described in the claims of the present application will be appended.
(Appendix)
(Claim 1)
The invention described in claim 1
A data processing apparatus that includes a plurality of applications and performs corresponding data processing according to the execution of each application,
Index storage means for storing and managing a plurality of types of instruction information that can be accepted for each application during execution of the application and a plurality of types of indexes that express each instruction information;
Discriminating means for discriminating an index arbitrarily designated by a user operation from within an output medium in which each index corresponding to a plurality of types of instruction information that can be received by an application currently being executed is arrayed,
A selection unit that selects instruction information corresponding to the index by referring to the index storage unit based on the index determined by the determination unit;
Control means for performing control for causing the currently executing application to perform an operation according to the instruction information selected by the selection means;
A data processing apparatus comprising:
(Claim 2)
The invention according to claim 2 is the data processing device according to claim 1,
The index storage means stores and manages, for each application, an information sheet that associates a plurality of types of instruction information that can be received during execution of an application with a plurality of types of indexes that express each instruction information, and the information Stores and manages link information that associates individual sheets with individual applications.
An acquisition means for acquiring the link information from the output medium in which the plurality of types of indicators are output in sequence and the link information is output;
A specifying unit for specifying an application associated with the link information by referring to the index storage unit based on the link information acquired by the acquiring unit;
Further comprising
The determining means determines an index arbitrarily designated by a user operation from within the output medium when the application specified by the specifying means is currently being executed;
This is a data processing apparatus characterized by being configured as described above.
(Claim 3)
The invention according to claim 3 is the data processing apparatus according to claim 2,
In the case where the application specified by the specifying means is not currently being executed, it further comprises notification means for notifying that effect.
This is a data processing apparatus characterized by being configured as described above.
(Claim 4)
The invention according to claim 4 is the data processing apparatus according to claim 2,
If the application specified by the specifying means is not currently being executed, the apparatus further comprises start means for starting the application.
This is a data processing apparatus characterized by being configured as described above.
(Claim 5)
The invention according to claim 5 is the data processing device according to any one of claims 1 to 4,
Pointing input means for inputting instruction information that can be accepted by the currently running application by specifying the icon or button displayed on the display screen,
The control unit is configured to cause an application currently being executed to perform an operation according to the instruction information in accordance with either the instruction information input by the pointing input unit or the instruction information selected by the selection unit. Do control,
This is a data processing apparatus characterized by being configured as described above.
(Claim 6)
The invention according to claim 6 is the data processing apparatus according to any one of claims 1 to 4,
Key input means for inputting instruction information that can be accepted by the currently running application by key operation,
The control means is for causing the currently executing application to perform an operation according to the instruction information in accordance with either the instruction information input by the key input means or the instruction information selected by the selection means. Do control,
This is a data processing apparatus characterized by being configured as described above.
(Claim 7)
The invention according to claim 7 is the data processing device according to any one of claims 1 to 6,
The index storage means includes, for each configuration file in one application, a plurality of types of instruction information that can be received when the configuration file is executed by the application, and a plurality of types of indexes that express the instruction information. To store and manage
This is a data processing apparatus characterized by being configured as described above.
(Claim 8)
The invention according to claim 8 is the data processing apparatus according to any one of claims 1 to 7,
It further comprises an index generating means for acquiring a plurality of types of instruction information that can be accepted during execution of the application and generating an index that expresses each instruction information.
The index storage means stores and manages the acquired instruction information and the generated indexes in association with each other.
This is a data processing apparatus characterized by being configured as described above.
(Claim 9)
The invention according to claim 9 is the data processing device according to any one of claims 1 to 8,
The discriminating means captures an image taken by the imaging means in a state in which an arbitrary index is designated by a user operation from a paper medium on which the indices are arranged and printed or a display medium on which the indices are arranged and displayed. To determine which indicator was specified by parsing,
This is a data processing apparatus characterized by being configured as described above.
(Claim 10)
The invention according to claim 10 is the data processing device according to any one of claims 1 to 8,
The determination means determines which index is specified based on a position touched by a user operation on the touch display unit on which the respective indexes are arrayed and displayed.
This is a data processing apparatus characterized by being configured as described above.
(Claim 11)
The invention according to claim 11
A data processing apparatus that includes a plurality of applications and performs corresponding data processing according to the execution of each application,
An acquisition means for acquiring multiple types of instruction information that can be accepted during execution of the application;
For each of a plurality of types of instruction information acquired by the acquisition unit, a generation unit that generates a plurality of types of indicators expressing each instruction information;
Index storage means for storing and managing a plurality of types of instruction information acquired by the acquisition means and a plurality of types of indexes generated by the generation means;
An output means for outputting a plurality of types of indices generated by the generating means;
A data processing apparatus comprising:
(Claim 12)
The invention according to claim 12 is the data processing apparatus according to claim 11,
The plurality of types of instruction information that can be accepted during the execution of the application is instruction information assigned to each icon or each button on the display screen that can be designated with a pointing input means.
This is a data processing apparatus characterized by the above.
(Claim 13)
The invention according to claim 13 is the data processing apparatus according to claim 11 or 12,
The multiple types of instruction information that can be accepted during the execution of the application are designation information assigned to each key on the key input means.
This is a data processing apparatus characterized by the above.
(Claim 14)
The invention according to claim 14 is the data processing device according to any one of claims 11 to 13,
The acquisition means acquires a plurality of types of instruction information that can be received for each of a plurality of screens displayed during execution of the application,
The generation means generates a plurality of types of indicators based on each instruction information acquired by the acquisition means for each of a plurality of screens displayed during execution of the application,
The index storage unit stores and manages a plurality of types of instruction information acquired by the acquisition unit and a plurality of types of indexes generated by the generation unit for each of a plurality of screens displayed during execution of the application. And
The output means outputs a plurality of types of indices generated by the generation means for each of a plurality of screens displayed during execution of the application.
This is a data processing apparatus characterized by being configured as described above.
(Claim 15)
The invention according to claim 15 is the data processing device according to any one of claims 11 to 15,
The output means is a data processing apparatus characterized in that a plurality of types of indices generated by the generating means are arranged and printed on a paper medium or arranged and displayed on a display medium.
(Claim 16)
The invention described in claim 16
Against the computer,
A function for storing and managing a plurality of types of instruction information that can be accepted during the execution of the application for each application and a plurality of types of indexes that express each instruction information;
A function for discriminating an index arbitrarily designated by a user operation from within an output medium in which each index corresponding to a plurality of types of instruction information that can be received by an application currently being executed is output;
A function of selecting instruction information corresponding to the index by referring to the index storage means based on the determined index;
A function of performing control for causing the currently executing application to perform an operation according to the selected instruction information;
It is a program for realizing.
(Claim 17)
The invention described in claim 17
Against the computer,
A function to acquire multiple types of instruction information that can be accepted during application execution,
For each of the acquired plural types of instruction information, a function of generating a plurality of types of indicators expressing each instruction information;
A function of storing and managing the acquired plural types of instruction information and the generated plural types of indices in association with each other;
A function of array-outputting the generated plural types of indices;
It is a program for realizing.

1 Data Processing Device 3 Document Camera 5 Printing Device 11 CPU
DESCRIPTION OF SYMBOLS 13 Memory | storage part 15 Key operation part 16 Mouse | mouth 17 Display part 18 Built-in camera 19 External connection part X Configuration file Y Mark table Z Sheet management table

Claims (6)

Among a plurality of display choices displayed on the front示画surface, the position indicator with respect to the display position of an arbitrary display choices selection process corresponding to the display choices the position indicator as a selection instruction for displaying choices of the display position Selection processing means for executing
On medium in which a plurality of marks are arranged corresponding to the plurality of display choices when the position of any of the mark is indicated by the user, determining means for determining the mark indicated position,
Comprising
The selection processing unit refers to a storage unit that stores a correspondence relationship between a plurality of marks arranged on the medium and the plurality of display options, and displays display options corresponding to the marks determined by the determination unit. By specifying the mark instruction on the medium as an instruction instead of a position instruction for the display position of the display option corresponding to the mark, the selection process corresponding to the display option is executed.
The data processing apparatus according to claim and this. Medium discriminating means for discriminating whether or not the medium on which the plurality of marks are arranged is a medium corresponding to display contents of a plurality of display options displayed on the display screen;
Further comprising
The determining means determines the mark when the medium determining means determines that the medium is a medium corresponding to the display content,
When the medium determining unit determines that the medium is a medium corresponding to the display content, the selection processing unit converts the indication of the mark on the medium into a position instruction with respect to the display position of the display option. Execute the selection process as a substitute instruction;
The data processing apparatus according to claim 1. Informing that the medium is not a medium corresponding to the display content displayed on the display screen when the medium determination unit determines that the medium is not a medium corresponding to the display content ;
The data processing apparatus according to claim 2 . If the medium in the medium discrimination means is not determined to be the medium corresponding to the display content, the display contents displayed on the display screen, the display option corresponding to the medium in which the mark is arranged Change the display contents to include
The data processing apparatus according to claim 2 . The storage means stores a correspondence relationship between a plurality of display options and a plurality of marks for each of a plurality of applications ,
The medium is a plurality of media prepared for each application,
On the display screen, a plurality of display options corresponding to an arbitrarily selected application are displayed.
The data processing device according to any one of claims 1-4, characterized in that. A program for controlling a computer of a data processing device that executes data processing by selecting a display option displayed on a display screen,
The computer,
Among a plurality of display choices displayed on the display screen, the position indicator with respect to the display position of an arbitrary display choices a selection process corresponding to the display choices the position indicator as a selection instruction for displaying choices of the display position Selection processing means to execute ,
On media disposed a plurality of marks corresponding to the plurality of display choices or when the position of the mark is indicated by the user, determining means for determining marks indicated position,
Function as
The selection processing unit refers to a storage unit that stores a correspondence relationship between a plurality of marks arranged on the medium and the plurality of display options, and displays display options corresponding to the marks determined by the determination unit. By specifying the mark instruction on the medium as an instruction instead of a position instruction for the display position of the display option corresponding to the mark, the selection process corresponding to the display option is executed.
A program characterized by that .

Images