WO2013073100A1 - Display control apparatus, display control method, and program - Google Patents
Display control apparatus, display control method, and program Download PDFInfo
- Publication number
- WO2013073100A1 WO2013073100A1 PCT/JP2012/006382 JP2012006382W WO2013073100A1 WO 2013073100 A1 WO2013073100 A1 WO 2013073100A1 JP 2012006382 W JP2012006382 W JP 2012006382W WO 2013073100 A1 WO2013073100 A1 WO 2013073100A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- display
- user interface
- information processing
- information
- representation
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
- G06F1/16—Constructional details or arrangements
- G06F1/1613—Constructional details or arrangements for portable computers
- G06F1/1633—Constructional details or arrangements of portable computers not specific to the type of enclosures covered by groups G06F1/1615 - G06F1/1626
- G06F1/1684—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
- G06F1/1686—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675 the I/O peripheral being an integrated camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/31—User authentication
- G06F21/36—User authentication by graphic or iconic representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/0482—Interaction with lists of selectable items, e.g. menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/038—Indexing scheme relating to G06F3/038
- G06F2203/0383—Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2340/00—Aspects of display data processing
- G09G2340/04—Changes in size, position or resolution of an image
- G09G2340/0464—Positioning
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2354/00—Aspects of interface with display user
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2370/00—Aspects of data communication
- G09G2370/06—Consumer Electronics Control, i.e. control of another device by a display or vice versa
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2380/00—Specific applications
- G09G2380/14—Electronic books and readers
Definitions
- the present disclosure relates to a display control apparatus, a display control method, and a program.
- AR augmented reality
- a remote desktop technology is known as the technology in which the execution of a command by an execution terminal is controlled using an operation terminal according to a remote operation. If the remote desktop technology is used, for example, a right to execute the command can be assigned to the operation terminal. For example, the execution of the command can be controlled using the operation terminal if the execution of the command is permitted, but the control of the command execution using the operation terminal is restricted if the command execution is restricted.
- PTL Patent Literature
- the disclosure is directed to an information processing system, comprising: one or more processing units that acquire captured image data corresponding to a user interface displayed on a first display, the user interface including a plurality of graphic representations each corresponding to a respective function; and control a second display to display a representation of the user interface based on a recognition process performed on the user interface, the representation of the user interface indicating whether a function corresponding the plurality of the graphic representations is capable of being accessed.
- the system may include a user interface that receives a selection of one of the plurality of graphic representations displayed in the representation of user interface displayed on the second display, and a communication interface, wherein the one or more processing units control the communication interface to transmit information corresponding to the selection of the one of the plurality of graphic representations displayed in the representation of user interface displayed on the second display to an information processing apparatus including the first display.
- the disclosure is directed to an information processing method performed by an information processing system, the method comprising: acquiring captured image data corresponding to a user interface displayed on a first display, the user interface including a plurality of graphic representations each corresponding to a respective function; and controlling a second display to display a representation of the user interface based on a recognition process performed on the user interface, the representation of the user interface indicating whether a function corresponding the plurality of the graphic representations is capable of being accessed.
- the disclosure is directed to a non-transitory computer-readable medium including computer program instructions, which when executed by an information processing system, cause the information processing system to perform a process, the process comprising: acquiring captured image data corresponding to a user interface displayed on a first display, the user interface including a plurality of graphic representations each corresponding to a respective function; and controlling a second display to display a representation of the user interface based on a recognition process performed on the user interface, the representation of the user interface indicating whether a function corresponding the plurality of the graphic representations is capable of being accessed.
- Fig. 1 is an explanatory diagram illustrating a configuration of a display control system according to an embodiment of the present disclosure.
- Fig. 2 is a block diagram illustrating a functional configuration example of an execution terminal.
- Fig. 3 is a block diagram illustrating a functional configuration example of an operation terminal.
- Fig. 4 is a diagram illustrating an example of information that is transmitted by the execution terminal to the operation terminal.
- Fig. 5 is a diagram illustrating display control by a display control unit of the operation terminal.
- Fig. 6 is a diagram illustrating an example of a screen displayed according to the display control by the display control unit of the operation terminal.
- Fig. 7 is a diagram illustrating transmission control by a transmission control unit of the operation terminal.
- Fig. 1 is an explanatory diagram illustrating a configuration of a display control system according to an embodiment of the present disclosure.
- Fig. 2 is a block diagram illustrating a functional configuration example of an execution terminal.
- Fig. 3 is a block diagram
- Fig. 8 is a diagram illustrating an example of a command execution result.
- Fig. 9 is a flowchart illustrating a flow of an operation of a display control system according to an embodiment of the present disclosure.
- Fig. 10 is a diagram illustrating an example of transmission information according to a modified example of the embodiment of the present disclosure.
- Fig. 11 is a flowchart illustrating a flow of an operation of a display control system according to the modified example of the embodiment of the present disclosure.
- Fig. 1 is an explanatory diagram illustrating a configuration of the display control system according to the embodiment of the present disclosure.
- the display control system 1 according to the embodiment of the present disclosure includes an execution terminal 10 and an operation terminal 20.
- the execution terminal 10 can display a screen including one or more items. The number of items is not particularly limited.
- a screen D10 including items Itm1 to Itm15 is displayed by the execution terminal 10. These items may each correspond, for example, to respective applications implemented by the execution terminal 10.
- the operation terminal 20 can image a screen displayed by the execution terminal 10.
- the screen D10 displayed by the execution terminal 10 is imaged by the operation terminal 20.
- an image obtained by imaging (hereinafter referred to as "a captured image”) is included in a screen D20 displayed by the operation terminal 20.
- This "captured image” may correspond to an image that is presently being captured and displayed as screen D20, or may be an image previously captured and stored by that is subsequently displayed as screen D20.
- the execution terminal 10 executes a command corresponding to the item.
- the execution terminal 10 executes a command "Transition to Photo Selection Screen" corresponding to the item Itm3.
- the execution terminal 10 and the operation terminal 20 may each be personal digital assistants (PDA), portable phones, portable music playback apparatus, portable video processing apparatus, or portable game devices.
- PDA personal digital assistants
- portable phones portable music playback apparatus
- portable video processing apparatus portable game devices.
- the case in which the control of command execution from the operation terminal 20 is desired to be restricted is also considered.
- a user of the execution terminal 10 does not desire to arbitrarily show a moving image to other persons, there is a case in which the execution of a command "Transition to Moving-Image Selection Screen" is desired to be prohibited even when an operation of selecting the item Itm2 for the operation terminal 20 is performed.
- convenience for the user of the operation terminal 20 may be improved.
- the operation terminal 20 can enable the user of the operation terminal 20 to easily recognize a command whose execution is restricted when the control of command execution by the execution terminal 10 is performed using a recognition result of a screen displayed by the execution terminal 10.
- a functional configuration of the execution terminal 10 according to the embodiment of the present disclosure will be described with reference to Fig. 2.
- Fig. 2 is a block diagram illustrating the functional configuration example of the execution terminal 10.
- the execution terminal 10 includes an input apparatus 12, a storage apparatus 14, a display apparatus 16, a communication apparatus 18, and a display control apparatus 100.
- the display control apparatus 100 includes an operation detection unit 110, a transmission control unit 120, a display control unit 130, an acquisition unit 140, and a command execution unit 150.
- the input apparatus 12 is an input device to be used to allow the user to operate the execution terminal 10 or input information to the execution terminal 10.
- the input apparatus 12 can include, for example, a keyboard, a keypad, a mouse, a button, a switch, or a touch panel.
- the input apparatus 12 may include a gesture recognition module that recognizes a gesture of the user.
- the storage apparatus 14 stores data and programs for processing by the execution terminal 10 using a storage medium such as a semiconductor memory or a hard disk.
- a storage medium such as a semiconductor memory or a hard disk.
- the storage apparatus 14 stores a program for display control of a screen including items. Further, for example, the storage apparatus 14 stores a program for executing a command corresponding to the item.
- the display apparatus 16 is a display module constituted by a liquid crystal display (LCD), an organic light-emitting diode (OLED), or a cathode ray tube (CRT).
- the display apparatus 16 may be a part of the execution terminal 10 or may be configured separately from the execution terminal 10.
- the communication apparatus 18 is a communication interface that communicates with the operation terminal 20 in a wired or wireless manner. When connected to a network, the communication apparatus 18 may communicate with the operation terminal 20 via the network. In addition, the communication apparatus 18 may directly communicate with the operation terminal 20 without involving the network. For example, the communication apparatus 18 may perform near-field wireless communication by radio waves or infrared communication with the operation terminal 20.
- the display control apparatus 100 corresponds to a processor such as a central processing unit (CPU) or a digital signal processor (DSP).
- the display control apparatus 100 operates various functions of the display control apparatus 100 to be described later by executing a program stored in the storage apparatus 14 or another storage medium.
- the display control apparatus 100 may not be embedded in the execution terminal 10, and may be embedded in another apparatus (for example, a server).
- Fig. 3 is a block diagram illustrating the functional configuration example of the operation terminal 20.
- the operation terminal 20 includes an imaging apparatus 21, an input apparatus 22, a storage apparatus 24, a display apparatus 26, a communication apparatus 28, and a display control apparatus 200.
- the display control apparatus 200 includes an image acquisition unit 210, a recognition unit 220, an information acquisition unit 230, a display control unit 240, an operation detection unit 250, and a transmission control unit 260.
- the imaging apparatus 21 generates a captured image by imaging a real space using an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).
- an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).
- CCD charge coupled device
- CMOS complementary metal oxide semiconductor
- the imaging apparatus 21 is assumed to be a part of the operation terminal 20 in the embodiment of the present disclosure, the imaging apparatus 21 may be configured separately from the operation terminal 20.
- the input apparatus 22 is an input device to be used to allow the user to operate the operation terminal 20 or input information to the operation terminal 20.
- the input apparatus 22 can include, for example, a keyboard, a keypad, a mouse, a button, a switch, or a touch panel.
- the input apparatus 22 may include a gesture recognition module that recognizes a gesture of the user.
- the storage apparatus 24 stores data and programs for processing by the operation terminal 20 using a storage medium such as a semiconductor memory or a hard disk.
- the storage apparatus 24 stores a feature quantity dictionary to be used for object recognition and virtual object data, which is data of a virtual object serving as a display target. Further, the storage apparatus 24 can store a recognition result generated as an object recognition result.
- the display apparatus 26 is a display module constituted by a liquid crystal display (LCD), an organic light-emitting diode (OLED), or a cathode ray tube (CRT).
- the display apparatus 26 may be a part of the operation terminal 20, or may be configured separately from the operation terminal 20.
- the communication apparatus 28 is a communication interface that communicates with the execution terminal 10 in a wired or wireless manner. When connected to a network, the communication apparatus 28 may communicate with the execution terminal 10 via the network. In addition, the communication apparatus 28 may directly communicate with the execution terminal 10 without involving the network. For example, the communication apparatus 28 may perform near-field wireless communication by radio waves or infrared communication with the execution terminal 10.
- the display control apparatus 200 corresponds to a processor such as a central processing unit (CPU) or a digital signal processor (DSP).
- the display control apparatus 200 operates various functions of the display control apparatus 200 to be described later by executing a program stored in the storage apparatus 24 or another storage medium.
- the display control apparatus 200 may not be embedded in the operation terminal 20, or may be embedded in another apparatus (for example, a server).
- Fig. 4 is a diagram illustrating an example of information (hereinafter referred to as "transmission information") that is transmitted by the execution terminal 10 to the operation terminal 20.
- the transmission information includes configuration information and authority information.
- the configuration information indicates a configuration of a screen including one or more items.
- the display control unit 130 can display the screen according to the configuration information.
- the configuration information indicates positions and sizes of the items Itm1 to Itm15 constituting the screen D10.
- a position of an item can be expressed by a position of a predetermined point of the item, for example, when the item position is based on the position of the predetermined point of the screen.
- a top-left corner of the screen D10 can be a predetermined point of the screen, but a predetermined point of the screen may not be the top left corner of the screen D10.
- a top left corner of the item can be a predetermined point of the item, but a predetermined point of the item may not be the top left corner of the item.
- coordinates of the item are expressed by expressing a position of a predetermined point of the screen D10 as (0, 0), expressing positions of predetermined points of the items Itm1 to Itm15 as (X1, Y1) to (X15, Y15), and expressing sizes (Width, Height) of the items Itm1 to Itm15 as (W1, H1) to (W15, H15) in the example illustrated in Fig. 4, a type of expression is not particularly limited.
- a size of an interval between items and a size of an interval between a screen end and the item may be included in transmission information as configuration information.
- information indicating that execution of a command is permitted or information indicating that the execution of the command is restricted as will be described later may be arranged in order of item number.
- the coordinates of each item can be defined in another type of expression.
- each item may be a point, but, in this case, the size of each item may not be included in the transmission information as the configuration information. Further, for example, if the size of each item is already recognized by the operation terminal 20, the size of each item may not be included in the transmission information as the configuration information.
- the authority information is constituted by information (hereinafter simply referred to as "permission") indicating that the execution of the command is permitted or information (hereinafter simply referred to as “restriction”) indicating that the execution of the command is restricted.
- “permission” or “restriction” is associated with the execution of a command to be executed from each of the items Itm1 to Itm15.
- the authority information may be set, for example, if a setting operation from the user has been detected by the operation detection unit 110.
- the setting operation is input, for example, by the input apparatus 12 from the user.
- the transmission control unit 120 controls the transmission information to be transmitted to the operation terminal 20, the transmission information is transmitted by the communication apparatus 18 to the operation terminal 20 according to the control. For example, if the operation detection unit 110 has detected a transmission operation from the user, the transmission information may be transmitted. The transmission operation is input, for example, by the input apparatus 12 from the user.
- the transmission information transmitted from the execution terminal 10 is received by the communication apparatus 28 of the operation terminal 20.
- the information acquisition unit 230 of the display control apparatus 200 can acquire the transmission information received by the communication apparatus 28.
- the image acquisition unit 210 acquires a captured image of the screen D10.
- the captured image of the screen D10 can be obtained, for example, according to imaging by the imaging apparatus 21 of the operation terminal 20.
- the display control unit 240 may control the captured image acquired by the image acquisition unit 210 to be displayed.
- the display apparatus 26 can perform a display according to display control by the display control unit 240. For example, in the example illustrated in Fig. 1, the display control unit 240 controls the captured image to be displayed as the screen D20.
- the recognition unit 220 acquires a recognition result by recognizing the screen D10. For example, the recognition unit 220 recognizes the screen D10 included in the captured image acquired by the image acquisition unit 210 and a position and posture in the captured image of the screen D10. For example, the recognition unit 220 can recognize the screen included in the captured image by matching a feature quantity determined from the captured image against a feature quantity of the screen D10.
- the recognition unit 220 determines a feature quantity of a real object within the captured image according to a feature-quantity determination method such as a scale invariant feature transform (SIFT) method or a random ferns method, and matches the determined feature quantity against the feature quantity of the real object.
- a feature-quantity determination method such as a scale invariant feature transform (SIFT) method or a random ferns method.
- SIFT scale invariant feature transform
- the recognition unit 220 recognizes information for identifying the screen D10 associated with a feature quantity best matching with the feature quantity of the real object within the captured image and a position and posture of the screen D10 in the captured image.
- the recognition unit 220 uses a feature-quantity dictionary in which the feature-quantity data of the real object is associated with information for identifying the real object.
- the feature-quantity dictionary may be stored by the storage apparatus 24 or received by the communication apparatus 28 from the server.
- the feature-quantity data of the real object may be, for example, a set of feature quantities determined from a learning image of the real object according to the SIFT method or the random ferns method.
- the recognition of the screen D10 is not limited to this example.
- the recognition unit 220 may indirectly recognize the screen D10 by recognizing a known graphic or symbol or a marker such as an artificial marker (for example, a barcode or quick response (QR) code) or a natural marker associated with the screen D10.
- the recognition unit 220 may recognize the screen D10 and recognize the position and posture of the screen D10 from the size and shape in the captured image of the screen D10.
- a technique of recognizing the position and posture of the screen D10 is not limited to a recognition technique based on the image processing.
- the recognition unit 220 can detect a direction of the imaging apparatus 21 and a current location of the operation terminal 20, and estimate the screen D10 included in the captured image and the position and posture of the screen D10 in the captured image based on the detection result.
- Fig. 5 is a diagram illustrating display control by the display control unit 240.
- the display control unit 240 performs control so that a restriction display, which is a display indicating that the execution of the command is restricted, is performed based on the configuration information and the screen recognition result if the authority information indicates the execution of the command is restricted (the case of "restriction").
- the display control unit 240 can add the restriction display to the captured image acquired by the image acquisition unit 210.
- “restriction” is associated with item coordinates (X2, Y2, W2, H2), (X10, Y10, W10, H10), and (X11, Y11, W11, H11). Accordingly, it is preferable that the display control unit 240 control the restriction display to be performed based on the item coordinates (X2, Y2, W2, H2), (X10, Y10, W10, H10), and (X11, Y11, W11, H11) and the recognition result of the screen D10.
- the display control unit 240 converts item coordinates (X2, Y2, W2, H2), (X10, Y10, W10, H10), and (X11, Y11, W11, H11) based on the position and posture of the screen D10. Coordinates (hereinafter referred to as "converted coordinates") after the display control unit 240 has converted the item coordinates are illustrated as (X2', Y2', W2', H2'), (X10', Y10', W10', H10'), and (X11', Y11', W11', H11') in Fig. 5.
- the display control unit 240 can add the restriction display to a position indicated by the converted coordinates.
- the display control unit 240 can add the restriction display to positions indicated by the converted coordinates (X2', Y2', W2', H2'), (X10', Y10', W10', H10'), and (X11', Y11', W11', H11').
- the display apparatus 26 can perform a display according to display control by the display control unit 240.
- Fig. 6 is a diagram illustrating an example of a screen displayed according to the display control by the display control unit 240.
- a captured image in which restriction displays R2, R10, and R11 are added to positions indicated by the converted coordinates (X2', Y2', W2', H2'), (X10', Y10', W10', H10'), and (X11', Y11', W11', H11') is displayed as a screen D21.
- a restriction display method is not particularly limited.
- the restriction display may be a display from which each item is deleted.
- the display control apparatus 200 enables the user to easily recognize a command whose execution is restricted when the control of the command execution by the display control apparatus 100 is performed using the recognition result of a screen subjected to display control by the display control apparatus 100.
- the user easily recognizes that the authority is restricted for each of a command "Transition to Moving-Image Selection Screen” corresponding to the item Itm2, a command “Fasebook” corresponding to the item Itm10, and a command "Diary” corresponding to the item Itm11.
- the user can select the command whose execution is permitted while browsing the screen D11 in which the restriction display is made as described above.
- the user can select one of the items Itm1, Itm3 to Item9, and Itm12 to Itm15.
- the user desires to input an operation of designating coordinates on the screen to the input apparatus 22 as designated coordinates (s', t').
- the operation detection unit 250 detects an operation of designating the designated coordinates (s', t').
- Fig. 7 is a diagram illustrating transmission control by the transmission control unit 260.
- the transmission control unit 260 converts the designated coordinates (s', t') based on a position and posture of the screen D10. Coordinates after the conversion of the designated coordinates (s', t') are denoted by (s, t).
- the transmission control unit 260 controls the coordinates (s, t) to be transmitted to the display control apparatus 100. If the coordinates (s, t) are controlled to be transmitted by the transmission control unit 260 to the display control apparatus 100, the coordinates (s, t) are transmitted by the communication apparatus 28 to the execution terminal 10 according to the control.
- the coordinates (s, t) transmitted by the communication apparatus 28 are received by the communication apparatus 18 of the execution terminal 10. Subsequently, if the coordinates (s, t) are received by the communication apparatus 18, the acquisition unit 140 of the display control apparatus 100 acquires the coordinates (s, t) received by the communication apparatus 18. If the item coordinates have been acquired by the acquisition unit 140 and the authority information indicates that the execution of the command is permitted (the case of "permission"), the command execution unit 150 executes the command. Because "permission" is associated with the item Itm3 as illustrated in Fig.
- the command execution unit 150 execute the command "Transition to Photo Selection Screen" corresponding to the item Itm3 if the coordinates (s, t) acquired by the acquisition unit 140 are consistent with the coordinates (for example, coordinates within a range defined by (X3, Y3, W3, H3)) of the item Itm3 as illustrated in Fig. 7.
- Fig. 8 is a diagram illustrating an example of a command execution result. If the command execution unit 150 has executed the command "Transition to Photo Selection Screen" corresponding to the item Itm3, the display control unit 130 controls a screen D12 corresponding to a photo selection screen to be displayed as illustrated in Fig. 8. If the display control by the display control unit 130 is performed, the screen D12 is displayed by the display apparatus 16 according to the display control. In addition, in the example illustrated in Fig. 8, if the imaging apparatus 21 images the screen D12 and the image acquisition unit 210 acquires a captured image, the display control unit 240 controls the captured image to be displayed as a screen D22.
- the command execution unit 150 not execute the command.
- the authority information indicates that the execution of the command is restricted (the case of "restriction")
- the transmission control unit 260 may determine whether or not a predetermined condition has been satisfied and control the coordinates (s, t) to be transmitted if the predetermined condition has been satisfied. In addition, if the predetermined condition has not been satisfied, the transmission control unit 260 may control the coordinates (s, t) not to be transmitted.
- the transmission control unit 260 controls the coordinates (s, t) to be transmitted to the display control apparatus 100.
- the transmission control unit 260 may control the coordinates (s, t) not to be transmitted to the display control apparatus 100.
- a proximity range can be preset or may not be provided.
- the transmission control unit 260 may control the coordinates (s, t) to be transmitted to the display control apparatus 100 if the authority information indicates that the execution of the command is permitted (the case of "permission"). On the other hand, if the authority information indicates that the execution of the command is restricted (the case of "restriction"), the transmission control unit 260 may control the coordinates (s, t) not to be transmitted to the display control apparatus 100.
- the command execution unit 150 may execute a command designated according to a command designation operation if the operation detection unit 110 has detected the command designation operation from the user. In such a case, if the operation detection unit 110 has detected the command designation operation from the user, it is preferable that the command execution unit 150 execute the command regardless of the authority information. On the other hand, when the acquisition unit 140 has acquired item coordinates, it is preferable that the command execution unit 150 execute the command if the authority information indicates that the execution of the command is permitted (the case of "permission"). In addition, when the acquisition unit 140 has acquired the item coordinates, it is preferable that the command execution unit 150 restrict the execution of the command if the authority information indicates that the execution of the command is restricted (the case of "restriction"). The command designation operation is input, for example, by the input apparatus 12 from the user.
- FIG. 9 is a flowchart illustrating a flow of the operation of the display control system 1 according to the embodiment of the present disclosure.
- the display apparatus 16 displays the screen including the items according to display control by the display control unit 130 (step S11).
- the number of items is not particularly limited as described above.
- the transmission control unit 120 controls configuration information of the screen and authority information for a command to be executed from the item to be transmitted to the operation terminal 20.
- the communication apparatus 18 transmits the configuration information and the authority information to the operation terminal 20 (step S12).
- the configuration information and the authority information are transmitted, for example, if the operation detection unit 110 has detected a transmission operation from the user.
- the information acquisition unit 230 acquires the configuration information and the authority information received by the communication apparatus 28.
- the imaging apparatus 21 images the screen subjected to the display control by the display control unit 130
- the image acquisition unit 210 acquires a captured image obtained according to imaging by the imaging apparatus 21 (step S22).
- the recognition unit 220 recognizes the screen from the captured image (step S23).
- the display control unit 240 performs display control of the captured image, the display apparatus 26 displays the captured image (step S24).
- step S25 the display control unit 240 adds a restriction display to the captured image based on the configuration information and the recognition result (step S26), and proceeds to step S27. If the authority information does not indicate that the execution of the command is restricted (that is, if the authority information indicates that the execution of the command is permitted) ("No" in step S25), the display control unit 240 proceeds to step S27. Subsequently, if the operation detection unit 250 has detected designated coordinates from the user, the transmission control unit 260 converts the designated coordinates based on the recognition result obtained according to recognition by the recognition unit 220 (step S27), and controls the converted coordinates to be transmitted.
- the communication apparatus 28 transmits the converted coordinates according to the control by the transmission control unit 260 (step S28).
- the display control unit 240 adds a restriction display to the captured image based on the configuration information and the recognition result (step S26), it is possible for a restriction display to be removed by changing a status corresponding to the command from restricted to permitted.
- a password or other authentication information may be entered at the operation terminal 20 and transmitted to the execution terminal 10.
- the execution terminal 10 may then, upon successful authentication, transmit data to the operation terminal 20 indicating that a status of a command be changed from restricted to permitted.
- An exemplary process of entering this authentication information may include selecting one of the restriction displays R2, R10 and R11 shown in Fig. 6.
- the display control unit 240 controls the display to display an interface prompting a user to enter the authentication information. This authentication information is then sent from the operation terminal 20 to the execution terminal 10, as outlined above.
- the display control unit 240 controls the display to remove the restriction display from the captured image.
- step S13 the acquisition unit 140 acquires the coordinates received by the communication apparatus 18. If the coordinates received by the communication apparatus 18 are not item coordinates (or are not close to item coordinates) ("No" in step S14), the command execution unit 150 ends the operation without executing the command. Alternatively, the display control unit 130 may re-perform the display control (step S11). If the coordinates received by the communication apparatus 18 are item coordinates (or are close to item coordinates) ("Yes" in step S14), the command execution unit 150 proceeds to step S15.
- step S15 If the authority information does not indicate that the execution of the command is permitted (that is, if the execution of the command is indicated as being restricted) ("No" in step S15), the command execution unit 150 ends the operation without executing the command. Alternatively, the display control unit 130 may re-perform the display control (step S11). If the authority information indicates that the execution of the command is permitted ("Yes" in step S15), the command execution unit 150 executes the command (step S16), and ends the operation. Alternatively, the display control unit 130 may re-perform the display control (step S11). The display control unit 130 may perform display control of the command execution result.
- Fig. 10 is a diagram illustrating an example of transmission information according to the modified example of the embodiment of the present disclosure.
- the transmission information may include information (hereinafter referred to as "command identification information") for identifying a command in addition to configuration information and authority information.
- the command identification information is not indispensable information. If the transmission control unit 120 of the execution terminal 10 controls the transmission information as described above to be transmitted, the communication apparatus 18 transmits the transmission information. The transmission information transmitted by the communication apparatus 18 is received by the communication apparatus 28 of the operation execution 20 and acquired by the information acquisition unit 230.
- the transmission control unit 260 can control command identification information for identifying the command to be transmitted to the display control apparatus 100.
- the transmission control unit 260 may determine whether or not a predetermined condition has been satisfied, and control the command identification information to be transmitted if the predetermined condition has been satisfied.
- the transmission control unit 260 may control the command identification information not to be transmitted if the predetermined condition has not been satisfied.
- the transmission control unit 260 can control command identification information corresponding to an item to be transmitted to the display control apparatus 100.
- the transmission control unit 260 may control command identification information "COM3" associated with their item to be transmitted to the display control apparatus 100.
- command identification information of which the transmission is controlled by the transmission control unit 260 may be included in the transmission information, the command identification information may be an item number or the like defined by the configuration information when the command identification information is not included in the transmission information.
- the transmission control unit 260 may control command identification information to be transmitted to the display control apparatus 100.
- the transmission control unit 260 may control command identification information not to be transmitted to the display control apparatus 100.
- the communication apparatus 28 transmits the command identification information according to control by the transmission control unit 260
- the communication apparatus 18 of the execution terminal 10 receives the command identification information and the acquisition unit 140 acquires the command identification information received by the communication apparatus 18. If the acquisition unit 140 has acquired the command identification information and the authority information indicates that the execution of the command is permitted, the command execution unit 150 can execute the command. If the acquisition unit 140 has not acquired the command identification information, the command execution unit 150 may not execute the command. In addition, if the authority information indicates that the execution of the command is restricted, the command execution unit 150 can restrict the execution of the command.
- Fig. 11 is a flowchart illustrating a flow of the operation of the display control system 1 according to the modified example of the embodiment of the present disclosure.
- the display apparatus 16 displays the screen including the items according to the display control by the display control unit 130 (step S31).
- the number of items is not particularly limited as described above.
- the transmission control unit 120 controls configuration information of the screen and authority information and command identification information for the command to be executed from the item to be transmitted to the operation terminal 20.
- the communication apparatus 18 transmits the configuration information, the authority information, and the command identification information to the operation terminal 20 (step S32).
- the configuration information, the authority information, and the command identification information may be transmitted, for example, if the operation detection unit 110 has detected the transmission operation from the user.
- the information acquisition unit 230 acquires the configuration information, the authority information, and the command identification information received by the communication apparatus 28.
- the imaging apparatus 21 images a screen subjected to display control by the display control unit 130
- the image acquisition unit 210 acquires a captured image obtained according to imaging by the imaging apparatus 21 (step S42).
- the recognition unit 220 recognizes the screen from the captured image (step S43).
- the display control unit 240 performs display control of the captured image, the display apparatus 26 displays the captured image (step S44).
- step S45 the display control unit 240 adds a restriction display to the captured image based on the configuration information and the recognition result (step S46), and proceeds to step S47. If the authority information does not indicate that the execution of the command is restricted (that is, if the execution of the command is indicated as being permitted) ("No" in step S45), the display control unit 240 proceeds to step S47. Subsequently, if the operation detection unit 250 detects designated coordinates from the user, the transmission control unit 260 converts the designated coordinates based on the recognition result obtained according to recognition by the recognition unit 220 (step S47).
- the transmission control unit 260 ends the operation without executing the command.
- the display control unit 130 may re-perform the display control (step S31).
- the transmission control unit 260 controls the command identification information corresponding to the item to be transmitted if the coordinates after the conversion are the item coordinates (or are close to the item coordinates) ("Yes" in step S48).
- the communication apparatus 28 transmits the command identification information according to the control by the transmission control unit 260 (step S49). If the communication apparatus 18 of the execution terminal 10 receives the command identification information (step S33), the acquisition unit 140 acquires the command identification information received by the communication apparatus 18.
- step S34 the command execution unit 150 ends the operation without executing the command identified by the command identification information.
- the display control unit 130 may re-perform display control (step S31). If the authority information indicates that the execution of the command is permitted ("Yes" in step S35), the command execution unit 150 executes the command identified by the command identification information (step S36), and ends the operation. Alternatively, the display control unit 130 may re-perform the display control (step S31). The display control unit 130 may perform display control of the command execution result.
- the display control apparatus 200 has the information acquisition unit 230 that acquires configuration information of a screen including one or more items of the display control apparatus 100 and authority information for a command to be executed from the item.
- the display control apparatus 200 includes the display control unit 240 that controls a restriction display, which is a display indicating that the execution of the command is restricted, to be performed based on the screen recognition result and the configuration information.
- the server may have the above-described function instead of the display control apparatus 200.
- the server may recognize a screen from the captured image in place of the display control apparatus 200.
- the technology of the present disclosure can also be applied to cloud computing.
- steps in the operation of the display control system 1 of this specification are not necessarily performed in time series in the order described as the flowchart.
- the steps in the operation of the display control system 1 may be processed in an order different from the order described in the flowchart or processed in parallel.
- a computer program for causing hardware such as a CPU, a read only memory (ROM), and a random access memory (RAM) embedded in the execution terminal 10 or the operation terminal 20 to have the same functions as each configuration of the above-described execution terminal 10 or operation terminal 20.
- a storage medium recording the computer program is also provided.
- An information processing system comprising: one or more processing units that acquire captured image data corresponding to a user interface displayed on a first display, the user interface including a plurality of graphic representations each corresponding to a respective function; and control a second display to display a representation of the user interface based on a recognition process performed on the user interface, the representation of the user interface indicating whether a function corresponding the plurality of the graphic representations is capable of being accessed.
- the information processing system of (1) further comprising: an image capturing unit that captures the image data corresponding to the user interface displayed on the first display.
- the one or more processing units control the communication interface to transmit coordinate information corresponding to the selection of the one of the plurality of graphic representations displayed in the representation of user interface displayed on the second display to an information processing apparatus including the first display.
- a communication interface wherein the one or more processing units control the communication interface to transmit command identification information corresponding to the selection of the one of the plurality of graphic representations displayed in the representation of user interface displayed on the second display to an information processing apparatus including the first display.
- one of the plurality of graphic representations corresponds to an photo browsing application.
- the information processing system is a first information processing apparatus and the plurality of graphic representations each corresponding to a respective function implemented at a second information processing apparatus.
- a non-transitory computer-readable medium including computer program instructions, which when executed by an information processing system, cause the information processing system to perform a process, the process comprising: acquiring captured image data corresponding to a user interface displayed on a first display, the user interface including a plurality of graphic representations each corresponding to a respective function; and controlling a second display to display a representation of the user interface based on a recognition process performed on the user interface, the representation of the user interface indicating whether a function corresponding the plurality of the graphic representations is capable of being accessed.
- Display control system 10 Execution terminal 20 Operation terminal 100 Display control apparatus 110 Operation detection unit 120 Transmission control unit 130 Display control unit 140 Acquisition unit 150 Command execution unit 200 Display control apparatus 210 Image acquisition unit 220 Recognition unit 230 Information acquisition unit 240 Display control unit 250 Operation detection unit 260 Transmission control unit
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- User Interface Of Digital Computer (AREA)
- Controls And Circuits For Display Device (AREA)
- Testing And Monitoring For Control Systems (AREA)
Abstract
Description
1. Overview of Display Control System
2. Functional Configuration Example of Execution Terminal
3. Functional Configuration Example of Operation Terminal
4. Functions Provided in Display Control Apparatus
5. Operation of Display Control System
6. Modified Example
7. Conclusion
Hereinafter, first, the overview of the display control system according to an embodiment of the present disclosure will be described with reference to Fig. 1.
Fig. 2 is a block diagram illustrating the functional configuration example of the
The
The
The
The
The
Fig. 3 is a block diagram illustrating the functional configuration example of the
The
The
The
The
The
The
Fig. 4 is a diagram illustrating an example of information (hereinafter referred to as "transmission information") that is transmitted by the
Fig. 9 is a flowchart illustrating a flow of the operation of the
Fig. 10 is a diagram illustrating an example of transmission information according to the modified example of the embodiment of the present disclosure. As illustrated in Fig. 10, the transmission information may include information (hereinafter referred to as "command identification information") for identifying a command in addition to configuration information and authority information. However, the command identification information is not indispensable information. If the
As described above, the
(1) An information processing system, comprising: one or more processing units that acquire captured image data corresponding to a user interface displayed on a first display, the user interface including a plurality of graphic representations each corresponding to a respective function; and control a second display to display a representation of the user interface based on a recognition process performed on the user interface, the representation of the user interface indicating whether a function corresponding the plurality of the graphic representations is capable of being accessed.
(2) The information processing system of (1), further comprising: an image capturing unit that captures the image data corresponding to the user interface displayed on the first display.
(3) The information processing system of any of (1) to (2), wherein the one or more processing units acquire at least one of configuration information and authority information corresponding to the user interface displayed on the first display.
(4) The information processing system of any of (1) to (3), wherein the configuration information indicates a position at which each of the plurality of graphic representations are displayed on the first display.
(5) The information processing system of any of (1) to (4), wherein the authority information indicates whether a function corresponding to each of the plurality of the graphic representations is capable of being accessed.
(6) The information processing system of any of (1) to (5), wherein the one or more processing units generate the representation of the user interface based on the acquired configuration information.
(7) The information processing system of any of (1) to (6), wherein the one or more processing units control the second display to display the representation of the user interface such that a predetermined visual effect is applied to at least one of the plurality of graphic representations corresponding to a function that is not capable of being accessed.
(8) The information processing system of any of (1) to (7), wherein the one or more processing units control the second display to display the representation of the user interface such that at least one of the plurality of graphic representations corresponding to a function that is not capable of being accessed is omitted.
(9) The information processing system of any of (1) to (8), wherein the one or more processing units control the second display to display the representation of the user interface such that an image is superimposed on at least one of the plurality of graphic representations corresponding to a function that is not capable of being accessed is omitted.
(10) The information processing system of any of (1) to (9), further comprising: a user interface that receives a selection of one of the plurality of graphic representations displayed in the representation of user interface displayed on the second display.
(11) The information processing system of any of (1) to (10), further comprising: a communication interface, wherein the one or more processing units control the communication interface to transmit information corresponding to the selection of the one of the plurality of graphic representations displayed in the representation of user interface displayed on the second display to an information processing apparatus including the first display.
(12) The information processing system of any of (1) to (11), further comprising: a communication interface, wherein the one or more processing units control the communication interface to transmit coordinate information corresponding to the selection of the one of the plurality of graphic representations displayed in the representation of user interface displayed on the second display to an information processing apparatus including the first display.
(13) The information processing system of any of (1) to (12), further comprising: a communication interface, wherein the one or more processing units control the communication interface to transmit command identification information corresponding to the selection of the one of the plurality of graphic representations displayed in the representation of user interface displayed on the second display to an information processing apparatus including the first display.
(14) The information processing system of any of (1) to (13), wherein one of the plurality of graphic representations corresponds to an photo browsing application.
(15) The information processing system of any of (1) to (14), further comprising: a user interface that receives a selection of the graphic representation corresponding to the photo browsing application displayed in the representation of user interface displayed on the second display; and a communication interface, wherein the one or more processing units control the communication interface to transmit information corresponding to the selection of the graphic representation corresponding to the photo browsing application to an information processing apparatus including the first display, wherein the information processing apparatus executes the photo browsing application in response to receiving the information corresponding to the selection of the graphic representation.
(16) The information processing system of any of (1) to (15), wherein the information processing system is a first information processing apparatus and the plurality of graphic representations each corresponding to a respective function implemented at a second information processing apparatus.
(17) The information processing system of any of (1) to (18), wherein the captured image data corresponding to a user interface displayed on a first display is still image data.
(18) The information processing system of any of (1) to (17), further comprising: a user interface that receives authentication information corresponding to a function that is not capable of being accessed.
(19) The information processing system of any of (1) to (18), wherein the one or more processing units controls the display to change the representation of the user interface to indicate that a function is capable of being accessed based on the received authentication information.
(20) The information processing apparatus of any of (1) to (19), wherein the user interface displays a screen to input the authentication information when an graphic representation corresponding to a function that is not capable of being accessed is selected at the second display.
(21) The information processing apparatus of any of (1) to (20), wherein the plurality of graphic representations include at least an icon corresponding to a function.
(22) An information processing method performed by an information processing system, the method comprising: acquiring captured image data corresponding to a user interface displayed on a first display, the user interface including a plurality of graphic representations each corresponding to a respective function; and controlling a second display to display a representation of the user interface based on a recognition process performed on the user interface, the representation of the user interface indicating whether a function corresponding the plurality of the graphic representations is capable of being accessed.
(23) A non-transitory computer-readable medium including computer program instructions, which when executed by an information processing system, cause the information processing system to perform a process, the process comprising: acquiring captured image data corresponding to a user interface displayed on a first display, the user interface including a plurality of graphic representations each corresponding to a respective function; and controlling a second display to display a representation of the user interface based on a recognition process performed on the user interface, the representation of the user interface indicating whether a function corresponding the plurality of the graphic representations is capable of being accessed.
10 Execution terminal
20 Operation terminal
100 Display control apparatus
110 Operation detection unit
120 Transmission control unit
130 Display control unit
140 Acquisition unit
150 Command execution unit
200 Display control apparatus
210 Image acquisition unit
220 Recognition unit
230 Information acquisition unit
240 Display control unit
250 Operation detection unit
260 Transmission control unit
Claims (20)
- An information processing system, comprising:
one or more processing units that
acquire captured image data corresponding to a user interface displayed on a first display, the user interface including a plurality of graphic representations each corresponding to a respective function; and
control a second display to display a representation of the user interface based on a recognition process performed on the user interface, the representation of the user interface indicating whether a function corresponding the plurality of the graphic representations is capable of being accessed.
- The information processing system of claim 1, further comprising:
an image capturing unit that captures the image data corresponding to the user interface displayed on the first display.
- The information processing system of claim 1, wherein
the one or more processing units acquire at least one of configuration information and authority information corresponding to the user interface displayed on the first display.
- The information processing system of claim 3, wherein
the configuration information indicates a position at which each of the plurality of graphic representations are displayed on the first display.
- The information processing system of claim 3, wherein
the one or more processing units generate the representation of the user interface based on the acquired configuration information.
- The information processing system of claim 1, wherein
the one or more processing units control the second display to display the representation of the user interface such that a predetermined visual effect is applied to at least one of the plurality of graphic representations corresponding to a function that is not capable of being accessed.
- The information processing system of claim 1, wherein
the one or more processing units control the second display to display the representation of the user interface such that at least one of the plurality of graphic representations corresponding to a function that is not capable of being accessed is omitted.
- The information processing system of claim 1, wherein
the one or more processing units control the second display to display the representation of the user interface such that an image is superimposed on at least one of the plurality of graphic representations corresponding to a function that is not capable of being accessed is omitted.
- The information processing system of claim 1, further comprising:
a user interface that receives a selection of one of the plurality of graphic representations displayed in the representation of user interface displayed on the second display.
- The information processing system of claim 9, further comprising:
a communication interface, wherein the one or more processing units control the communication interface to transmit information corresponding to the selection of the one of the plurality of graphic representations displayed in the representation of user interface displayed on the second display to an information processing apparatus including the first display.
- The information processing system of claim 9, further comprising:
a communication interface, wherein the one or more processing units control the communication interface to transmit command identification information corresponding to the selection of the one of the plurality of graphic representations displayed in the representation of user interface displayed on the second display to an information processing apparatus including the first display.
- The information processing system of claim 9, wherein
one of the plurality of graphic representations corresponds to an photo browsing application.
- The information processing system of claim 12, further comprising:
a user interface that receives a selection of the graphic representation corresponding to the photo browsing application displayed in the representation of user interface displayed on the second display; and
a communication interface, wherein the one or more processing units control the communication interface to transmit information corresponding to the selection of the graphic representation corresponding to the photo browsing application to an information processing apparatus including the first display, wherein
the information processing apparatus executes the photo browsing application in response to receiving the information corresponding to the selection of the graphic representation.
- The information processing system of claim 1, wherein
the information processing system is a first information processing apparatus and the plurality of graphic representations each corresponding to a respective function implemented at a second information processing apparatus.
- The information processing system of claim 1, wherein
the captured image data corresponding to a user interface displayed on a first display is still image data.
- The information processing system of claim 1, further comprising:
a user interface that receives authentication information corresponding to a function that is not capable of being accessed.
- The information processing system of claim 16, wherein
the one or more processing units controls the display to change the representation of the user interface to indicate that a function is capable of being accessed based on the received authentication information.
- The information processing apparatus of claim 16, wherein
the user interface displays a screen to input the authentication information when an graphic representation corresponding to a function that is not capable of being accessed is selected at the second display.
- An information processing method performed by an information processing system, the method comprising:
acquiring captured image data corresponding to a user interface displayed on a first display, the user interface including a plurality of graphic representations each corresponding to a respective function; and
controlling a second display to display a representation of the user interface based on a recognition process performed on the user interface, the representation of the user interface indicating whether a function corresponding the plurality of the graphic representations is capable of being accessed.
- A non-transitory computer-readable medium including computer program instructions, which when executed by an information processing system, cause the information processing system to perform a process, the process comprising:
acquiring captured image data corresponding to a user interface displayed on a first display, the user interface including a plurality of graphic representations each corresponding to a respective function; and
controlling a second display to display a representation of the user interface based on a recognition process performed on the user interface, the representation of the user interface indicating whether a function corresponding the plurality of the graphic representations is capable of being accessed.
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020147010059A KR20140094512A (en) | 2011-11-16 | 2012-10-04 | Display control apparatus, display control method, and program |
EP12849577.7A EP2780787B1 (en) | 2011-11-16 | 2012-10-04 | Display control apparatus, display control method, and program |
US14/343,987 US9880713B2 (en) | 2011-11-16 | 2012-10-04 | Display control apparatus, display control method, and program |
US15/851,265 US20180136806A1 (en) | 2011-11-16 | 2017-12-21 | Display control apparatus, display control method, and program |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011-250955 | 2011-11-16 | ||
JP2011250955A JP5987299B2 (en) | 2011-11-16 | 2011-11-16 | Display control apparatus, display control method, and program |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/343,987 A-371-Of-International US9880713B2 (en) | 2011-11-16 | 2012-10-04 | Display control apparatus, display control method, and program |
US15/851,265 Continuation US20180136806A1 (en) | 2011-11-16 | 2017-12-21 | Display control apparatus, display control method, and program |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2013073100A1 true WO2013073100A1 (en) | 2013-05-23 |
Family
ID=48429205
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2012/006382 WO2013073100A1 (en) | 2011-11-16 | 2012-10-04 | Display control apparatus, display control method, and program |
Country Status (8)
Country | Link |
---|---|
US (2) | US9880713B2 (en) |
EP (1) | EP2780787B1 (en) |
JP (1) | JP5987299B2 (en) |
KR (1) | KR20140094512A (en) |
CN (2) | CN203689435U (en) |
AR (1) | AR088822A1 (en) |
TW (1) | TW201331824A (en) |
WO (1) | WO2013073100A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015075888A1 (en) * | 2013-11-19 | 2015-05-28 | Sony Corporation | Preventing an operation when a processing device communicates with an acquisition device |
EP3276455A4 (en) * | 2015-03-27 | 2018-03-21 | Fujitsu Limited | Display method, program, and display control device |
WO2020147258A1 (en) * | 2019-01-14 | 2020-07-23 | 平安科技(深圳)有限公司 | Remote desktop operation method and apparatus, readable storage medium, and terminal device |
Families Citing this family (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR101844395B1 (en) * | 2012-08-24 | 2018-04-02 | 엠파이어 테크놀로지 디벨롭먼트 엘엘씨 | Virtual reality applications |
US9607436B2 (en) | 2012-08-27 | 2017-03-28 | Empire Technology Development Llc | Generating augmented reality exemplars |
US20140195968A1 (en) * | 2013-01-09 | 2014-07-10 | Hewlett-Packard Development Company, L.P. | Inferring and acting on user intent |
JP6179227B2 (en) * | 2013-07-08 | 2017-08-16 | 沖電気工業株式会社 | Information processing device, portable terminal, and information input device |
CN104516618B (en) * | 2013-09-27 | 2020-01-14 | 中兴通讯股份有限公司 | Interface function analysis display method and device |
CN104571776A (en) * | 2013-10-22 | 2015-04-29 | 胜华科技股份有限公司 | Electronic device |
JP2017058748A (en) * | 2015-09-14 | 2017-03-23 | セイコーエプソン株式会社 | Equipment function execution method, equipment and program |
US10613739B2 (en) * | 2016-06-11 | 2020-04-07 | Apple Inc. | Device, method, and graphical user interface for controlling multiple devices in an accessibility mode |
US10261600B2 (en) * | 2016-08-09 | 2019-04-16 | International Business Machines Corporation | Remotely operating target device |
US10284492B2 (en) * | 2016-08-10 | 2019-05-07 | Microsoft Technology Licensing, Llc | Control of casting to a media renderer |
CN110431500B (en) * | 2017-03-21 | 2022-07-15 | 三菱电机株式会社 | Monitoring screen data generation device, monitoring screen data generation method, and storage medium |
CN108388155B (en) * | 2018-03-21 | 2020-06-23 | 联想(北京)有限公司 | Remote control method and system |
CN111801731B (en) * | 2019-01-22 | 2024-02-13 | 京东方科技集团股份有限公司 | Voice control method, voice control device, and computer-executable nonvolatile storage medium |
DE102019131897A1 (en) * | 2019-11-26 | 2021-05-27 | Bayerische Motoren Werke Aktiengesellschaft | System and process for the software-independent integration of mobile devices into the vehicle |
CN111273850B (en) * | 2020-01-21 | 2022-01-21 | 维达力实业(深圳)有限公司 | Terminal control method and device, computer equipment and storage medium |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001142825A (en) * | 1999-11-18 | 2001-05-25 | Sony Corp | Portable information processing terminal, system and method for inputting/outputting information |
JP2002062972A (en) * | 2000-05-30 | 2002-02-28 | Sony Corp | Electronic apparatus operation device, storage medium and electronic apparatus |
JP2003284031A (en) * | 2002-03-25 | 2003-10-03 | Sony Corp | Information image utilizing system, information image managing server, information image managing method, apparatus information image, program and recording medium |
JP2006081028A (en) * | 2004-09-10 | 2006-03-23 | Kyocera Mita Corp | Remote operation system, communication equipment, and electric apparatus |
JP2007243726A (en) * | 2006-03-09 | 2007-09-20 | Fujifilm Corp | Remote control apparatus, method and system |
JP2007334633A (en) | 2006-06-15 | 2007-12-27 | Hitachi Information Systems Ltd | Remote desktop system |
US20090174653A1 (en) | 2008-01-07 | 2009-07-09 | Samsung Electronics Co., Ltd. | Method for providing area of image displayed on display apparatus in gui form using electronic apparatus, and electronic apparatus applying the same |
JP2009246646A (en) * | 2008-03-31 | 2009-10-22 | Kenwood Corp | Remote control apparatus and setting method |
US20110138416A1 (en) | 2009-12-04 | 2011-06-09 | Lg Electronics Inc. | Augmented remote controller and method for operating the same |
US20110138444A1 (en) | 2009-12-04 | 2011-06-09 | Lg Electronics Inc. | Augmented remote controller and method for operating the same |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1497739B1 (en) * | 2002-04-22 | 2016-08-10 | Microsoft Technology Licensing, LLC | Application sharing security |
US20060002315A1 (en) * | 2004-04-15 | 2006-01-05 | Citrix Systems, Inc. | Selectively sharing screen data |
US7451921B2 (en) * | 2004-09-01 | 2008-11-18 | Eric Morgan Dowling | Methods, smart cards, and systems for providing portable computer, VoIP, and application services |
US7580933B2 (en) * | 2005-07-28 | 2009-08-25 | Microsoft Corporation | Resource handling for taking permissions |
US9189124B2 (en) * | 2009-04-15 | 2015-11-17 | Wyse Technology L.L.C. | Custom pointer features for touch-screen on remote client devices |
CN101674110A (en) * | 2009-10-21 | 2010-03-17 | 中兴通讯股份有限公司 | Method for implementing desktop sharing and control of mobile terminals and mobile terminal |
US8645866B2 (en) * | 2010-06-29 | 2014-02-04 | Exelis Inc. | Dynamic icon overlay system and method of producing dynamic icon overlays |
EP2625646B1 (en) * | 2010-10-06 | 2022-06-22 | Citrix Systems Inc. | Mediating resource access based on a physical location of a mobile device |
US9178981B2 (en) * | 2010-12-22 | 2015-11-03 | Lg Electronics Inc. | Mobile terminal and method of sharing information therein |
US20130141471A1 (en) * | 2011-06-05 | 2013-06-06 | Apple Inc. | Obscuring graphical output on remote displays |
US20130083003A1 (en) * | 2011-09-30 | 2013-04-04 | Kathryn Stone Perez | Personal audio/visual system |
US9195388B2 (en) * | 2013-03-15 | 2015-11-24 | Apple Inc. | Specifying applications to share in a shared access mode |
-
2011
- 2011-11-16 JP JP2011250955A patent/JP5987299B2/en active Active
-
2012
- 2012-10-04 KR KR1020147010059A patent/KR20140094512A/en not_active Application Discontinuation
- 2012-10-04 US US14/343,987 patent/US9880713B2/en active Active
- 2012-10-04 WO PCT/JP2012/006382 patent/WO2013073100A1/en active Application Filing
- 2012-10-04 EP EP12849577.7A patent/EP2780787B1/en active Active
- 2012-11-02 TW TW101140801A patent/TW201331824A/en unknown
- 2012-11-08 CN CN201220584715.2U patent/CN203689435U/en not_active Expired - Fee Related
- 2012-11-08 CN CN201210442952.XA patent/CN103176690B/en not_active Expired - Fee Related
- 2012-11-09 AR ARP120104232A patent/AR088822A1/en unknown
-
2017
- 2017-12-21 US US15/851,265 patent/US20180136806A1/en not_active Abandoned
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001142825A (en) * | 1999-11-18 | 2001-05-25 | Sony Corp | Portable information processing terminal, system and method for inputting/outputting information |
JP2002062972A (en) * | 2000-05-30 | 2002-02-28 | Sony Corp | Electronic apparatus operation device, storage medium and electronic apparatus |
JP2003284031A (en) * | 2002-03-25 | 2003-10-03 | Sony Corp | Information image utilizing system, information image managing server, information image managing method, apparatus information image, program and recording medium |
JP2006081028A (en) * | 2004-09-10 | 2006-03-23 | Kyocera Mita Corp | Remote operation system, communication equipment, and electric apparatus |
JP2007243726A (en) * | 2006-03-09 | 2007-09-20 | Fujifilm Corp | Remote control apparatus, method and system |
JP2007334633A (en) | 2006-06-15 | 2007-12-27 | Hitachi Information Systems Ltd | Remote desktop system |
US20090174653A1 (en) | 2008-01-07 | 2009-07-09 | Samsung Electronics Co., Ltd. | Method for providing area of image displayed on display apparatus in gui form using electronic apparatus, and electronic apparatus applying the same |
JP2009246646A (en) * | 2008-03-31 | 2009-10-22 | Kenwood Corp | Remote control apparatus and setting method |
US20110138416A1 (en) | 2009-12-04 | 2011-06-09 | Lg Electronics Inc. | Augmented remote controller and method for operating the same |
US20110138444A1 (en) | 2009-12-04 | 2011-06-09 | Lg Electronics Inc. | Augmented remote controller and method for operating the same |
Non-Patent Citations (1)
Title |
---|
See also references of EP2780787A4 |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2015075888A1 (en) * | 2013-11-19 | 2015-05-28 | Sony Corporation | Preventing an operation when a processing device communicates with an acquisition device |
EP3276455A4 (en) * | 2015-03-27 | 2018-03-21 | Fujitsu Limited | Display method, program, and display control device |
US10466835B2 (en) | 2015-03-27 | 2019-11-05 | Fujitsu Limited | Display method and display control apparatus |
EP3929710A1 (en) * | 2015-03-27 | 2021-12-29 | Fujitsu Limited | Display method and display control apparatus |
WO2020147258A1 (en) * | 2019-01-14 | 2020-07-23 | 平安科技(深圳)有限公司 | Remote desktop operation method and apparatus, readable storage medium, and terminal device |
Also Published As
Publication number | Publication date |
---|---|
EP2780787B1 (en) | 2020-04-15 |
AR088822A1 (en) | 2014-07-10 |
TW201331824A (en) | 2013-08-01 |
JP2013105454A (en) | 2013-05-30 |
US20140223323A1 (en) | 2014-08-07 |
CN103176690A (en) | 2013-06-26 |
CN203689435U (en) | 2014-07-02 |
KR20140094512A (en) | 2014-07-30 |
EP2780787A1 (en) | 2014-09-24 |
JP5987299B2 (en) | 2016-09-07 |
US20180136806A1 (en) | 2018-05-17 |
EP2780787A4 (en) | 2015-04-22 |
CN103176690B (en) | 2018-06-12 |
US9880713B2 (en) | 2018-01-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2013073100A1 (en) | Display control apparatus, display control method, and program | |
US8443302B2 (en) | Systems and methods of touchless interaction | |
US9223487B2 (en) | Electronic apparatus, method of controlling the same, and related computer program | |
US9268410B2 (en) | Image processing device, image processing method, and program | |
EP2791883B1 (en) | Information processing device, information processing method and program | |
US20180089867A1 (en) | Calculating device, graph display method of calculating device, and storage medium retaining graph display program | |
KR20140130675A (en) | Image processing device, and computer program product | |
JP6349800B2 (en) | Gesture recognition device and method for controlling gesture recognition device | |
KR20140129412A (en) | Icon arrangement method for electronic device and electronic device thereof | |
US12039791B2 (en) | Image processing method, and electronic device supporting same | |
KR20130050701A (en) | Method and apparatus for controlling content of the remote screen | |
JP6287382B2 (en) | Gesture recognition device and method for controlling gesture recognition device | |
US20170124765A1 (en) | Control method and information processing system | |
EP2747404A1 (en) | Image processing terminal, image processing system, and computer-readable storage medium storing control program of image processing terminal | |
US20160147294A1 (en) | Apparatus and Method for Recognizing Motion in Spatial Interaction | |
US9262689B1 (en) | Optimizing pre-processing times for faster response | |
CN111913674A (en) | Virtual content display method, device, system, terminal equipment and storage medium | |
JP6304238B2 (en) | Display control device, display control method, and recording medium | |
JP6358072B2 (en) | Image display terminal, image display system, and image display program | |
US20150062038A1 (en) | Electronic device, control method, and computer program product | |
WO2021178255A1 (en) | Electronic input system | |
KR101439178B1 (en) | System and Method for remote control using camera | |
KR101491648B1 (en) | System and Method for remote control using camera | |
US20230205320A1 (en) | Gesture Recognition Systems and Methods for Facilitating Touchless User Interaction with a User Interface of a Computer System | |
US20150077340A1 (en) | Method, system and computer program product for real-time touchless interaction |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 12849577 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14343987 Country of ref document: US |
|
ENP | Entry into the national phase |
Ref document number: 20147010059 Country of ref document: KR Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2012849577 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |