JP6225469B2 - Apparatus, information processing apparatus, method, and program - Google Patents

Description

  The present invention relates to an apparatus, an information processing apparatus, a method, and a program.

  For example, there may be a case where processing across a plurality of devices is required, such as when an image stored in the first device is pasted at a specific position on the display screen of the second device.

  However, conventionally, in such a case, for example, the recording medium is taken out from the first device, attached to the second device, read out by the second device, the pasting position is designated, and the pasting operation is performed. The user needs to perform operations across a plurality of devices, which is complicated.

  In the first aspect of the present invention, an identification image acquisition unit that acquires an identification image for identifying an information processing device or a user of the information processing device displayed on the display screen of the information processing device, and an imaging device An apparatus comprising: a captured image acquisition unit that acquires a captured image of a captured display screen; and an identifying unit that compares the identification image and the captured image to identify an information processing device or a user corresponding to the identification image, A method executed by an apparatus and a program executed by the apparatus are provided.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

1 shows a configuration of a transfer system 10 according to the present embodiment. 1 is a block diagram of a transfer system 10 according to the present embodiment. 3 is a flowchart of a transfer process executed by the transfer system 10. FIG. 10 is a diagram of an identification image IG displayed on the display screen 18. It is a figure of the picked-up image PG which imaged the identification image IG from the front. It is a figure of the picked-up image PG which imaged the identification image IG from diagonally forward. It is a figure explaining designation | designated of the position on the captured image PG by the user. It is a figure explaining the display screen 18 on which the memory image MG was pasted. It is a figure explaining operation of identification picture IG on picked-up picture PG by a user. It is a figure explaining the identification image IG which moves so that it can image. 2 shows an exemplary hardware configuration of a computer 1900 according to the present embodiment.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 shows a configuration of a transfer system 10 according to the present embodiment. The transfer system 10 images the display screen 18 of the information processing device 14 by the imaging device 12 and causes the user to operate the information processing device 14 on the display screen 20 of the imaging device 12.

  For example, in the transfer system 10, the information processing device 14 to be operated is specified by the transfer device 16 based on the identification image IG displayed by the information processing device 14, and the imaging device 12 is stored in the specified information processing device 14. The stored image MG is transferred, and the stored image MG is pasted on the display screen 18 of the information processing apparatus 14. The transfer system 10 includes an imaging device 12, an information processing device 14, and a transfer device 16.

  The transfer system 10 may include a plurality of imaging devices 12 and / or a plurality of information processing devices 14. In the example shown in FIG. 1, the transfer system 10 includes two information processing apparatuses 14a and 14b. In the following description, when it is not necessary to individually identify any of the information processing devices 14a and 14b, the information processing device 14a is represented by reference numeral “14”.

  The imaging device 12 captures a subject, generates an image of the subject, and stores the image as a stored image MG. The imaging device 12 is connected to the transfer device 16 by wireless communication or the like, and transmits the stored image MG and the captured image PG obtained by capturing the display screen 18 of the information processing device 14 to the transfer device 16.

  The imaging device 12 displays images such as the stored image MG and the captured image PG on the display screen 20. Further, the imaging device 12 inputs an operation on the information processing device 14 from the user, and transmits the input operation to the transfer device 16.

  The information processing device 14 may be a large computer or a portable computer that executes various application programs connected to the transfer device 16 by wireless communication or the like, and transmits the identification image IG and the display image DG to the transfer device 16. The stored image MG is received from the imaging device 12 via the transfer device 16.

  The information processing apparatus 14 displays an image or the like generated by the application program on the display screen 18 as the display image DG together with the identification image IG. In addition, the information processing device 14 pastes the stored image MG received from the imaging device 12 via the transfer device 16 on the display screen 18 and displays it.

  Based on the identification image IG received from the information processing device 14 and the captured image PG received from the imaging device 12, the transfer device 16 is subject to user operations (for example, transfer and pasting of the stored image MG). The information processing apparatus 14 is specified. The transfer device 16 acquires the stored image MG and the user's operation from the imaging device 12, transfers the stored image MG to the identified information processing device 14, and stores the stored image MG on the display screen 18 of the information processing device 14. paste.

  In the example illustrated in FIG. 1, the transfer device 16 transfers the stored image MG to the information processing device 14 a in which the imaging device 12 images the display screen 18. As a result, the transfer device 16 specifies the information processing device 14 to which the user intends to transfer the stored image MG, even if the information specifying the information processing device 14 is not directly input by the user. The stored image MG is transferred to the device 14.

  FIG. 2 is a block diagram of the transfer system 10 according to the present embodiment. The imaging device 12 includes a processing unit 30, an imaging unit 32, a storage unit 34, a display unit 36, an input unit 38, and a communication unit 40.

  The processing unit 30 includes an arithmetic and control circuit including, for example, a CPU (Central Processing Unit). The processing unit 30 executes a program stored in the storage unit 34. The processing unit 30 is connected to the imaging unit 32, the storage unit 34, the display unit 36, the input unit 38, and the communication unit 40, and controls each of them.

  The imaging unit 32 stores an image generated by imaging a subject in the storage unit 34 as a stored image MG in accordance with a user operation. In addition, the imaging unit 32 generates a captured image PG obtained by imaging the display screen 18 of the information processing apparatus 14 to be operated. The imaging unit 32 supplies the stored image MG and the captured image PG to the communication unit 40.

  The display unit 36 includes a liquid crystal display device, an organic EL device, or the like provided on the back surface of the imaging device 12, and a captured image PG or a stored image MG stored in the storage unit 34 according to a user operation. Is displayed on the display screen 20.

  The input unit 38 includes a button and a touch panel provided on the entire surface of the display unit 36, and receives an operation of the user on the imaging unit 32, an operation of specifying a position on the captured image PG, and the like. The input unit 38 supplies the input user operation to the processing unit 30. For example, the input unit 38 inputs an operation for designating a position to paste the stored image MG on the display screen 18 in the imaging screen PG.

  The communication unit 40 transmits / receives data to / from the transfer device 16 by wireless communication or the like. For example, the communication unit 40 receives the stored image MG, the captured image PG, and the user operation from the processing unit 30 and transmits them to the transfer device 16.

  The information processing apparatus 14 includes a processing unit 42, an input unit 44, a storage unit 46, a communication unit 48, and an identification image display unit 50.

  The processing unit 42 includes, for example, an arithmetic and control circuit including a CPU and the like. The processing unit 42 executes a program stored in the storage unit 46. The processing unit 42 is connected to the input unit 44, the storage unit 46, the communication unit 48, and the identification image display unit 50, and controls each of them. The processing unit 42 acquires the display image DG displayed on the display screen 18 by screen capture or the like, and supplies it to the communication unit 48.

  The input unit 44 includes a keyboard, a mouse, and / or a touch panel, and inputs an operation on the information processing apparatus 14 from a user. The input unit 44 supplies the input user operation to the processing unit 42.

  The storage unit 46 stores a program executed by the information processing apparatus 14 and an identification image IG that can be identified by human vision for identifying the information processing apparatus 14 or a user of the information processing apparatus 14. The storage unit 46 may store one identification image IG, or may store a plurality of identification images IG corresponding to a plurality of users of the information processing device 14.

  The communication unit 48 transmits / receives data to / from the transfer device 16 by wireless communication or the like. For example, the communication unit 48 acquires the display image DG and the identification image IG from the processing unit 42 and transmits these images to the transfer device 16. Further, the communication unit 48 receives the stored image MG, the operation on the stored image MG, and the operation on the identification image IG from the transfer device 16 and supplies them to the processing unit 42.

  The identification image display unit 50 may include a liquid crystal display device, an organic EL device, or the like. The identification image display unit 50 acquires the identification image IG from the storage unit 46 and displays the image generated by the program executed by the processing unit 42 together with the identification image IG. It is displayed on the screen 18. The identification image display unit 50 changes or moves the display of the identification image IG in accordance with an instruction for the identification image IG received via the processing unit 42.

  Further, the identification image display unit 50 receives an image such as the stored image MG transferred from the imaging device 12 via the transfer device 16 from the processing unit 42. The identification image display unit 50 pastes and displays the stored image MG on the display screen 18 in response to an instruction for the stored image MG received by the communication unit 48.

  The transfer device 16 includes a communication unit 52, an identification data storage unit 54, and a control unit 56.

  The communication unit 52 transmits and receives data between the imaging device 12 and the information processing device 14. The communication unit 52 is connected to the control unit 56 and outputs data received from the imaging device 12 and the information processing device 14 to the control unit 56. The communication unit 52 transmits the data input from the control unit 56 to the imaging device 12 and the information processing device 14.

  The identification data storage unit 54 is connected to the control unit 56 and stores a correspondence relationship between the plurality of identification images IG and the plurality of information processing devices 14 or a correspondence relationship between the plurality of identification images IG and a plurality of users. . For example, the identification data storage unit 54 associates identification information of the information processing device 14 such as a MAC address or identification information of the information processing device 14 and user identification information such as a user ID with the identification image IG. Remember the table.

  The identification data storage unit 54 stores data input from the control unit 56. For example, the identification data storage unit 54 may store the captured image PG and the stored image MG received from the imaging device 12 via the control unit 56.

  The control unit 56 is a control circuit including a CPU and the like, for example. The control unit 56 includes an identification image acquisition unit 60, a captured image acquisition unit 62, a specification unit 64, an operation input unit 66, an identification image operation unit 68, and a transfer processing unit 70.

  The identification image acquisition unit 60 is connected to the identification unit 64 and the operation input unit 66, acquires the identification image IG for each information processing apparatus 14 or user from the identification data storage unit 54, and identifies the identification unit 64 and the operation input unit 66. To output.

  The captured image acquisition unit 62 is connected to the specifying unit 64, the operation input unit 66, and the identification image operation unit 68. The captured image PG of the display screen 18 of the information processing apparatus 14 captured by the imaging device 12 is transmitted to the communication unit 52. Via the imaging device 12 and output to the specifying unit 64, the operation input unit 66, and the identification image operation unit 68. In addition, the captured image acquisition unit 62 transmits an imaging notification to the information processing apparatus 14 that the imaging apparatus 12 has captured the captured image PG.

  Whether the identification unit 64 compares the identification image IG retrieved from the identification data storage unit 54 and the captured image PG, and includes either the information processing device 14 or the identification image IG corresponding to the user in the captured image PG. Judge whether or not. When determining that the identification image IG is included in the captured image PG, the specifying unit 64 specifies the information processing apparatus 14 or the user corresponding to the identification image IG.

  When the position on the captured image PG is designated by the input unit 38 of the imaging device 12, the operation input unit 66 performs an operation on the position on the display screen 18 of the information processing device 14 corresponding to the position. Input from the imaging device 12 via the unit 52.

  The operation input unit 66 determines whether the position designated on the captured image PG matches or substantially matches the position where the identification image IG exists in the captured image PG. The operation input unit 66 transmits a pasting operation of the stored image MG to the information processing apparatus 14 when both positions match or substantially match each other, and otherwise, the position specified on the captured image PG. Is supplied to the identification image operation unit 68.

  The identification image operation unit 68 is connected to the operation input unit 66 and operates the identification image IG on the display screen 18 of the information processing device 14 in accordance with a user operation on the captured image PG. For example, the identification image operation unit 68 gives an instruction to move the identification image IG to a corresponding position on the display screen 18 of the information processing device 14 in response to an operation of designating the position of the captured image PG of the user. Send to.

  The transfer processing unit 70 receives the stored image MG stored in the imaging device 12 via the communication unit 52 and transfers the stored image MG to the information processing device 14 specified by the specifying unit 64, whereby the stored image MG is transferred to the information processing device. 14 display screens 18. In addition, when the stored image can be transferred, the transfer processing unit 70 notifies the information processing apparatus 14 and changes the identification image IG on the display screen 18 of the information processing apparatus 14 and displays it. Let

  As described above, the transfer system 10 provides an environment in which the user can easily perform processing across the imaging device 12 and the information processing device 14 by using the identification image IG.

  FIG. 3 is a flowchart of the transfer process executed in the transfer system 10 of the present embodiment.

  First, in SC50, the identification image display unit 50 of the information processing apparatus 14 displays an image generated by the program being executed by the processing unit 42 on the display screen 18. Here, the identification image display unit 50 acquires the identification image IG corresponding to the information processing apparatus 14 or the identification image IG corresponding to the user who is using the information processing apparatus 14 from the storage unit 46, and displays the display screen. 18 is also displayed. For example, the identification image display unit 50 may display an avatar image corresponding to the information processing apparatus 14 or the user as the identification image IG, and may display a series of moving images that constitute the operation of the user's avatar.

  Further, the processing unit 42 may cause the identification image display unit 50 to change the avatar image and notify the user of the operation method of the transfer processing of the stored image MG. For example, the processing unit 42 causes the identification image display unit 50 to display the avatar image character used as the identification image IG in a transfer processing method (for example, a method in which the user selects the stored image MG and a user stores the stored image). An image for commenting with a speech balloon may be displayed. Further, for example, the processing unit 42 may cause the identification image display unit 50 to display an avatar image indicated by pointing a button on which a help screen or the like is displayed.

  Subsequent to SC50, in SA10, the processing unit 30 of the imaging device 12 causes the imaging unit 32 to image the display screen 18 of the information processing device 14 in accordance with the imaging instruction input by the user to the input unit 38, and the captured image PG. Is generated. The imaging device 12 may use the live view image as the captured image PG.

  In SA <b> 12, the processing unit 30 transmits the captured image PG to the transfer device 16 via the communication unit 40.

  Next, following SA12, in SB30, the captured image acquisition unit 62 of the transfer device 16 acquires the captured image PG from the imaging device 12 via the communication unit 52, and supplies the acquired captured image PG to the specifying unit 64. .

  In SB 32, the captured image acquisition unit 62 transmits to the information processing device 14 an imaging notification that notifies the imaging device 12 that the display screen 18 has been captured.

  Following SC <b> 32 and SC <b> 50, when the processing unit 42 of the information processing device 14 receives an imaging notification via the communication unit 48 in SC <b> 52, the processing unit 42 identifies the identification image IG and the information processing device 14 via the communication unit 48. Information (for example, MAC address and IP address of the information processing device 14) and / or user identification information (for example, user ID), and display image displayed on the display screen 18 by the identification image display unit 50 The DG is transmitted to the transfer device 16.

  Subsequent to SB32 and SC52, in SB34, the identification image acquisition unit 60 of the transfer device 16 acquires the identification image IG from the information processing device 14, and the acquired identification image IG is used as the identification information and / or the user of the information processing device 14. The identification data is stored in the identification data storage unit 54 in association with the identification information. Further, the identification image operation unit 68 acquires the display image DG.

  When there are a plurality of information processing devices 14 around the transfer device 16, the identification image acquisition unit 60 may acquire a plurality of identification images IG and the like from the plurality of information processing devices 14. Further, instead of the information processing device 14 transmitting the identification image IG in SC52 and the transfer device 16 acquiring the identification image IG in SB34, the identification data storage unit 54 of the transfer device 16 stores the identification image IG in advance. You may keep it.

  Next, in SB 36, the specifying unit 64 compares the captured image PG acquired from the imaging device 12 with the identification image IG acquired from the identification data storage unit 54, and any one of the information processing devices in the captured image PG. 14 or whether or not an identification image IG corresponding to the user is included.

  For example, the specifying unit 64 calculates the degree of coincidence between the identification image IG and at least a part of the captured image PG, and determines whether or not the captured image PG includes the identification image IG having a coincidence degree equal to or higher than the reference. to decide.

  Here, the degree of coincidence may be, for example, the ratio of the number of positions that can be associated with the number of positions that cannot associate the positions of the identification image IG and the captured image PG (or a part thereof). Further, the reference may be a predetermined threshold value. Even if one or more identification images IG are acquired, the specifying unit 64 does not have to determine that the identification image IG is included in the captured image PG if all the matching degrees are less than the reference.

  Here, when the specifying unit 64 determines that the display screen 18 in the captured image PG is inclined (for example, as a result of the imaging device 12 imaging the display screen 18 of the information processing device 14 from a direction other than the front, For example, when the outer frame of the display screen 18 has a trapezoidal shape), the specifying unit 64 corrects the inclination of the captured image PG using a homography matrix or the like, and determines whether the corrected captured image PG includes the identification image IG. You can judge.

  Next, in SB38, the specifying unit 64 specifies the information processing apparatus 14 or the user corresponding to the identification image IG included in the captured image PG. For example, the specifying unit 64 refers to the identification data storage unit 54 and specifies the information processing apparatus 14 or the user having the identification information corresponding to the identification image IG determined to be included in the captured image PG.

  Here, the transfer processing unit 70 confirms that the transfer processing is possible for the specified information processing device 14 or the information processing device 14 being used by the specified user via the communication unit 52. Notify that transfer is possible.

  Subsequent to SB 38 and SC 52, when the processing unit 42 of the information processing apparatus 14 receives a transfer enable notification via the communication unit 48 in SC 54, the identification image display unit 50 changes and displays the identification image IG on the display screen 18. Let

  For example, the identification image display unit 50 may display an image in which a character of the avatar image used as the identification image IG comments with a balloon that transfer is possible. Further, for example, the identification image display unit 50 may change the color of the avatar image used as the identification image IG (as an example, the color of the avatar image is changed from red to blue).

  Subsequent to SA12, in SA14, the input unit 38 of the imaging device 12 inputs selection of an image stored in the storage unit 34 from the user. Thereby, the input unit 38 receives the selected image as the stored image MG.

  Next, in SA <b> 16, the processing unit 30 transmits the stored image MG to the transfer device 16 via the communication unit 40.

  Subsequent to SA 16, in SB 40, the control unit 56 of the transfer device 16 acquires the stored image MG via the communication unit 52 and stores the acquired stored image MG in the identification data storage unit 54.

  In SA18, the input unit 38 receives an operation for designating a position on the captured image PG from the user. For example, the input unit 38 inputs an operation for designating a position where the user pastes the stored image MG on the display screen 18 in the captured image PG. The processing unit 30 transmits the specified position on the captured image PG (for example, coordinates on the captured image PG) to the transfer device 16 via the communication unit 40.

  Subsequent to SA18 and SB38, in SB42, the operation input unit 66 of the transfer device 16 determines whether the position specified on the captured image PG matches or substantially matches the position where the identification image IG exists in the captured image PG. To do. For example, the operation input unit 66 acquires the identification image IG from the identification image acquisition unit 60, acquires the captured image PG from the captured image acquisition unit 62, the position where the identification image IG exists in the captured image PG, and the captured image A distance from the position designated on the PG is calculated, and it is determined whether the calculated distance is equal to or less than a predetermined threshold. The operation input unit 66 advances the process to SB46 when the specified position and the position where the identification image IG exists match, or advances the process to SB44 otherwise.

  In SB44, the identification image operation unit 68 of the transfer device 16 moves the identification image IG on the display screen 18 of the information processing device 14 in accordance with a user operation on the captured image PG. For example, the identification image operation unit 68 acquires the captured image PG from the captured image acquisition unit 62, compares the display image DG and the captured image PG, and determines the correspondence between the position on the captured image PG and the position on the display image DG. wear.

  Next, the identification image operation unit 68 calculates a pasting range DA on the display image DG corresponding to the position designated on the captured image PG. As an example of a method for calculating the pasting range DA, the identification image operation unit 68 detects edges such as sides and vertices of objects included in the display image DG and the captured image PG, and associates them with each other.

  For example, the specifying unit 64 associates an edge in the captured image PG showing the same tendency with an edge in the display image DG. Here, the tendency includes a color change, an interval at which the color changes, and the like. The specifying unit 64 associates several positions of the display image DG and the picked-up image PG, which are determined to have the same tendency in the edges.

  The identification image operation unit 68 determines a conversion formula for calculating a corresponding position on the display image DG from a position designated on the captured image PG based on the plurality of positions where the association is completed. The identification image operation unit 68 calculates a pasting range DA corresponding to the position designated on the captured image PG based on the conversion formula. When it is determined that the display image DG is tilted with respect to the captured image PG, the identification image operation unit 68 may correct the tilt of the display image DG using a homography matrix.

  The identification image operating unit 68 supplies the calculated pasting range DA to the information processing apparatus 14 specified by the specifying unit 64 in SB38 together with an instruction to move the identification image IG. Further, the control unit 56 of the transfer device 16 instructs the imaging device 12 to execute the process of SA18 again.

  Following SC <b> 44 and SC <b> 54, in SC <b> 56, the processing unit 42 of the information processing apparatus 14 receives the pasting range DA and the instruction to move the identification image IG via the communication unit 48, and the identification image IG moves to the pasting range DA. The image is displayed on the identification image display unit 50. For example, the processing unit 42 may display a moving image in which the avatar of the identification image IG walks and moves to the pasting range DA on the display screen 18.

  In SB46, the transfer processing unit 70 reads the stored image MG stored in the imaging device 12 from the identification data storage unit 54, and transfers the read stored image MG to the information processing device 14 specified by the specifying unit 64 in SB38. Thus, the stored image MG is pasted on the display screen 18 of the information processing apparatus 14.

  Subsequent to SB46, in SC58, the processing unit 42 of the information processing device 14 receives the stored image MG from the transfer device 16 via the communication unit 48, and the received stored image MG corresponds to the identification image IG in the display screen. Paste to.

  As described above, in the embodiment shown in the flowchart of FIG. 3, the information processing apparatus 14 moves the position of the identification image IG on the display screen to indicate the position where the memory image MG designated by the user is pasted. . Instead of this, the information processing apparatus 14 may display the identification image IG fixedly on the display screen, and may not cause the identification image IG to follow the pasting position designated by the user.

  In this case, after SA18 and SB40, the transfer device 16 executes the processing of SB42 to SB46, and the information processing device 14 executes the processing of SC54 and SC56. Instead, the operation input unit 66 of the transfer device 16 performs imaging. The captured image PG is acquired from the image acquisition unit 62, the display image DG and the captured image PG are compared, the correspondence between the position on the captured image PG and the position on the display image DG is specified, and the image is specified on the captured image PG. A pasting range DA on the display image DG corresponding to the position is calculated. Next, the transfer processing unit 70 acquires the pasting range DA from the operation input unit 66, acquires the stored image MG from the identification data storage unit 54, and transfers the pasting range DA and the stored image MG to the information processing device 14. Subsequently, the information processing apparatus 14 executes the process of SC58. Thereby, the transfer device 16 pastes the stored image MG on the display screen 18 of the information processing device 14 while fixing the identification image IG.

  Next, display forms such as the identification image IG, the captured image PG, and the storage image MG will be described.

  FIG. 4 is a diagram of the identification image IG displayed on the display screen 18. It is assumed that the identification image display unit 50 of the information processing apparatus 14 displays a human avatar image as shown in FIG. 4 as the identification image IG. The information processing apparatus 14 may display an avatar such as a cursor, a person, an animal or plant, or an article, an icon, an illustration, a photograph, or the like as the identification image IG instead of / in addition to the person's avatar image. Further, in addition to these images, the information processing apparatus 14 may display identification information such as character information indicating information related to the information processing apparatus 14, a barcode, or a two-dimensional code as an identification image IG.

  FIG. 5 is a captured image PG obtained by capturing the display image DG of the information processing apparatus 14 from the front. As illustrated, the captured image PG includes an information processing apparatus 14 that displays a display image DG including the identification image IG illustrated in FIG. 4 on the display screen 18.

  FIG. 6 is a diagram of a captured image PG obtained by capturing the display image DG from diagonally forward. As shown in FIG. 6, when the imaging device 12 is disposed obliquely forward of the information processing device 14, for example, the left front, the display image DG and the identification image IG in the captured image PG are images that are compressed left and right. It becomes. In this case, the specifying unit 64 and the identification image operation unit 68 may correct the captured image PG into an image as shown in FIG. 5 using a homography matrix.

  FIG. 7 is a diagram illustrating the designation of the position on the captured image PG by the user. For example, as illustrated in FIG. 7, the information processing apparatus 14 displays a plurality of pasting ranges DA as pasting destination candidates on the display screen 18 in the display image DG included in the captured image PG. The information processing apparatus 14 displays the inside of one pasting range DA1 among the plurality of pasting ranges DA on the identification image IG.

  In the state illustrated in FIG. 7, the input unit 38 of the imaging device 12 performs an operation (for example, an operation in which the user touches with a finger FN or the like) specifying the position of the pasting range DA1 including the identification image IG on the captured image PG. When input, the transfer processing unit 70 of the transfer device 16 accepts the pasting range DA1 as a pasting destination of the stored image MG, and transmits the stored image MG received from the imaging device 12 to the information processing device 14.

  FIG. 8 is a diagram illustrating the display screen 18 on which the stored image MG is pasted. The processing unit 42 of the information processing device 14 receives the stored image MG. As a result, the processing unit 42 pastes and displays the stored image MG at the position of the pasting range DA1 corresponding to the identification image on the display screen 18 as shown in FIG. As shown in FIG. 8, the information processing apparatus 14 does not have to display the identification image IG on the display screen 18 that has been pasted, but instead displays the identification image IG in the pasting range DA1 or another pasting range DA. You may continue to display.

  FIG. 9 is a diagram for explaining the operation of the identification image IG on the captured image PG by the user. Here, the information processing apparatus 14 displays a plurality of pasting ranges DA, which are pasting destination candidates, on the display screen 18 in the display image DG included in the captured image PG, as in FIG. It is assumed that the identification image IG is displayed in advance.

  Here, an operation in which the input unit 38 of the imaging device 12 specifies the position of the lower right pasting range DA that does not include the identification image IG on the captured image PG (for example, an operation in which the user touches with the finger FN or the like). When input, the identification image operation unit 68 of the transfer device 16 accepts the designated lower right pasting range DA as the pasting destination of the stored image MG, and on the display image DG corresponding to the designated position on the captured image PG. Information such as the coordinates of the pasting range DA is transmitted to the information processing device 14.

  In response to this, the processing unit 42 of the information processing apparatus 14 moves the identification image IG to the position in the lower right pasting range DA received by the identification image display unit 50. For example, the identification image display unit 50 may display an operation in which the avatar that is the identification image IG is walking or running toward the pasting range DA. Here, when the input unit 38 of the imaging device 12 inputs an operation for designating the position of the pasting range DA to which the identification image IG has moved in the captured image PG, the transfer processing unit 70 of the transfer device 16 causes the pasting range to be included. The stored image MG is pasted on DA.

  FIG. 10 is a diagram illustrating an identification image IG that moves so as to be imaged by the imaging device 12. Here, the imaging device 12 captures the upper left part of the display screen 18 ′ of the information processing device 14 as the captured image PG, and does not capture the remaining portion (portion indicated by a dotted line). Since the information processing device 14 displays the identification image IG in a portion that is not captured by the imaging device 12, the identification screen PG does not include the identification image IG. In such a case, the imaging device 12 cannot specify the information processing device 14 to be operated.

  Therefore, when the identification image IG is not captured on the display screen 18 in the captured image PG and the transfer device 16 cannot identify the information processing device 14 in the process of SB38 in FIG. Instructs the information processing apparatus 14 that has received the identification image IG in SB 34 to move the identification image IG on the display screen 18 until the identification image IG appears in the imaging screen PG. 12 may be instructed to image the display screen 18 of the information processing apparatus 14 again.

  In response to this, the processing unit 42 of the information processing device 14 moves the identification image IG to the identification image display unit 50 so as to cover the entire area on the display screen 18, and the imaging device 12 performs the process of SA10. And display the display screen 18 of the information processing apparatus 14. The processing unit 42 of the information processing apparatus 14 may move the identification image IG in a zigzag manner on the display screen 18 as shown in FIG. 10, and instead moves around the frame of the display screen 18. You may let them.

  In the transfer system 10 described above, the identification unit 64 of the transfer device 16 compares the identification image IG acquired by the identification image acquisition unit 60 with the captured image PG acquired by the captured image acquisition unit 62, and determines the identification image IG. The information processing device 14 to be imaged is specified. Thereby, in the transfer system 10, the transfer device 16 can specify the information processing device 14 that is the transfer destination even if the information specifying the information processing device 14 is not directly input from the user. Therefore, the transfer device 16 can improve the degree of freedom of the transfer destination.

  In the transfer system 10 described above, when the transfer device 16 acquires the stored image MG from the imaging device 12, the operation input unit 66 transmits a paste instruction to the information processing device 14 specified as the transfer destination by the specifying unit 64. When the information processing apparatus 14 receives the paste instruction, it pastes the stored image MG on the display screen 18. Thereby, the user can paste and display the stored image MG on the information processing device 14 by performing a simple operation such as selecting the stored image MG with the imaging device 12. As a result, the transfer system 10 can provide an environment in which the user can easily perform processing across the imaging device 12 and the information processing device 14.

  In the transfer system 10 described above, the stored image MG is pasted at the position of the identification image IG on the display screen 18 of the information processing apparatus 14. Further, the user can change the position of the identification image IG, that is, the pasting range DA of the stored image MG, by designating the position on the captured image PG using the input unit 38 of the imaging device 12.

  In the transfer system 10 described above, even when the imaging device 12 captures the display screen 18 from diagonally forward of the information processing device 14, the specifying unit 64 uses the homography matrix or the like according to the inclination of the captured image PG. Then, the captured image PG is corrected to determine whether the identification image IG is included. Thereby, the specifying unit 64 can specify the information processing apparatus 14 that is the transfer destination regardless of the positional relationship between the imaging apparatus 12 and the information processing apparatus 14.

  In the transfer system 10 described above, the transfer device 16 specifies, as a transfer destination, the information processing device 14 in which the degree of coincidence between the captured image PG and the identification image IG is greater than or equal to the reference. As a result, the transfer device 16 maintains the accuracy and has a high probability with respect to the information processing device of the transfer destination as compared with the case where the transfer destination is specified when the degree of coincidence between the captured image PG and the identification image IG completely matches. 14 can be specified.

  In the transfer system 10 described above, the transfer device 16 causes the information processing device 14 to move the identification image IG on the display screen 18 when the captured image PG does not include the identification image IG. Thereby, even when the imaging device 12 images only a part of the display screen 18 and the identification image IG cannot be captured by the first imaging, the transfer device 16 can specify the information processing device 14 to be operated. it can.

  Here, the transfer system 10 according to a modification of the present embodiment will be described.

  In the modification of the present embodiment, the transfer device 16 can perform authentication of the imaging device 12 by executing challenge / response authentication between the imaging device 12 and the information processing device 14. For example, the transfer device 16 receives an authentication request from the imaging device 12 and transfers the received authentication request to the information processing device 14.

  The processing unit 42 of the information processing device 14 causes the identification image display unit 50 to change the identification image IG that is an avatar image in response to the authentication request received from the imaging device 12 via the transfer device 16. For example, the identification image display unit 50 displays on the display screen 18 an image in which the right direction is directed to the avatar and an image of a sentence that asks the user which direction the avatar is directed.

  On the other hand, the processing unit 30 of the imaging device 12 inputs an operation for a position on the imaging screen PG of the user via the input unit 38 and transmits the input operation to the transfer device 16. The operation input unit 66 of the transfer device 16 receives the input operation and determines whether the operation is a correct operation. For example, the operation input unit 66 determines whether or not the input operation is an operation for designating the right position of the avatar facing right. If the operation input unit 66 determines that the operation is correct, the operation input unit 66 may authenticate the imaging device 12 and execute an operation for transferring the stored image MG from the imaging device 12.

  In the present embodiment, an example is shown in which the user operates while looking at the identification image IG displayed on the display screen 18 of the information processing apparatus 14, and the identification image IG moves to a position to be operated on the display screen 18. However, the display for the user's operation is not limited to this. For example, the information processing apparatus 14 may fix the position where the identification image IG is displayed, and display a mark image at a position to be operated on the display screen 18 of the information processing apparatus 14 separately from the identification image IG. .

  As an example, the mark image is an icon, a cursor, an avatar, or the like. In this form, the operation input unit 66 displays a mark image at a corresponding position on the display screen 18 of the information processing apparatus 14 in response to the user specifying a position on the captured image PG. In this case, the operation input unit 66 converts the coordinates on the captured image PG to the coordinates on the display screen 18 and displays the mark image. In addition, the operation input unit 66 moves the mark image to a corresponding position on the display screen of the information processing device 14 in response to an operation on the mark image captured in the captured image PG on the imaging device 12. You may let them.

  Further, when there are a plurality of imaging devices 12, the operation input unit 66 may display different mark images for each imaging device 12 on the display screen 18 of one information processing device 14. On the other hand, even when there are a plurality of imaging devices 12, the operation input unit 66 may preferentially display the mark image by the imaging device 12 selected by the user of the information processing device 14. In the display mode with priority, for example, the operation input unit 66 displays only the mark image on the display screen 18 or highlights the mark image on the display screen 18.

  When a plurality of mark images are displayed on the display screen 18 of the information processing apparatus 14, the operation input unit 66 may count the mark images for each region within a certain range. For example, in a state where the transfer processing unit 70 displays the plurality of stored images MG of the plurality of imaging devices 12 on the display screen 18, the operation input unit 66 displays on the area where each stored image MG is displayed. By counting the marked image, the transfer device 16 can execute voting of the stored image MG.

  Further, the operation input unit 66 may appropriately change reception of the position of the mark image by the user. For example, the operation input unit 66 may cause the imaging device 12 to display the display screen 18 as a moving image, that is, accept a user input in a live view state and display a mark image on the display screen 18. Further, the operation input unit 66 may receive a user input and display a mark image in a state where the imaging device 12 displays the display screen 18 as a still image.

  In the above-described embodiment, the transfer processing unit 70 has shown the example in which the stored image MG is pasted to the pasting range DA where no image is displayed. However, the pasting form is not limited to this. For example, the transfer processing unit 70 may replace and paste the new stored image MG to the pasting range DA to which the stored image MG is pasted.

  In this case, the transfer processing unit 70 may paste a new stored image MG in accordance with the position and size of the previous stored image MG. Further, the transfer processing unit 70 may paste the stored image MG in a region where an image other than the stored image MG transferred from the imaging device 12 is pasted. The transfer processing unit 70 may paste a new stored image MG at the position of the stored image MG selected by the user. Furthermore, when pasting a new stored image MG, the transfer processing unit 70 may delete the previous stored image MG to be replaced selected by the user from the identification data storing unit 54.

  In the above-described embodiment, the transfer processing unit 70 pastes the stored image MG. However, the transfer processing unit 70 displays the pasting information instead of the stored image MG or pasted information in addition to the stored image MG. It may be displayed on the screen 18. For example, the transfer processing unit 70 may display caption information describing the stored image MG on the display screen 18 as pasting information.

  Further, the transfer processing unit 70 searches the storage unit 34 of the imaging device 12 and the storage unit 46 of the information processing device 14 to extract information related to the storage image MG to be pasted and generate paste information. Also good. For example, the transfer processing unit 70 may search the information on the schedule based on the captured position and date / time included in the stored image MG to generate the pasting information including the scheduled name.

  Further, the transfer processing unit 70 may recognize information included in the stored image MG by image recognition and generate pasting information related to the recognized target. For example, the transfer processing unit 70 may determine a person included in the stored image MG by image recognition, and use the person's name, email address, or the like as the pasting information.

  Furthermore, when the pasting range DA to which the stored image MG is pasted is an application icon, the transfer processing unit 70 may activate the application and paste the stored image MG. For example, when the application is an application related to mail, the transfer processing unit 70 may activate the mail application and attach the stored image MG to the transmission mail.

  The connection relationship, function, and the like of each component in the above-described embodiment may be changed as appropriate. Moreover, you may combine embodiment mentioned above.

  For example, in the above-described embodiment, the transfer device 16 includes the identification image acquisition unit 60, the captured image acquisition unit 62, the specifying unit 64, the operation input unit 66, and the transfer processing unit 70. However, the present invention is not limited to this. A part or all of the configuration of the transfer device 16 may be provided in the imaging device 12 or the information processing device 14. For example, some or all of the identification image acquisition unit 60, the captured image acquisition unit 62, the identification unit 64, the operation input unit 66, the identification image operation unit 68, and the transfer processing unit 70 may be replaced with the imaging device 12 or the information processing device 14. May be provided.

  FIG. 11 shows an example of a hardware configuration of a computer 1900 according to this embodiment. A computer 1900 according to the present embodiment is an example of an information processing unit. The computer 1900 includes a CPU peripheral unit having a CPU 2000, a RAM 2020, a graphic controller 2075, and a display unit 2080 that are connected to each other by a host controller 2082, and a communication interface 2030 that is connected to the host controller 2082 by an input / output controller 2084. And an input / output unit having a hard disk drive 2040 and a legacy input / output unit having a ROM 2010, a memory drive 2050 and an input / output chip 2070 connected to the input / output controller 2084.

  The host controller 2082 connects the RAM 2020 to the CPU 2000 and the graphic controller 2075 that access the RAM 2020 at a high transfer rate. The CPU 2000 operates based on programs stored in the ROM 2010 and the RAM 2020 and controls each unit. The graphic controller 2075 acquires image data generated by the CPU 2000 or the like on a frame buffer provided in the RAM 2020 and displays it on the display unit 2080. Instead of this, the graphic controller 2075 may include a frame buffer for storing image data generated by the CPU 2000 or the like.

  The input / output controller 2084 connects the host controller 2082 to the communication interface 2030 and the hard disk drive 2040 that are relatively high-speed input / output devices. The communication interface 2030 communicates with other devices via a network. The hard disk drive 2040 stores programs and data such as a display program used by the CPU 2000 in the computer 1900.

  The input / output controller 2084 is connected to the ROM 2010, the memory drive 2050, and the relatively low-speed input / output device of the input / output chip 2070. The ROM 2010 stores a boot program that the computer 1900 executes at startup and / or a program that depends on the hardware of the computer 1900. The memory drive 2050 reads a program or data such as a display program from the memory card 2090 and provides it to the hard disk drive 2040 via the RAM 2020. The input / output chip 2070 connects the memory drive 2050 to the input / output controller 2084, and also connects various input / output devices to the input / output controller 2084 via, for example, a parallel port, a serial port, a keyboard port, a mouse port, and the like. Connect to.

  A program provided to the hard disk drive 2040 via the RAM 2020 is stored in a recording medium such as a memory card 2090 or an IC card and provided by a user. A program such as a display program is read from a recording medium, installed in the hard disk drive 2040 in the computer 1900 via the RAM 2020, and executed by the CPU 2000.

  A program installed in the computer 1900 and causing the computer 1900 to function as the imaging device 12, the information processing device 14, and the transfer device 16 includes a processing module, a captured image acquisition module, an identification image acquisition module, a specific module, an operation input module, and an identification image. An operation module and a transfer processing module are provided. These programs or modules work on the CPU 2000 or the like to change the computer 1900 into the processing unit 30 of the imaging device 12, the processing unit 42 of the information processing device 14, and the identification image acquisition unit 60 included in the control unit 56 of the transfer device 16. The captured image acquisition unit 62, the specifying unit 64, the operation input unit 66, the identification image operation unit 68, and the transfer processing unit 70 are caused to function.

  The information processing described in these programs is read by the computer 1900, whereby the processing unit 30 of the imaging device 12, which is a specific means in which the software and the various hardware resources described above cooperate, the information processing device 14, the identification image acquisition unit 60, the captured image acquisition unit 62, the identification unit 64, the operation input unit 66, the identification image operation unit 68, and the transfer processing unit 70 included in the control unit 56 of the transfer device 16. Function. And the specific transfer apparatus 16 according to the intended purpose is constructed | assembled by implement | achieving the calculation or processing of the information according to the intended purpose of the computer 1900 in this embodiment by these specific means.

  As an example, when communication is performed between the computer 1900 and an external device or the like, the CPU 2000 executes a communication program loaded on the RAM 2020 and executes a communication interface based on the processing content described in the communication program. A communication process is instructed to 2030. Under the control of the CPU 2000, the communication interface 2030 reads transmission data stored in a transmission buffer area or the like provided on a storage device such as the RAM 2020, the hard disk drive 2040, or the memory card 2090, and transmits it to the network. The reception data received from the network is written into a reception buffer area or the like provided on the storage device. As described above, the communication interface 2030 may transfer transmission / reception data to / from the storage device by a DMA (direct memory access) method. Instead, the CPU 2000 transfers the storage device or the communication interface 2030 as a transfer source. The transmission / reception data may be transferred by reading the data from the data and writing the data to the communication interface 2030 or the storage device of the transfer destination.

  Further, the CPU 2000 causes the RAM 2020 to read all or necessary portions from the files or databases stored in the external storage device such as the hard disk drive 2040 and the memory drive 2050 (memory card 2090) into the RAM 2020 by DMA transfer or the like. Various processes are performed on the data on the RAM 2020. Then, CPU 2000 writes the processed data back to the external storage device by DMA transfer or the like. In such processing, since the RAM 2020 can be regarded as temporarily holding the contents of the external storage device, in the present embodiment, the RAM 2020 and the external storage device are collectively referred to as a memory, a storage unit, or a storage device. Various types of information such as various programs, data, tables, and databases in the present embodiment are stored on such a storage device and are subjected to information processing. Note that the CPU 2000 can also store a part of the RAM 2020 in the cache memory and perform reading and writing on the cache memory. Even in such a form, the cache memory bears a part of the function of the RAM 2020. Therefore, in the present embodiment, the cache memory is also included in the RAM 2020, the memory, and / or the storage device unless otherwise indicated. To do.

  In addition, the CPU 2000 performs various operations, such as various operations, information processing, condition determination, information search / replacement, etc., described in the present embodiment, specified for the data read from the RAM 2020 by the instruction sequence of the program. Is written back to the RAM 2020. For example, when performing the condition determination, the CPU 2000 determines whether the various variables shown in the present embodiment satisfy the conditions such as large, small, above, below, equal, etc., compared to other variables or constants. When the condition is satisfied (or not satisfied), the program branches to a different instruction sequence or calls a subroutine. Further, the CPU 2000 can search for information stored in a file or database in the storage device.

  The program or module shown above may be stored in an external recording medium. As the recording medium, in addition to the memory card 2090, an optical recording medium such as DVD or CD, a magneto-optical recording medium such as MO, a tape medium, a semiconductor memory such as an IC card, or the like can be used. Further, a storage device such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet may be used as a recording medium, and the program may be provided to the computer 1900 via the network.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

DESCRIPTION OF SYMBOLS 10 Transfer system 12 Imaging device 14 Information processing device 16 Transfer device 18 Display screen 20 Display screen 30 Processing unit 32 Imaging unit 34 Storage unit 36 Display unit 38 Input unit 40 Communication unit 42 Processing unit 44 Input unit 46 Storage unit 48 Communication unit 50 Identification image display unit 52 Communication unit 54 Identification data storage unit 56 Control unit 60 Identification image acquisition unit 62 Captured image acquisition unit 64 Identification unit 66 Operation input unit 68 Identification image operation unit 70 Transfer processing unit 1900 Computer 2000 CPU
2010 ROM
2020 RAM
2030 Communication interface 2040 Hard disk drive 2050 Memory drive 2070 Input / output chip 2075 Graphic controller 2080 Display unit 2082 Host controller 2084 Input / output controller 2090 Memory card

Claims (12)

An identification image acquisition unit for acquiring an identification image for identifying the information processing device or a user of the information processing device displayed on a display screen of the information processing device;
A captured image acquisition unit that acquires a captured image of the display screen imaged by the imaging device;
A specific unit that compares the identification image and the captured image and identifies the information processing apparatus or the user corresponding to the identification image;
A transfer processing unit that transfers a stored image stored in the imaging device to the information processing device and pastes the stored image at a position corresponding to the identification image in the display screen of the information processing device;
A device comprising: The apparatus according to claim 1, wherein the identification image is identifiable by human vision. The transfer processing unit, when the transfer of the stored image becomes possible, according to claim 1 or 2 is displayed by changing the identification image on the display screen to the information processing apparatus. An operation input unit that inputs an operation on a position on the display screen corresponding to the position in response to the position on the captured image being designated;
An identification image operation unit that moves the identification image on the display screen in response to a user operation on the captured image;
Apparatus according to any one of claims 1-3, further comprising a. The identification image operation unit moves the identification image on the display screen until the identification image appears in the imaging screen when the identification image is not captured on the display screen in the captured image. ,
The apparatus according to claim 4 . An identification data storage unit that stores correspondences between the plurality of identification images and the plurality of information processing apparatuses or the users;
The specifying unit searches the identification image from the identification data storage unit and specifies the information processing apparatus or the user corresponding to the identification image.
The apparatus according to any one of claims 1 to 5 . An apparatus according to any one of claims 1 to 3 ;
An identification image display unit for displaying the identification image on the display screen;
A communication unit that transmits the identification image to the device and receives the stored image from the device;
An information processing apparatus comprising: The identification image includes an image of the avatar of the user,
The information processing apparatus according to claim 7 . The identification image display unit changes the image of the avatar to notify the user of a transfer processing operation method,
The information processing apparatus according to claim 8 . The identification image display unit changes the image of the avatar in response to an authentication request received from the imaging device via the device.
The information processing apparatus according to claim 8 or 9 . Executed by the computer,
An identification image acquisition step for acquiring an identification image for identifying the information processing device or a user of the information processing device displayed on the display screen of the information processing device;
A captured image acquisition step of acquiring a captured image of the display screen imaged by the imaging device;
A specific step of comparing the identification image and the captured image to identify the information processing apparatus or the user corresponding to the identification image;
A transfer processing step of transferring a stored image stored in the imaging device to the information processing device and pasting the stored image at a position corresponding to the identification image in the display screen of the information processing device;
A method comprising: An identification image acquisition function for acquiring an identification image for identifying the information processing device or a user of the information processing device displayed on the display screen of the information processing device;
A captured image acquisition function for acquiring a captured image of the display screen imaged by the imaging device;
A specific function for comparing the identification image and the captured image to identify the information processing apparatus or the user corresponding to the identification image;
A transfer processing function of transferring a stored image stored in the imaging device to the information processing device and pasting the stored image at a position corresponding to the identification image in the display screen of the information processing device;
A program that causes a computer to execute.

Images