JP6514632B2 - INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING METHOD, INFORMATION PROCESSING APPARATUS, SERVER, AND PROGRAM - Google Patents

Description

  The present invention relates to an information processing system, an information processing method, an information processing apparatus, a server, and a program.

  Various methods for sharing comment information between different information processing apparatuses have been proposed. For example, in Patent Document 1, when distributing a moving image by streaming distribution, a technology for displaying a comment to which a moving image reproduction time based on the reproduction start time of the moving image is given to each terminal device browsing the moving image. Is disclosed. In this technique, in accordance with the moving image reproduction time after the start of the moving image reproduction, the comment contents for which the comment giving time that matches the moving image reproduction time is set are sequentially displayed on the moving image.

JP, 2009-077443, A

  When the application itself to be executed by the information processing apparatus implements the information sharing function as in the above technology, it goes without saying that the application not implementing the information sharing function shares information such as a comment with another information terminal apparatus. I can not

  The present invention has been made in view of the above points, and realizes sharing of information triggered by a display image between different terminals without implementing an additional function in an application that displays an image on the screen of the terminal itself. The purpose is to provide technology.

  A first aspect of the present invention is an information processing system including a first information processing apparatus and a second information processing apparatus capable of communicating with the first information processing apparatus. In this system, the first information processing apparatus includes a display unit that displays information to be presented to a user of the first information processing apparatus, and a screen acquisition unit that acquires a display screen of the display unit as an image. The second information processing apparatus includes a storage unit that associates and stores template information used to determine the relevancy of images and information indicating a process performed by the first information processing apparatus. When the image of the display screen acquired by the screen acquisition unit is associated with the template information stored in the storage unit of the second information processing apparatus, the first information processing apparatus indicates a process indicated by the information stored by the storage unit. And an execution unit that executes the program.

  The storage unit may store display information to be processed by display processing executed by the execution unit, and the execution unit may be configured to use the second information processing apparatus as the image of the display screen acquired by the screen acquisition unit. The display information may be displayed on the display unit when the storage unit is associated with the template information stored in the storage unit.

  The execution unit may acquire the information from the storage unit when the display screen is associated with the template information.

  The storage unit may receive and store information indicating processing executed by the first information processing apparatus from a third information processing apparatus different from the second information processing apparatus.

  The first information processing apparatus may further include an information transmitting unit that transmits information indicating processing performed by the third information processing apparatus.

  The first information processing apparatus may further include a relevance determination unit that obtains relevance between the template information acquired from the storage unit and the display screen acquired by the screen acquisition unit.

  The screen acquisition unit may transmit the acquired display screen to the second information processing apparatus, and the second information processing apparatus transmits the template information stored in the storage unit and the image acquisition unit. The information processing apparatus may further include a relevance determination unit that obtains relevance to the displayed display screen.

  The storage unit may store a hash value calculated from an image as the template information, and the relevance determination unit may calculate the hash value calculated from the image of the display screen and the hash value stored by the storage unit. The degree of matching may be output as the association between the template information and the display screen.

  The storage unit may store the template information for each screen ratio of an image displayable by the display unit.

  A second aspect of the present invention is an information processing method executed in an information processing system including a first information processing apparatus and a second information processing apparatus capable of communicating with the first information processing apparatus. In this method, the processor of the first information processing apparatus displays information presented to the user of the first information processing apparatus on a display unit of the first information processing apparatus, and a display screen of the display unit When the step of acquiring as an image, the template information used for determining the relevancy of the image stored in the second information processing apparatus, and the image of the display screen are associated, the second information is associated with the template information. And performing a process indicated by the information stored in the processing device.

  A third aspect of the present invention is a program that causes a computer to realize the above. This program may be provided as part of firmware incorporated in an apparatus to perform basic control of hardware resources such as a computer and a mobile phone. This firmware is stored, for example, in a semiconductor memory such as a ROM (Read Only Memory) or a flash memory in the device, and is executed by a processing processor such as a CPU (Central Processing Unit). In order to provide the firmware or update a part of the firmware, a computer readable non-volatile recording medium having the program recorded thereon may be provided, and the program is transmitted via a communication line. It is also good.

  A fourth aspect of the present invention is an information processing apparatus. This information processing apparatus determines the relevancy of the image from a display unit for displaying information, a screen acquisition unit for acquiring a display screen of the display unit as an image, and another information processing apparatus different from the information processing apparatus A template acquiring unit for acquiring template information used for the image, an information acquiring unit for acquiring information indicating a process to be executed by the information processing apparatus linked to the template, and an image of the display screen acquired by the screen acquiring unit And an execution unit configured to execute a process indicated by the information, when associated with the template information.

  A fifth aspect of the present invention is a server capable of communicating with an information processing apparatus. A storage unit configured to associate and store template information used to determine the relevancy of an image displayed on the display screen of the information processing apparatus and information indicating a process to be executed by the information processing apparatus; A communication unit for transmitting the template information to the information processing apparatus, and the communication unit is further executed by the information processing apparatus when an image displayed on a display screen of the information processing apparatus is associated with the template information Information indicating processing to be performed is transmitted to the information processing apparatus in response to a request from the information processing apparatus.

  It is to be noted that any combination of the above-described components, and a conversion of the expression of the present invention among a method, an apparatus, a system, a computer program, a data structure, a computer readable recording medium, etc. is also an aspect of the present invention. It is valid.

  According to the present invention, it is possible to realize sharing of information triggered by a display image between different terminals without implementing an additional function in the application itself for displaying an image on the screen of the terminal.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows typically the outline | summary of the information processing system which concerns on embodiment. It is a figure showing typically the functional composition of the information processing system concerning an embodiment. It is a figure which shows typically the data structure of the information which the memory | storage part of the 2nd information processing apparatus which concerns on embodiment memorize | stores. It is a figure which shows an example of template information. It is a figure which shows an example of the image of the display screen which a screen acquisition part acquires. It is a flowchart explaining the flow of the information processing performed with the information processing system concerning an embodiment. It is a figure showing typically the functional composition of the information processing system concerning the 1st modification of an embodiment. FIGS. 8A and 8B illustrate a method of calculating the hash value of an image. It is a figure which shows typically the data structure of the information which the memory | storage part of the 2nd information processing apparatus which concerns on a 1st modification memorize | stores. It is a figure which shows typically the outline | summary of the information processing system which concerns on a 2nd modification. It is a figure which shows an example of the stamp image selection screen which the display part which concerns on a 4th modification displays.

[Overview of the embodiment]
FIG. 1 is a view schematically showing an outline of an information processing system S according to the embodiment. An outline of an information processing system S according to the embodiment will be described with reference to FIG.
An information processing system S according to the embodiment of the present invention at least includes a first information processing device 1 and a second information processing device 2 which is an information processing device different from the first information processing device 1. As shown in FIG. 1, the first information processing apparatus 1 and the second information processing apparatus 2 are connected via a network N, and can communicate with each other.

  The first information processing apparatus 1 includes a display unit 10 for displaying information to be presented to the user of the first information processing apparatus 1. In the information processing system S according to the embodiment, when the display screen of the display unit 10 of the first information processing apparatus 1 becomes a predetermined display screen, the first information processing apparatus 1 is associated with the display screen. Execute the process. For example, when the user of the first information processing apparatus 1 reaches a specific stage while the game is being executed by the first information processing apparatus 1, the predetermined help information is displayed on the display unit 10 of the first information processing apparatus 1 Execute the display process.

  In order to realize this, the second information processing apparatus 2 includes template information for determining whether the display unit of the first information processing apparatus 1 is a predetermined display screen, and the first information processing apparatus 1 It is stored in association with information indicating processing to be executed. The first information processing apparatus 1 and the second information processing apparatus 2 realize the above-described processing by exchanging a display screen, template information, and information indicating processing via the network N.

  Therefore, the network N may have any form as long as the first information processing apparatus 1 and the second information processing apparatus 2 can communicate with each other. Specific examples of the network N include the Internet via a wireless LAN (Local Area Network) or a wired LAN, ad hoc communication using a wireless network such as WiFi (registered trademark), or communication via a mobile telephone communication network, or Any combination. The first information processing apparatus 1 is, for example, a smartphone, a tablet, a fablet, a mobile phone, a notebook PC (Personal Computer), a mobile game machine, an electronic dictionary, a mobile terminal such as a mobile TV, a wearable computer such as a smart watch or smart glass Alternatively, it is a stationary electronic device such as a television or a desktop PC. Moreover, in addition to the specific example of the first information processing apparatus 1 described above, a specific example of the second information processing apparatus 2 is a server such as a mainframe server or a cloud server. In the example shown in FIG. 1, the first information processing apparatus 1 is a smartphone, the second information processing apparatus 2 is a server, and the network N is the Internet and a mobile phone communication network.

[Functional configuration of information processing system]
FIG. 2 is a diagram schematically showing a functional configuration of the information processing system S according to the embodiment. As described above with reference to FIG. 1, the information processing system S includes the first information processing device 1 and the second information processing device 2 capable of communicating with the first information processing device 1 via the network N.
The first information processing apparatus 1 includes a display unit 10, a screen acquisition unit 11, a communication unit 12, a storage unit 13, a relevance determination unit 14, an execution unit 15, an input reception unit 16, and an information transmission unit 17. The second information processing apparatus 2 further includes a control unit 20, a storage unit 21, and a communication unit 22.

  The display unit 10 of the first information processing device 1 displays information to be presented to the user of the first information processing device 1. The display unit 10 is, for example, a liquid crystal panel or an organic electroluminescence (EL) panel. The screen acquisition unit 11 acquires the display screen of the display unit 10 as an image. The storage unit 13 is an operating system (Operating System; hereinafter referred to as “OS”) for controlling the respective units of the first information processing apparatus 1 and solid state drives (SSDs) or HDDs that store various software applications. Nonvolatile memories such as (Hard Disk Drive) and volatile memories such as Dynamic Random Access Memory (DRAM) that stores data generated by these programs.

  The screen acquisition unit 11 can be realized, for example, using a screen capture function provided by the OS. The screen acquisition unit 11 acquires the display screen displayed by the display unit 10 and stores the display screen in the storage unit 13.

  The communication unit 12 exchanges data with the second information processing apparatus 2 via the network N. The communication unit 12 can be realized by using, for example, an M2M (Machine-to-Machine) module such as a WiFi module or a mobile phone communication module.

  The relevance determination unit 14 acquires template information from the second information processing apparatus 2 via the communication unit 12 and the network N. Here, “template information” is information used by the relevance determination unit 14 to determine whether the display screen acquired by the screen acquisition unit 11 is associated with a predetermined screen. In this sense, the relevance determination unit 4 functions as a template acquisition unit in the first information processing apparatus 1. A specific example of the template information is an image of a predetermined display screen itself, a template image obtained by processing at least a part of the display screen, or information such as a hash value generated from the display screen. The relevance determination unit 14 determines the presence or absence of a relationship between the template information acquired from the second information processing apparatus 2 and the display screen of the display unit 10 acquired by the screen acquisition unit 11. Details of the relevance determination processing by the relevance determination unit 14 will be described later.

  The storage unit 21 of the second information processing apparatus 2 associates and stores the above-described template information and information indicating the process to be executed by the first information processing apparatus 1. Here, “information indicating processing to be executed” means that the image on the display screen of the display unit 10 acquired by the screen acquisition unit 11 of the first information processing apparatus 1 is related to the template information from the second information processing apparatus 2 These are information for specifying the processing to be executed by the first information processing apparatus 1 and the processing target.

  A display process is mentioned as an example of the process which the 1st information processing apparatus 1 should perform. In this case, the storage unit 21 stores the information indicating that the process to be executed by the first information processing apparatus 1 is the display process and the display information to be the process target of the display process in association with the template information. . When the process to be executed by the first information processing apparatus 1 is fixed, the “information indicating the process to be executed” may be only the process target of the display process. For example, when "processing to be executed" is determined to be display processing, "information indicating processing to be executed" is a comment or the like to be processed, and information for specifying "display processing" itself is not necessarily included. May be

  The display information may be in any form as long as the information can be displayed by the display unit 10 of the first information processing apparatus 1 such as character information, an image, and a moving image. Moreover, although the process which the 1st information processing apparatus 1 should perform may be what kind of process, it is not restricted to a display process, Below, it describes about the case of a display process.

  When the relevance determining unit 14 determines that the image on the display screen of the display unit 10 is related to the template information, the execution unit 15 of the first information processing device 1 takes that as a trigger for the second information processing device 2. Information is acquired from the storage unit 21. In this sense, the execution unit 15 functions as an information acquisition unit in the first information processing apparatus 1. The template information stored in the storage unit 21 of the second information processing apparatus 2 and the information indicating the processing executed by the first information processing apparatus 1 are not limited to the first via the communication unit 22 and the network N under the control of the control unit 20. It is transmitted to the information processing device 1. When the image on the display screen of the display unit 10 is associated with the template information, the execution unit 15 causes the display unit 10 to display the acquired display information. The relevance determining unit 14 sequentially acquires the display information at timing when the relationship between the image of the display screen of the display unit 10 and the template information is confirmed, thereby the first information processing apparatus 1 and the second information processing apparatus 2 It is possible to suppress the frequency of communication between

  The relevance determination unit 14 periodically acquires display information from the second information processing apparatus 2 independently of the presence or absence of the association between the image on the display screen of the display unit 10 and the template information, and stores the storage unit 13. May be stored in In this case, the relevance determination unit 14 is effective in that the latest display information possessed by the second information processing apparatus 2 can be held. The period for acquiring the display information by the relevance determining unit 14 may be determined by experiment in consideration of the intended use and the like assumed for the information processing system S, but is, for example, a 5-second period.

  FIG. 3 is a view schematically showing a data structure of information stored in the storage unit 21 of the second information processing apparatus 2 according to the embodiment. As shown in FIG. 3, the storage unit 21 associates and stores template information, a comment which is display information, and incidental information which indicates the display position of the comment. In the example shown in FIG. 3, the template information is a file of a template image obtained by processing a part of the display screen. In the example illustrated in FIG. 3, the storage unit 21 adds numbers in the order in which the template information is stored, and stores the numbers as an identifier.

  In the example shown in FIG. 3, the template information indicated by the identifier N is a template image of a display screen of a specific stage in a certain game application. When the display screen of the display unit 10 of the first information processing apparatus 1 matches the template information indicated by the identifier N, the execution unit 15 of the first information processing apparatus 1 displays a message "capture from the left!" Which is display information. It is displayed on the display unit 10. Although the details will be described later, the display information stored in the storage unit 21 may be information created by a user other than the user of the first information processing apparatus 1. In this way, the user of the first information processing apparatus 1 can share, during game play, the messages of other users who have already experienced a particular stage in a game.

It returns to the explanation of FIG.
The input reception unit 16 of the first information processing apparatus 1 acquires information input to the first information processing apparatus 1 by the user. When the first information processing apparatus 1 is, for example, a smartphone or a tablet PC, the input receiving unit 16 is a touch panel integrated with the display unit 10. The input reception unit 16 and the display unit 10 constitute a user interface 18 for the first information processing apparatus 1 to present information to the user and to acquire information from the user. For example, a user can input a comment that he / she wants to share with another user via the input reception unit 16.

  The information transmission unit 17 transmits the information accepted by the input acceptance unit 16 from the user to the network N via the communication unit 12. For example, the information transmission unit 17 transmits the information accepted from the user via the input acceptance unit 16. The information transmission unit 17 transmits the image of the display screen of the display unit 10 at the time of receiving the information to the second information processing apparatus 2 together with the information received from the user. The information transmission unit 17 compares the template information stored in the storage unit 13 with the display screen in place of the image on the display screen, whereby the identifier of the template information corresponding to the display screen is transmitted to the second information processing apparatus 2. It may be sent.

  When the control unit 20 of the second information processing apparatus 2 receives the information from the first information processing apparatus 1, the control unit 20 stores the information in the storage unit 21 as information indicating a process to be executed associated with the template information identified by the identifier. Thereby, the predetermined process can be executed when the display screen of the display unit of the second information processing device 2 is associated with the template information. Between the user of the first information processing device 1 and the user of the second information processing device 2, sharing of comments based on the same experience and continuous information update can be realized.

[Relevance judgment]
Next, the determination of the relativity between the display screen executed by the relativity determination unit 14 and the template information will be described.
FIG. 4 is a diagram showing an example of template information. More specifically, FIG. 4 is a template image for determining whether or not the display screen of the display unit 10 of the first information processing apparatus 1 is a screen for creating a transmission mail. In FIG. 4, the shaded area indicated by the symbol t is formed of transparent pixels.

FIG. 5 is a view showing an example of an image of the display screen acquired by the screen acquisition unit 11 (hereinafter referred to as “display screen 1A”). The relevance determining unit 14 reads the display screen 1A illustrated in FIG. 5 from the storage unit 13 and acquires the relevance to the template information illustrated in FIG. 3 according to the procedure described below.
Each pixel constituting the display screen 1A is S [x, y], each pixel of the template information T is T [x, y], each pixel of the composite image obtained by combining the display screen 1A and the template information T is Let C [x, y]. Here, x and y are integers indicating the coordinate position of the pixel. At this time, each pixel of the composite image C is as follows.
C [x, y] = S [x, y] (if T [x, y] = transparent pixel value)
C [x, y] = T [x, y] (if T [x, y] ≠ transparent pixel value)

  If C [x, y] = S [x, y] holds for all pixel values, T [x, y] ≠ T [x, y] for all pixels [x, y] that are transparent pixel values. ] = S [x, y] is satisfied, the relevance determination unit 14 can determine that the display screen 1A is a screen that matches the template information T. When the ratio of unmatched pixels [x, y] among all the pixels is equal to or less than a predetermined value, the relevance determining unit 14 determines that the display screen 1A is associated with the template information T. Here, the “predetermined value” is a reference threshold for determining the relevance between the display screen 1A and the template information T. The specific value of the predetermined value may be determined by an experiment in consideration of the processing to be executed by the first information processing apparatus 1 when the display screen 1A and the template information T are related, but is 90%, for example. . The relevance determining unit 14 sets the ratio of the matching pixel [x, y] among all the pixels as information indicating the relevance between the display screen 1A and the template information T.

  The relevance determination unit 14 is implemented as a resident program that resides in a memory (not shown) of the first information processing device 1 when the first information processing device 1 is powered on. If the user executes the application in the first information processing apparatus 1 while the relevance determination unit 14 is resident, the relevance determination unit 14 continues the determination of the relevance described above. In this way, sharing of information based on the screen of the application can be realized without implementing an additional function for sharing information in the application itself.

Processing flow
FIG. 6 is a flowchart for explaining the flow of information processing performed by the information processing system S according to the embodiment. The processing in this flowchart starts, for example, when the first information processing apparatus 1 and the second information processing apparatus 2 are connected via the network N.

  The relevance determination unit 14 of the first information processing device 1 transmits a request signal for requesting template information to the second information processing device 2 (S1). The control unit 20 of the second information processing device 2 receives a request signal of template information from the first information processing device 1 (S2). When receiving the request signal, the control unit 20 reads out the template information from the storage unit 21 and transmits the template information to the first information processing apparatus 1 (S3). The relevance determination unit 14 of the first information processing device 1 receives the template information from the second information processing device 2 via the network N, and stores the template information in the storage unit 13 (S4).

  The screen acquisition unit 11 acquires an image of a display screen displayed by the display unit 10 (S5). The screen acquisition unit 11 stores the acquired image of the display screen in the storage unit 13 (S6). The relevance determination unit 14 reads the image of the display screen and the template information from the storage unit 13, calculates the ratio of the pixel coincidence described above, and acquires the relevance between the two (S7).

  When the relevance between the image of the display screen acquired by the relevance determining unit 14 and the template information is less than a predetermined value, ie, when the image of the display screen is not associated with the template information (NO in S8), the relevance determining unit 14 , Erases the image on the display screen from the storage unit 13 (S9), and returns to the process of step S5. When the relativity between the image of the display screen acquired by the relevancy judgment unit 14 and the template information is equal to or more than a predetermined value, that is, when the image of the display screen is related to the template information (YES in S8), the relevancy judgment unit 14 Transmits a request signal of information indicating processing to be executed by the first information processing apparatus 1 to the second information processing apparatus 2 via the network N (S10).

  The control unit 20 of the second information processing device 2 receives, from the first information processing device 1, a request signal of information indicating processing to be executed by the first information processing device 1 (S11). The control unit 20 reads the information indicating the process to be executed by the first information processing apparatus 1 from the storage unit 21 and transmits the information to the first information processing apparatus 1 via the network N (S12).

  The relevance determination unit 14 of the first information processing apparatus 1 acquires information indicating processing from the second information processing apparatus 2 via the communication unit 12 and the network N (S13). The execution unit 15 executes the process according to the information indicating the process acquired by the relevance determination unit 14 (S14). When the execution unit 15 executes the process, the process in this flowchart ends. By repeating the above processing, the first information processing apparatus 1 in the information processing system S is triggered by the display screen becoming a specific screen, and a user of another information processing apparatus different from the first information processing apparatus 1 Can execute the specified process.

The operation of the information processing system S having the above configuration is as follows.
The storage unit 21 of the second information processing apparatus 2 stores template information and information indicating processing to be executed by the first information processing apparatus 1. For example, with respect to template information representing one screen of a game, the display unit 10 of the first information processing apparatus 1 performs a predetermined comment on the processing target of the first information processing apparatus 1 as display processing, and Information indicating a certain comment is stored in the storage unit 21 of the second information processing device 2.

  The user of the first information processing apparatus 1 starts a program for executing the process in the flowchart shown in FIG. When the program is activated, the first information processing device 1 establishes communication with the second information processing device 2, and the template information held by the second information processing device 2 is the information processing device 1 of the first information processing device 1. Download to storage unit 13. After that, the first information processing apparatus 1 continues the determination of the relation between the display screen and the template information. When the display screen of the first information processing apparatus 1 is associated with the template information, the first information processing apparatus 1 acquires a comment from the second information processing apparatus 2 and displays the comment superimposed on the display screen. Thereby, the user of the first information processing apparatus 1 can share information when the display screen is associated with the user who created the comment stored in the storage unit 21 of the second information processing apparatus 2 .

  As described above, according to the information processing system S according to the embodiment, the display screen of different information processing apparatuses can be displayed without implementing the additional function in the application itself that displays an image on the screen of the terminal. Can share the processing to be performed when related.

  In particular, the display processing displayed when the display screen of the first information processing apparatus 1 and the display screen of another information processing apparatus are related can be shared among the information processing apparatuses. In addition, the first information processing apparatus 1 acquires the process to be shared from another apparatus other than the first information processing apparatus 1 when the display screen is related to the display screen of another information processing apparatus. Since the first information processing apparatus 1 does not access another apparatus when the display screen is not related, the amount of communication can be suppressed. Furthermore, since the first information processing apparatus 1 acquires and updates the process to be shared with another apparatus when the display screen is related, the latest process can be executed.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It is apparent to those skilled in the art that various changes or modifications can be added to the above embodiment. In particular, the specific embodiment of the distribution and integration of the devices is not limited to the one illustrated above, and all or part of the specific embodiments according to various additions or the like or according to the functional load, in arbitrary units. It can be configured to be distributed or integrated functionally or physically. Hereinafter, modifications of such an embodiment will be described.

[First Modification]
In the above, the case where the relevance determination unit 14 of the first information processing device 1 acquires the relevance between the display screen of the first information processing device 1 and the template information has been described. Instead of this, the second information processing apparatus 2 may acquire the association between the display screen of the first information processing apparatus 1 and the template information.

  FIG. 7 is a diagram schematically showing a functional configuration of an information processing system S according to a first modified example of the embodiment. In the following, parts common to the information processing system S according to the embodiment shown in FIG. 2 will be described as appropriately omitted or simplified.

  In the information processing system S according to the first modification, the first information processing apparatus 1 does not have a relevance determination unit. Instead, in the information processing system S according to the first modification, the second information processing apparatus 2 includes the relevance determination unit 23. In the information processing system S according to the first modification, the screen acquisition unit 11 of the first information processing device 1 transmits the acquired image of the display screen to the second information processing device 2 via the communication unit 12 and the network N. Send.

  The relevance determination unit 23 of the second information processing apparatus 2 obtains the relevance between the template information stored in the storage unit 21 and the image of the display screen transmitted from the screen acquisition unit 11 of the first information processing apparatus 1. . When the template information and the transmitted image of the display screen are associated, the relevance determination unit 23 is information associated with the template information used for determining the relevance, and is executed by the first information processing apparatus 1. Information indicating the processing to be performed is read from the storage unit 21. The relevance determination unit 23 transmits the read information to the first information processing apparatus 1 via the communication unit 22 and the network N.

  Unlike the information processing system S according to the embodiment, the first information processing apparatus 1 according to the first modification includes a processing information acquisition unit 19. The processing information acquisition unit 19 acquires information indicating processing to be executed by the first information processing device 1 from the second information processing device 2. The execution unit 15 executes processing in accordance with the information acquired by the processing information acquisition unit 19.

  Thus, in the information processing system S according to the first modification, the first information processing device 1 transmits the image of the display screen of the display unit 10 to the second information processing device 2 via the network N. . Thereafter, the second information processing apparatus 2 acquires the relevance between the image of the display screen acquired from the first information processing apparatus 1 and the template information. Therefore, the information processing system S according to the first modification is more preferable in the case where the first information processing device 1 is a portable terminal and the second information processing device 2 is a server. In general, servers have more computing resources than mobile terminals. For this reason, when the second information processing apparatus 2 which is a server is in charge of the calculation related to the acquisition of the relevancy, the consumption amount of the calculation resource of the first information processing apparatus 1 can be suppressed.

  On the other hand, when the second information processing apparatus 2 acquires the association between the image of the display screen and the template information, the first information processing apparatus 1 naturally applies the acquired image of the display screen to the second information processing apparatus 2. Need to send. The communication cost may increase depending on the number of pixels of the image on the display screen and the frequency of transmission.

  Therefore, instead of transmitting the image itself of the display screen to the second information processing apparatus 2, the first information processing apparatus 1 may transmit a hash value calculated from the image of the display screen. Here, “a hash value calculated from an image” is information indicating the feature of the image, and refers to information having a much smaller capacity than the image data itself. Therefore, it is possible to suppress the amount of communication of data transmission performed between the first information processing device 1 and the second information processing device 2 for acquiring the relevancy.

  Although calculation of a hash value from an image can be realized by using a known algorithm such as SHA2 (Secure Hash Algorithm 2) or MD5 (Message Digest 5) algorithm, for example, in the following another calculation method of a hash value explain.

  In the information processing system S according to the first modification, the screen acquisition unit 11 of the first information processing device 1 calculates the hash value of the image of the display screen. FIGS. 8A and 8B illustrate a method of calculating the hash value of an image. More specifically, FIG. 8A is a diagram showing an example of image division which is preprocessing for hash value calculation. In FIG. 8A, the display unit 10 is divided into 8 × 13 = 104 divided areas. The divided area can be specified by the coordinates represented by (row number, column number). For example, the coordinates of the upper left divided area are (1, 1), the coordinates of the upper right divided area are (1, 8), the coordinates of the lower left divided area are (13, 1), the coordinates of the lower right divided area 13, 8).

  For example, the screen acquisition unit 11 generates, as a hash value, a quantization value obtained by performing quantization by comparing the average value of the luminance, hue, or saturation of each of the plurality of divided regions with a predetermined threshold. The screen acquisition unit 11 can use, for example, the average value of the luminance, the hue, or the saturation of the pixels included in the screen data of the entire screen displayed on the display unit 10 as the predetermined threshold.

  The screen acquisition unit 11 calculates the average value by averaging the luminance, the hue, or the saturation of all the pixels included in each of the plurality of divided areas. Then, when the calculated average value is equal to or greater than the predetermined threshold value, the screen acquisition unit 11 sets the quantization value in the corresponding divided area to “1”, and the quantum is calculated when the calculated average value is smaller than the predetermined threshold value. Set the conversion value to "0". The screen acquisition unit 11 calculates quantization values corresponding to all the divided areas, and arranges the quantized values of all the divided areas in a predetermined order to generate terminal hash information of the displayed screen data. . The screen acquisition unit 11 may generate a terminal hash value (pHash) by extracting a low frequency component by discrete cosine transform processing and generating a quantization value based on the presence or absence of the low frequency component.

  FIG. 8B shows the quantization value corresponding to the mail transmission screen shown in FIG. The black divided area in FIG. 8B is an area in which the average value of luminance in the divided area is smaller than a predetermined value, and the white divided area is an area in which the average value of luminance in the divided area is equal to or larger than a predetermined value. In the example illustrated in FIG. 8B, the screen acquisition unit 11 includes coordinates (1, 1), coordinates (1, 2), ... coordinates (1, 8), coordinates (2, 1), ... coordinates ( By arranging the quantization values in the order of 13, 8), a terminal hash value of a bit number equal to the number of divided areas, “1111111100000000001111110111111... 1111111101100010001000100” is generated.

FIG. 9 is a diagram schematically showing a data structure of information stored in the storage unit 21 of the second information processing apparatus 2 according to the first modification.
As shown in FIG. 9, the storage unit 21 stores a hash value calculated from an image as template information. In addition, comments that are display information and incidental information indicating the display position of the comments are stored in association with the template information. Since the template information is smaller than the image information, the data volume stored in the storage unit 21 can be suppressed by using the template information as a hash value.

  When the template information is a hash value, the relevance determination unit 23 of the second information processing device 2 displays the Hamming distance between the hash value acquired from the first information processing device 1 and the hash value that is the template information on the display screen It is acquired as an index indicating the relationship between the image of and the template information. In this case, the relevance determining unit 23 outputs the matching degree between the hash value obtained by hashing the display screen and the hash value stored in the storage unit 21 as the relevance between the template information and the display screen. More specifically, the relevance determination unit 23 uses the Hamming distance between the image on the display screen and the template information to calculate the degree of coincidence between the image on the screen and the template information. The relevance determining unit 23 determines that the relevance between the image of the display screen and the template information is larger as the Hamming distance between the image and the template information is shorter. Since the hash value calculated from the image data has a smaller data capacity than the image data itself, the processing for acquiring the relevancy is faster than the acquisition of the relevancy using the composite image described above with reference to FIG. It is also effective in becoming

Second Modified Example
In the above, the case where the information processing system S includes the first information processing apparatus 1 and the second information processing apparatus 2 was mainly described, but the number of information processing apparatuses included in the information processing system S is three or more. Good. This case will be described below.

  FIG. 10 is a diagram schematically showing an outline of an information processing system S according to a second modification. As shown in FIG. 10, an information processing system S according to the second modified example includes three information processing apparatuses, a first information processing apparatus 1a, a second information processing apparatus 2, and a third information processing apparatus 1b. An information processing system S according to a second modification is a so-called client server model in which the first information processing apparatus 1a and the third information processing apparatus 1b are client apparatuses, and the second information processing apparatus 2 is a server. It is a system.

  The functional configurations of the first information processing apparatus 1a and the third information processing apparatus 1b are the same as the functional configuration of the first information processing apparatus 1 according to the embodiment shown in FIG. 2, and the functional configuration of the second information processing apparatus 2 Is the same as the functional configuration of the second information processing apparatus 2 according to the embodiment shown in FIG. Alternatively, the functional configurations of the first information processing apparatus 1a and the third information processing apparatus 1b may be the same as the functional configuration of the first information processing apparatus 1 according to the first modification shown in FIG. 7, in that case The functional configuration of the second information processing apparatus 2 is the same as that of the second information processing apparatus 2 according to the first modification shown in FIG.

  The information transmission unit 17 of the first information processing apparatus 1a receives, from the user, information indicating processing to be executed by the third information processing apparatus 1b via the input reception unit 16, and the second information processing apparatus 2 together with the identifier of the template information. Send to For example, the information transmission unit 17 of the first information processing apparatus 1a performs, as the first information processing, a comment to be displayed on the display unit 10 of the third information processing apparatus 1b as information indicating processing to be executed by the third information processing apparatus 1b. It is transmitted to the second information processing apparatus 2 in association with the display screen of the display unit 10 of the apparatus 1a.

  When receiving the identifier of the information and the template information from the first information processing device 1a, the control unit 20 of the second information processing device 2 stores the information as the information indicating the processing to be executed associated with the template information identified by the identifier. Store in Thereby, the execution unit 15 of the third information processing device 1b can execute predetermined processing in response to the display screen of the display unit 10 of the third information processing apparatus 1b being associated with the template information. Between the user of the first information processing device 1 and the user of the third information processing device 1 b, sharing of comments based on the same experience and continuous information update can be realized.

Third Modified Example
The above description is based on the premise that the aspect ratio of the display unit 10 of the first information processing apparatus 1 is constant. Even in this case, in the first information processing apparatus 1, both “vertical orientation” in which the longitudinal direction of the display unit 10 is used in a longitudinal direction and “horizontal orientation” in which the longitudinal direction is in a horizontal direction are used. There is also one available for orientation. Assuming that the ratio of the number of pixels in the longitudinal direction to the number of pixels in the lateral direction of the display unit 10, that is, the aspect ratio of the display unit 10 is 16: 9, the display unit 10 is used vertically or horizontally. Then, the screen ratio of the display screen is different. In addition, when the information processing apparatus is different, the aspect of the display unit included in the apparatus may be different.

  Therefore, the storage unit 21 of the second information processing device 2 according to the third modification stores template information for each screen ratio of an image that can be displayed by the display unit 10 of the first information processing device 1. The first information processing apparatus 1 further includes a direction detection unit (not shown) that detects the direction of the display unit 10. The direction detection unit can be realized, for example, using a known magnetic sensor and an acceleration sensor. Thereby, since the relevancy can be calculated in consideration of the aspect ratio of the display unit 10 of the first information processing apparatus 1, the accuracy of relevancy calculation can be improved.

Fourth Modified Example
In the above, the display process has been described as a specific example of the information indicating the process to be executed by the first information processing apparatus 1, and in particular, the case of displaying a message has been mainly described. However, the information that the execution unit 15 causes the display unit 10 to display is not limited to character information such as a message. In addition to this, it may be a so-called "stamp image" configured by a symbol such as a character. Thus, communication between users who use the information processing apparatus can be further enriched.

  FIG. 11 is a view showing an example of a stamp image selection screen displayed by the display unit 10 according to the fourth modification. In the first information processing apparatus 1 according to the fourth modification, the “processing to be executed by another information processing apparatus” transmitted by the information transmission unit 17 to the second information processing apparatus 2 is display processing of a stamp image. FIG. 11 is a diagram illustrating an example of a graphical user interface (GUI) for causing the user to select a stamp image to be displayed on another information processing apparatus.

  In FIG. 11, an area indicated by reference numeral 101 is a stamp selection area in which a stamp image selectable by the user is displayed. The example shown in FIG. 11 shows a state in which the user has selected one stamp image to be the selected stamp image 102 from the stamp selection area 101. When the user selects the selected stamp image 102 from the stamp selection area 101, the user can paste the selected stamp image by touching an arbitrary position on the screen of the display unit 10. By tapping the transmission icon 103 in this state, the user can transmit, to the second information processing apparatus 2, information indicating a display process for causing the other information processing apparatus to display the selected stamp image 102. Note that the user can also finish the selection of the stamp image by tapping the back icon 104.

  As shown in FIG. 11, the user of the first information processing apparatus 1 can also select a pictogram or a file other than the stamp image. When the user selects the file selection, the user selects a desired file from the list of images and moving image files (not shown) stored in the storage unit 13 of the first information processing apparatus 1 and the image stored in the file Alternatively, the display unit 10 can display a thumbnail image of a moving image. The user may set the display position of the thumbnail image by touch operation on the display unit 10 or may be able to perform enlargement / reduction by pinch in / out. Thereby, the convenience of the user at the time of file selection can be improved.

[Fifth Modification]
In the above, the case where the incidental information stored in the storage unit 21 of the second information processing apparatus 2 is the display position of the display information has been described. However, the incidental information is not limited to the display position, and may be other information. Specifically, when the display information is a comment, font information indicating the color, size, and type of the font for displaying the comment may be included as incidental information. Further, the additional information may include a displayable level of the comment. The displayable level is a displayability criterion used to determine whether or not the information processing apparatus displays a comment, for example, R specification, exposing a story development in a game, a novel, etc. Information indicating the degree of In this case, the other information processing apparatus can control the presence / absence of comment display in accordance with the displayable level included in the incidental information.

  The incidental information may further include user information (for example, a user identifier, an identifier of a user group, information for identifying whether an administrator or a general user, etc.) for identifying a user who can view a comment. In this case, the other information processing apparatus can determine whether the user of the information processing terminal is a user permitted by the user information included in the supplementary information, and control the presence or absence of the comment display.

  The incidental information may further include time information indicating a display time for displaying a comment on the screen. In this case, the other information processing apparatus displays the comment at the time when the image on the display screen and the template information are associated, and ends the display of the comment when the time indicated by the time information has elapsed. This makes it possible to control the display time of the comment.

Sixth Modification
In the above, the execution unit 15 of the first information processing apparatus 1 has mainly described the execution of the display process for causing the display unit 10 to display some information. However, the process executed by the execution unit 15 is not limited to the display process. For example, in response to the display screen of the display unit 10 of the first information processing apparatus 1 becoming a lock screen, the execution unit 15 performs the first information processing with a GPS (Global Positioning System) (not shown) that the first information processing apparatus 1 comprises. A position information acquisition process of acquiring position information of the device 1 and transmitting it to another device, and a transmission process of transmitting the acquired position information to another device may be executed. As a result, when the first information processing apparatus 1 is lost or the first information processing apparatus 1 is stolen, the user operates the first information processing apparatus 1 as a trigger to operate the position information. It can be acquired. The execution unit 15 may further execute processing such as shutdown of the first information processing apparatus 1, initialization of a terminal, deletion of a storage unit, and the like.

N ... network S ... information processing system 1 ... first information processing apparatus 2 ... second information processing apparatus 10 ... display unit 11 ... screen acquisition unit 12 ... communication unit 13 ... storage unit 14 ... relevance determination unit 15 ... execution unit 16 ... input reception unit 17 ... information transmission unit 18 ... user interface 19 ... processing information acquisition unit 20 ... Control unit 21 Storage unit 22 Communication unit 23 Relevance determination unit

Claims (12)

An information processing system including a first information processing apparatus and a second information processing apparatus capable of communicating with the first information processing apparatus, the information processing system comprising:
The first information processing apparatus is
A display unit for displaying information;
And a screen acquisition unit for acquiring the display screen of the display unit as an image,
The second information processing apparatus is
A storage unit that associates and stores template information used to determine the relevancy of images and information indicating a process to be executed by the first information processing apparatus;
When the image of the display screen acquired by the screen acquisition unit is associated with the template information stored in the storage unit of the second information processing apparatus, the first information processing apparatus acquires the information acquired from the second information processing apparatus And an execution unit that executes the processing indicated by
The storage unit included in the second information processing apparatus stores the template information for each screen ratio of an image displayable by the display unit.
An information processing system characterized by The storage unit stores display information to be processed by display processing executed by the execution unit,
The execution unit displays the display information stored in the storage unit on the display unit when the image of the display screen acquired by the screen acquisition unit is associated with the template information stored in the storage unit of the second information processing apparatus. Let
The information processing system according to claim 1, characterized in that:   The information processing system according to claim 1, wherein the execution unit acquires the information from the storage unit when the display screen is associated with the template information.   4. The apparatus according to claim 1, wherein the storage unit receives and stores information indicating processing executed by the first information processing apparatus from a third information processing apparatus different from the second information processing apparatus. The information processing system according to any one of the above. The information processing system according to claim 4, wherein the third information processing apparatus further includes an information transmitting unit that transmits information indicating a process performed by the first information processing apparatus.   The first information processing apparatus further includes a relevance determination unit configured to obtain a relationship between the template information acquired from the storage unit and the display screen acquired by the screen acquisition unit. The information processing system according to any one of to 5. The screen acquisition unit transmits the acquired display screen to the second information processing apparatus,
The second information processing apparatus further includes a relation determination unit that acquires a relation between the template information stored in the storage unit and the display screen transmitted from the screen acquisition unit. The information processing system according to any one of Items 1 to 5. The storage unit stores a hash value calculated from an image as the template information.
The relevance determining unit is characterized in that the matching degree between the hash value calculated from the image of the display screen and the hash value stored in the storage unit is output as the relevance between the template information and the display screen. The information processing system according to claim 6 or 7. A processor of the first information processing apparatus in an information processing system including a first information processing apparatus and a second information processing apparatus capable of communicating with the first information processing apparatus,
Displaying information to be presented to the user of the first information processing apparatus on a display unit of the first information processing apparatus;
Acquiring the display screen of the display unit as an image;
The image of the display screen is related to the template information stored for each screen ratio of the image that can be displayed by the display unit, the template information being used to determine the relevancy of the image stored in the second information processing apparatus. And performing a process indicated by the information stored in the second information processing apparatus in association with the template information.
An information processing method characterized in that: In the information processing apparatus,
A function of displaying information to be presented to a user of the information processing apparatus on a display unit of the information processing apparatus;
A function of acquiring the display screen of the display unit as an image;
Template information used to determine the relevance of an image from another information processing apparatus different from the information processing apparatus, the template information stored for each screen ratio of an image that can be displayed by the display unit, and the template information A function of acquiring information associated with the information and indicating processing executed by the information processing apparatus;
A function of acquiring the association between the template information and the display screen;
A function of executing a process indicated by information acquired from the other information processing apparatus when the template information and the display screen are associated;
A program that is characterized by realizing An information processing apparatus,
A display unit for displaying information;
A screen acquisition unit that acquires the display screen of the display unit as an image;
A template acquisition unit that acquires template information used to determine the relevancy of images from another information processing apparatus different from the information processing apparatus, the template information being different for each screen ratio of the image that can be displayed by the display unit; ,
An information acquisition unit which acquires information indicating a process to be executed by the information processing apparatus in association with the template information;
An execution unit that executes a process indicated by the information when the image of the display screen acquired by the screen acquisition unit is associated with the template information;
An information processing apparatus comprising: A server capable of communicating with an information processing apparatus,
A process executed by the information processing apparatus, which is template information used to determine the relevancy of the image displayed on the display screen of the information processing apparatus and which is different for each screen ratio of the image that can be displayed on the display screen A storage unit that associates and stores information indicating
A communication unit that transmits the template information to the information processing apparatus;
Equipped with
The communication unit further performs, in response to a request from the information processing apparatus, information indicating processing to be executed by the information processing apparatus when an image displayed on a display screen of the information processing apparatus is associated with the template information. Send to information processing device
Server characterized by

Images