JP7029557B1 - Judgment device, judgment method and judgment program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce labor related to a UI test. A determination device according to the present application includes reference content displayed when a reference terminal executes a predetermined application, and comparison content displayed when a comparison terminal to be compared executes a predetermined application. It is characterized by having an acquisition unit for acquiring the above, and a determination unit for determining whether or not the similarity between the feature of the reference content and the feature of the comparison content satisfies a predetermined condition. [Selection diagram] Fig. 1

Description

The present invention relates to a determination device, a determination method, and a determination program.

Conventionally, various services have been provided to users. In the provision of such a service, content having a preset UI (User Interface) is distributed to the user, and information is provided to the user and operations are accepted from the user.

On the other hand, when providing a service or changing the UI, it is necessary to test whether the UI can perform its function properly. As a method for reducing the time and effort involved in such UI testing, a technique for generating a new test script based on the execution result of a test case and a technique for performing a UI test on a rule basis have been proposed. ..

Japanese Patent No. 6635963 Japanese Patent Publication No. 2013-518321 U.S. Pat. No. 9841956

However, in the above-mentioned conventional technique, there is room for reducing the labor involved in the UI test.

For example, when an application that provides a certain service is executed by a platform such as an OS (Operating System) or a plurality of information processing devices having different screen sizes, screen resolutions, etc., that is, different types of information processing devices, each information is executed. There is a risk that the UI of the content provided via the processing device will be different. For this reason, in the above-mentioned conventional technique, it takes time and effort to prepare test cases and rules for testing the UI for each type of information processing apparatus that executes the application.

It should be noted that such a UI test requires the same time and effort not only when developing an application but also when changing the code of the application due to various updates.

Further, in the above-mentioned conventional technique, it is necessary to prepare in advance test cases and rules indicating what kind of test should be performed for each UI. However, when test cases and rules are prepared manually in this way, mistakes such as lack of content to be tested may occur.

The present application has been made in view of the above, and an object thereof is to reduce the labor related to the UI test.

The determination device according to the present application acquires the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application. It is characterized by having a unit and a determination unit for determining whether or not the similarity between the feature of the reference content and the feature of the comparative content satisfies a predetermined condition.

According to one aspect of the embodiment, it is possible to reduce the labor involved in the UI test.

FIG. 1 is a diagram showing an example of an information providing device according to an embodiment. FIG. 2 is a diagram showing an implementation example of a generation process executed by the information providing device according to the embodiment. FIG. 3 is a diagram showing an implementation example of an application process executed by the information providing device according to the embodiment. FIG. 4 is a diagram showing an example of a flow of determination processing executed by the information providing device according to the embodiment. FIG. 5 is a flowchart showing an example of the flow of the determination process executed by the information providing device according to the embodiment. FIG. 6 is a diagram showing an example of processing for comparing each content. FIG. 7 is a diagram showing a configuration example of the information providing device according to the embodiment. FIG. 8 is a diagram showing an example of information registered in the operation database according to the embodiment. FIG. 9 is a diagram showing an example of information registered in the combination database according to the embodiment. FIG. 10 is a diagram showing an example of information registered in the content database according to the embodiment. FIG. 11 is a flowchart showing an example of the flow of the determination process according to the embodiment. FIG. 12 is a diagram showing an example of a hardware configuration.

Hereinafter, a determination device, a determination method, and an embodiment for implementing the determination program (hereinafter, referred to as “embodiment”) according to the present application will be described in detail with reference to the drawings. It should be noted that this embodiment does not limit the determination device, determination method and determination program according to the present application. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description will be omitted.

[Embodiment]
[1-1. About an example of information providing device]
First, an example of the information providing device 10 which is an example of the determination device according to the embodiment will be described with reference to FIG. FIG. 1 is a diagram showing an example of an information providing device according to an embodiment. Note that FIG. 1 describes, as an example of the functions executed by the information providing device 10, an example of a process of generating operation information for testing an application in order to debug an application installed in various terminal devices. did. As will be described later, the information providing device 10 executes a determination process for determining whether the screen displayed on the terminal device by the application in the test is an appropriate screen in parallel with the process as shown in FIG. ..

In the following description, the flow of the process realized by the operation information generated by the generation process will be described with reference to FIGS. 1 to 3, and then the test will be performed in parallel with the test realized by the operation information. The test for determining whether the screen on which the application is displayed is an appropriate screen, that is, the content of the determination process will be described.

The information providing device 10 shown in FIG. 1 is an information processing device that executes a generation process, and is realized by, for example, a server device, a cloud system, or the like. For example, the information providing device 10 generates operation information for automatically testing the operation of various software such as an OS (Operating System) installed by the terminal devices 201 to 203 and various applications running on the OS. The operation information will be described later.

In addition to the generation process described above, the information providing device 10 has a function of testing the application installed in the terminal devices 201 to 203 according to the generated operation information. It should be noted that such a function does not need to be executed by the information providing device 10 itself, and may be realized by, for example, another information processing device (for example, an applicable device) realized by a server device, a cloud system, or the like. For example, the information providing device 10 automatically tests the operation of various software installed in the terminal devices 201 to 203.

Terminal devices 201 to 203 (hereinafter, may be collectively referred to as "terminal devices 200") are terminal devices used by users U1 to U3, and are, for example, information processing terminals such as smartphones. For example, the terminal device 200 can communicate with the information providing device 10 via a network N such as the Internet. The terminal device 200 may be an information processing device such as a PC (Personal Computer). Further, in the example shown in FIG. 1, three terminal devices 201 to 203 are illustrated, but the embodiment is not limited to this. As will be clarified in the explanation described later, the information providing device 10 may perform tests of an arbitrary number of terminal devices 200 in parallel at the same time. In the example shown in FIG. 1, the terminal device 201 is a smartphone, the terminal device 202 is a tablet, and the terminal device 203 is a PC. The information providing device 10 can also simultaneously test the operation of software executed in such various types of terminal devices 200.

The operator terminal 300 is a terminal device used by a user who desires a test of the terminal device 200 or an operator who has received a request from a user who desires a test, and is a server device, a cloud system, a PC, or the like. It is realized by an information processing device.

[1-2. About an example of the test flow]
Here, before explaining the generation process executed by the information providing apparatus 10, the flow of the test executed by the information providing apparatus 10 will be described. For example, when checking the operation of each terminal device 200 or debugging software, various processes are repeatedly input to each terminal device 200 in various orders, and each operation is performed to check whether an erroneous operation is performed. It will be judged. However, if such a test is performed manually, it takes time and effort. In addition, in the manual test, an erroneous operation may be performed and an appropriate test may not be realized.

Therefore, the information providing device 10 acquires operation information indicating the user's behavior performed on the predetermined target, in other words, the function provided by the predetermined target such as an application, in natural language. For example, when an application provides a login function to a user, the user performs an action such as login by exerting the function. Further, when the application is an application that exerts various payment functions, the user performs an action such as payment by exerting the payment function provided by the application. Then, the information providing device 10 generates application information for applying the operation performed on the target to the target when the action indicated by the operation information is performed from the acquired operation information.

For example, when the information providing device 10 tests whether or not a certain software can be properly logged in, the information providing device 10 acquires an instruction indicating the user's operation at the time of login, such as "log in", in natural language. In such a case, the information providing device 10 specifies an operation for logging in in the software to be operated, and generates application information for realizing the specified operation on the software.

For example, the information providing device 10 performs an operation of "displaying login contents for logging in" and then an operation of "entering a user ID" and "entering a password" in order to perform an action of "login". If you need to perform the "to do" operation and then "tap the login button to log in", specify each of these operations. It should be noted that the identification of various operations necessary for performing such an action is dictionary data generated for each application to be operated, and is performed when the expected action and the action are performed. It is specified by the rule base using the dictionary data associated with the operation.

Then, the information providing device 10 generates application information for applying each specified operation to the terminal device 200. For example, the information providing device 10 tests the terminal device 200 with various publicly known techniques (for example, various UIs (User Interfaces) such as Apium) for reproducing various tap operations and text input operations on the terminal device 200. (Technology of automation framework) is used to generate application information for performing each specified operation on the terminal device 200.

[1-3. About generation processing]
As described above, the information providing device 10 is based on an instruction that is a higher-level conceptualization of the function to be tested, that is, an instruction that indicates the user's behavior performed on the application and the function used by the user in natural language. You can test your application. Therefore, the operator can perform the test only with the information expressing the behavior and function of the user who wants to be the test target in natural language.

However, it cannot be said that an appropriate test can be performed only by such a treatment. For example, in a login test provided by an application, it is necessary to know not only whether or not the login was performed properly, but also what kind of operation was not possible to log in when the login was performed. However, it is troublesome to manually set such detailed operation contents. In addition, when various operation contents are set manually, the operation contents to be tried are not set due to human error, and there is a possibility that a bug may be overlooked.

Therefore, the information providing device 10 executes the following generation processing. First, the information providing device 10 identifies a function exhibited by the user's operation target. Subsequently, the information providing device 10 determines a combination of user operations performed on the operation target according to the specified function. Then, the information providing device 10 generates operation information expressing the determined combination of operations in natural language.

For example, the information providing device 10 analyzes an application to be operated by the user and specifies various functions of the application. For example, the information providing device 10 is an application such as various functions related to payment such as login, reading and transmission of a QR code (registered trademark), and a function of applying various contents such as music contents and map contents as functions provided by the application. Identify the functions that can be used by the user. It should be noted that such a function can be specified by using a model that learns the relationship between the characteristics of the application and the functions provided by the application.

Subsequently, the information providing device 10 determines, for each specified function, a combination of operations performed on the application when the function is exerted. For example, when the specified function is "login", the information providing device 10 performs operations such as "input of user ID", "input of password", and "selection of login button" as operations performed on the application. Identify. That is, the information providing device 10 specifies an operation that can be accepted from the user (in other words, an operation that can be executed by the user) in the screen for exerting the specified function. It should be noted that such a specific process can be realized by using, for example, a table or the like in which the function of the application and the operation for exerting the function are associated in advance.

Then, the information providing device 10 determines a combination of operations that appropriately combines these specified operations. For example, the information providing device 10 may be used for combinations such as "user ID input", "user ID input", and "login button selection", as well as "user ID input", "password input", and "user ID input". Generates a predetermined number of combinations in which specified operations such as "input" are arranged in a random order by a random number. Further, the information providing device 10 determines a combination of operations to be actually applied to the application from the generated combinations by the process described later.

Then, the information providing device 10 generates operation information expressing the combination of the determined operations in natural language, that is, an operation scenario (in other words, a test scenario) as the operation information. For example, when the information providing device 10 determines a combination of operations such as "input of user ID", "input of user ID", and "selection of login button" for the function of "login", "on the login screen," Enter the user ID, then enter the user ID, and then tap the login button "to generate operation information.

[1-4. About an example of generation processing and application processing]
Hereinafter, an example of a flow of generation processing executed by the information providing apparatus 10 and a flow of application processing executed by the information providing apparatus 10 will be described with reference to FIG. 1. For example, the operator installs the application to be operated on each terminal device 200 (step S1). Here, the application installed for each terminal device 200 may be a different version of the application. Further, the application installed on each terminal device 200 may be an application that operates on each individual OS, or may be an application for providing different services. Further, each terminal device 200 may be of different types of hardware. That is, the combination of hardware and software may be different from that of the terminal device 200 to be tested.

Subsequently, the information providing device 10 receives the application to be tested from the operator terminal 300 (step S2). That is, the information providing device 10 accepts an AUT (Application Under Test). For example, the information providing device 10 receives data of an application installed in each terminal device 200. In the example shown in FIG. 1, the information providing device 10 receives the data of the application installed in each terminal device 200 by the operator, but the embodiment is not limited to this. For example, when the information providing device 10 receives an application, the information providing device 10 may transmit and install the accepted application to each terminal device 200. Further, the information providing device 10 may accept an application installed in each terminal device 200 by the operator terminal 300.

Subsequently, the information providing device 10 specifies the service provided by the application as a domain (step S3). As described above, the information providing device 10 determines the combination of operation contents based on the function of the application to be operated, and the function of such an application differs depending on the service provided by the application. For example, it is considered that a login function is indispensable for a payment application, but it is considered that an application such as a photo viewer often does not provide a login function. In this way, it is considered that the functions exhibited by the operation target change depending on the type of service provided by the operation target.

Therefore, the information providing device 10 specifies the service provided by the application to be operated as a domain. For example, the content of the text displayed on the screen by the application can be an accurate indicator of the domain of the service provided by the application. For example, an application that displays text such as "playback" is presumed to be an application that provides a service for playing back various contents such as music. In addition, an application that displays a text such as "payment" is presumed to be an application that provides various payment services. Therefore, the information providing device 10 specifies the domain of the service provided by the operation target based on the text displayed on the screen by the content.

Subsequently, the information providing device 10 specifies a function provided on each screen of the application according to the domain (step S4). For example, the information providing device 10 specifies as a function a configuration service constituting a service provided via an application such as login. In the following description, such a configuration service may be expressed as a "function" of the application.

For example, in the information providing device 10, when two text boxes are included on a certain screen for an application whose domain is "payment service", the function provided on this screen is "login". Is determined. Further, when the information providing device 10 includes two text boxes on a certain screen for an application whose domain is "SNS (Social Networking Service)", the function provided on this screen is ". Judged as "post".

As a result of the above-mentioned processing, the information providing device 10 will specify the function provided through the screen for each screen displayed by the application. Therefore, the information providing device 10 determines a combination of operations for exerting the specified function (step S5). For example, the information providing device 10 identifies an operation associated with a specified function in advance, and generates candidates for a combination of operations in which the specified operations are combined in a random order by a random number. Then, the information providing device 10 identifies a combination candidate that is presumed to be appropriate as a test target from the generated combination candidates.

Subsequently, the information providing device 10 generates operation information indicating the combination of the determined operations in natural language (step S6). For example, the information providing device 10 refers to dictionary data in which each operation is associated with a natural language indicating the operation, and generates operation information indicating the determined operation content in the natural language. For example, in the information providing device 10, when the combination of operations is "enter user ID in text box" or "click button", the information providing device 10 "clicks the button after entering the user ID in the text box". , Generates operation content that shows each operation content and the order in which each operation is executed in natural language. Note that such operation information may be expressed as, for example, a test case, a test scenario, a scenario outline, a feature file, a user flow, or the like. That is, the information providing device 10 generates operation information indicating in natural language the operations that can be input by the user when the configuration service is provided.

[1-5. About an example of application processing]
Subsequently, an example of the provision process executed by the information providing device 10 will be described. First, the information providing device 10 generates application information for causing each terminal device 200 to provide the operation indicated by the operation information (step S7). For example, the information providing device 10 specifies the operation content from the operation information for each OS executed in each terminal device 200 and the software to be logged in. Subsequently, the information providing device 10 generates application information for applying the specified operation to each terminal device 200. For example, the information providing device 10 generates a test code for executing the specified operation for each terminal device 200. In other words, the information providing device 10 generates various scripts for virtually inputting the specified operation to each terminal device 200 as application information.

Then, the information providing device 10 outputs the application information generated for each terminal device 200 to each terminal device 200 (step S8). For example, when the operation information indicates a test case on the screen for logging in, the information providing device 10 distributes the application information for applying the operation in each test case to the terminal device 201 to the terminal device 201. .. Similarly, the information providing device 10 distributes application information for applying the operation in each test case to the terminal devices 202 and 203 to the terminal devices 202 and 203.

In such a case, each terminal device 200 operates the application as if the operation according to the application information received from the information providing device 10 was performed (step S9). Here, the application information output from the information providing device 10 includes information for causing each terminal device 200 to collect the result of applying the operation. As a result, the terminal device 200 collects the application result of the operation and transmits the collected application result to the information providing device 10 (step S10). In such a case, the information providing device 10 generates a report of the application result and provides the generated report to the operator terminal 300 (step S11). For example, the information providing device 10 includes the type of OS and application executed in each terminal device 200, which terminal device 200 the login succeeded or failed, and how many terminal devices 200 the login succeeded or failed. Provide a report such as. As a result of such processing, the information providing device 10 can promote easy understanding of test results such as debugging.

[1-6. Implementation example of generation process]
As described above, the information providing device 10 generates a test case for testing the application based on the analysis result of the application. In other words, the information providing device 10 generates operation information indicating a combination of user operations performed on an operation target in natural language. In such a generation process, the information providing device 10 generates operation information by an improved method of IAGA (Improved Adaptive Genetic Algorithm).

Hereinafter, an implementation example of the generation process executed by the information providing device 10 will be described with reference to FIG. FIG. 2 is a diagram showing an implementation example of a generation process executed by the information providing device according to the embodiment. In FIG. 2, an example of a process flow in which the information providing device 10 generates operation information has been described, but the embodiment is not limited to this. In addition to the processing shown in FIG. 2, the information providing device 10 can generate operation information by any processing if it generates operation information based on the analysis result of the application, for example, various features of the application. You may go.

For example, in the example shown in FIG. 2, the information providing device 10 generates operation information by using the information registered in the operation database and the combination database. In the operation database, all the operation contents (for example, installation) that the user can perform for the application to be operated, and the information indicating each operation content in natural language are registered in association with each other. .. The operation database may be referred to as a user action database.

In the combination database, a combination of user operations applied to the application is registered. Here, for example, not only the combination of operations for appropriately exerting the function provided by the application but also the combination of operations that cannot properly exert the function is registered in the combination database. Will be. That is, in the combination database, information indicating a combination of operations applied at the time of testing the application, in other words, a test scenario is registered. The combination database may be referred to as a population database.

In the following description, the combination of user operations may be collectively referred to as combination information. Further, the combination information may or may not be registered in the combination database in the initial state.

First, the information providing device 10 extracts text from the application. For example, the information providing device 10 analyzes the code of the application to be tested and extracts all the text displayed on the screen by the application. Then, the information providing device 10 identifies the domain of the application based on the extracted text. For example, the information providing device 10 uses a domain model to specify the domain of the function provided by the application from the text displayed by the application.

Subsequently, the information providing device 10 specifies all the screens displayed by the application. For example, the information providing device 10 analyzes the code of the application and identifies all the screens displayed by the application. The information providing device 10 will execute the process described below for each specified screen.

Subsequently, the information providing device 10 identifies a function to be applied via the specified screen. For example, the information providing device 10 specifies the function provided on the screen based on the type of the object arranged on the screen and the content of the operation applicable on the screen for each specified screen. Here, when specifying the function provided on each screen, the information providing device 10 uses the screen model to specify the function provided by the application based on the domain of the application and the characteristics of each screen. conduct.

For example, the information providing device 10 indicates the domain of an application and the characteristics of a certain screen among the screens displayed by the application (for example, the type, number, appearance, various operations that can be performed on the screen, and the like). When inputting information, a model trained to output information indicating the function provided on the screen is retained as a screen model. Then, the information providing device 10 identifies (estimates) the function provided on the screen by inputting information indicating the domain of the application to be tested and the characteristics of the screen displayed by the application into the screen model. do.

Subsequently, the information providing device 10 refers to the operation database and the combination database, and generates / acquires a combination candidate in which the operations for exerting the specified function are combined. For example, in the operation database, a function exhibited by an application and one or a plurality of operations performed by a user in order to exert the function are registered in association with each other. The information providing device 10 refers to the operation database and identifies the operation associated with the specified function. Then, the information providing device 10 generates a predetermined number of combination candidates in which a random number of specified operations are combined in a random order.

For example, in the operation database, three operations such as "input user ID", "enter password", and "click the login button" are associated with the function "login" and registered in the operation database. It shall be. In such a case, the information providing device 10 prescribes the same operation such as "input of user ID", "input of user ID", and "input of user ID" as combination candidates corresponding to "login". Candidates for combinations that repeat as many times as , Generate multiple combinations of candidates with different numbers of operations to perform. The information providing device 10 may generate, for example, a combination candidate that randomly combines up to 10 operations.

Further, the combination information generated by the process described later is registered in the combination database in association with the function. The information providing device 10 refers to the combination database and acquires the combination information associated with the function specified as the function exhibited by the application to be tested as a combination candidate.

Subsequently, the information providing device 10 counts the number of steps until the function is exhibited for each combination candidate. For example, the information providing device 10 executes an operation included in each combination candidate on the screen to be tested, determines whether or not the function is properly exhibited by the time the screen transitions to another screen. , The number of operations performed before the screen transitions to another screen is coefficiented as the number of steps. The coefficient of the number of steps may be referred to as fitness curation.

Subsequently, the information providing device 10 determines whether or not the processing has been looped a predetermined number of times. Then, when the information providing device 10 does not loop the processing a predetermined number of times, the information providing device 10 selects a pair of combination candidates to be a parent from the combination candidates determined to have exerted the function. Then, the information providing device 10 generates a new combination candidate that randomly combines the operations included in the combination candidate selected as the parent. Further, the information providing device 10 generates a new combination candidate in which an operation is randomly added or changed with respect to the newly generated combination candidate. Then, the information providing device 10 re-executes the coefficient of the number of steps until the function is exerted, whether or not the function is properly exerted for each combination candidate.

That is, the information providing device 10 generates a new combination candidate by crossing over the coefficient of the step until the function is exhibited, and the combination candidate by mutating the operation for each combination candidate. Is repeatedly generated. By performing such processing, the information providing device 10 can generate a combination candidate that combines various operations for each screen of the application to be tested.

Further, when the processing is looped a predetermined number of times, the information providing device 10 registers a combination candidate whose number of steps satisfies a predetermined condition in the combination database as combination information. For example, the information providing device 10 specifies a distribution of the number of steps until the function is exhibited when the operation included in each combination candidate is executed, and the number of steps satisfies a predetermined condition based on the specified distribution. Combination candidates are subject to registration. At this time, the information providing device 10 may select a combination candidate having a large number of steps to some extent instead of preferentially selecting a combination candidate having a small number of steps.

Then, the information providing device 10 generates operation information indicating the contents of the operation in natural language from the combination information registered in the combination database. For example, the information providing device 10 identifies each operation included in the combination information, and specifies a natural language indicating the specified operation from the operation database. Then, the information providing device 10 generates operation information indicating a combination of operations indicated by the operation information in the natural language based on the specified natural language.

[1-8. About an example of the flow of application processing]
Subsequently, an implementation example of the application process executed by the information providing device 10 will be described with reference to FIG. FIG. 3 is a diagram showing an implementation example of an application process executed by the information providing device according to the embodiment. Here, the information providing device 10 has a function of generating application information from information indicating the function to be tested in natural language among the functions exhibited by the application, in addition to the operation information provided by the information providing device 10. Have. Therefore, in the example shown in FIG. 3, the flow of the function of generating the application information from the information indicating the function to be tested in natural language is described. The information providing device 10 uses a part of such a function to generate application information from the operation information and execute the test.

For example, the information providing device 10 acquires a natural language as operation information from the operator terminal 300. For example, the information providing device 10 acquires the function of the application in the test such as "the top screen is displayed after logging in" and the information indicating the result when the function is provided.

Here, the "login" is, for example, a service that constitutes a service provided through a terminal device 200 or an application executed by the terminal device 200, such as an SNS, that is, a configuration service that the user can use. It can be said that it is a service provided when a predetermined operation is performed. Such a configuration service includes, for example, not only "login" but also a function for accepting various posted information posted by a user and for taking a QR code in payment using a QR code. Become.

In such a case, the information providing device 10 uses the extraction unit to extract words included in the natural language that indicate an action performed by the terminal device 200 and a result that occurs when the action is performed. And generate feature information that summarizes the extracted words. That is, the information providing device 10 extracts a word indicating a user's action from the operation information for each action.

For example, the information providing device 10 can change from operation information such as "when you log in, the top content is displayed" to a user action such as "login" (that is, an action to be operated) or "top content is displayed". A word indicating the state of the terminal device 200 (that is, the result of the action) when the action is performed is extracted. Then, the information providing device 10 generates feature information indicating the extracted word. For such processing, various techniques related to a test description language format called Gherkin, such as Cucumber and Turnip, can be adopted.

Subsequently, the information providing device 10 uses the driver unit to poll the step information (step function) used for converting the natural language into a command for the device and the target before sending the command, thereby performing a step. Generates assistant information (assistant function) used to support the function.

For example, when the information providing device 10 generates step information, the information providing device 10 specifies from the feature information the content of the operation that occurs in the terminal device 200, the order of the operations, the result when the operation is performed, and the like. In other words, the information providing device 10 executes a process of translating the user's behavior into an operation on the hardware such as the terminal device 200 to be operated and the software executed in the hardware.

For example, when the feature information indicates the action of "login", the information providing device 10 first executes "display of login content", second executes "input of user ID", and third executes "display of login content". Generates step information (step function) indicating a plurality of operations such as "input of password" and "press of login button" to be executed fourth and the order of each operation. That is, the information providing device 10 specifies various operations performed on the terminal device 200 and the order in which each operation is performed when the action indicated by the "login" word is performed, and step information indicating the specified operation or order. To generate.

Further, when the feature information includes the result of an action such as "the top content is displayed", the information providing device 10 generates assistant information (assistant function) for determining whether or not such a result has occurred. do. For example, the information providing device 10 generates assistant information indicating a process such as "determining whether or not the top content is displayed" when the result of an action such as "the top screen is displayed" is included. The information providing device 10 may generate assistant information for determining whether or not a predetermined result generated when various operations indicated by the step information are performed, for example. For example, when the step information includes an operation such as "displaying login content", the information providing device 10 may generate assistant information indicating a process such as "determining whether or not the login content is displayed".

That is, the information providing device 10 generates step information indicating various operations and the order of operations for realizing the actions indicated by the operation information, based on the operation information described in natural language. Further, the information providing device 10 generates assistant information indicating the content of the process performed for determining whether or not the result generated when the operation indicated by each step information is actually performed.

Note that step information and assistant information are generated for each type of OS or application executed by the target terminal device 200, or a combination thereof. Therefore, the driver unit acquires various inputs generated in the user action (behavior) indicated by the feature information generated from the operation information, and identifies the device or application to be the target of the user action and the element to be operated. conduct. Then, the driver unit generates step information based on the identification result.

Similarly, the information providing device 10 generates assistant information for collecting (polling and querying) the target state at regular intervals to ensure that each operation indicated by the step information has been performed. do. More specifically, the information providing device 10 collects the state of the target generated when the operation indicated by the step information is performed for each target, and determines whether or not the collected state satisfies a predetermined condition. Generate assistant information for. In other words, the assistant information is a condition for determining whether the result of performing a certain operation indicated by the step information is appropriate, and if it is determined to be appropriate, performing the next operation indicated by the step information. It can be regarded as information.

Subsequently, the information providing device 10 uses the generation unit to generate application information for performing the operation and various determinations indicated by the step information and the assistant information. In other words, the information providing device 10 generates application information for applying the operation performed on the terminal device 200 when the action indicated by the word extracted by the extraction unit is performed.

Such a generation unit can be realized by various frameworks for performing automatic testing of various terminal devices 200 and applications such as Appium. More specifically, the information providing device 10 applies each operation indicated by the step information to the target in the order indicated by the step information, and causes the information providing device 10 to determine whether the result of each operation is appropriate according to the conditions indicated by the assistant information. Generate a script for.

The script generated in this way is a script for applying the user's action that can be performed when providing the configuration service such as "login", that is, the operation corresponding to the configuration service to the terminal device 200. .. As described above, the information providing device 10 is a configuration service that constitutes a service provided by a predetermined target, and provides behavioral information indicating the configuration service provided when the user performs a predetermined operation in natural language. When acquired, application information for applying a predetermined operation corresponding to the configuration service indicated by the behavior information to the target is generated from the acquired behavior information.

Then, the information providing device 10 outputs the application information to the target terminal device 200. For example, the information providing device 10 is a terminal device for a virtual terminal device ET reproduced in a predetermined information processing device, a terminal device RT such as a smartphone, and various terminal device CTs existing on the cloud. The application information generated every 200 is transmitted. As a result, the information providing device 10 can test the operation of the terminal device 200 when the action indicated by the operation information described in natural language is performed.

In the above-mentioned example, the driver unit may be set for each combination of the extraction unit and the generation unit. For example, the driver unit may generate step information and assistant information in a format that can be identified by the generation unit from the test description language in the format adopted by the implemented extraction unit. Such a driver unit is, for example, dictionary data in which an action indicated by a test description language generated by an extraction unit is associated with an operation content or an operation order for realizing the action, each operation, and each operation. Any program that operates on a rule basis can be adopted by the dictionary data associated with the condition for determining whether the result of the above is appropriate.

For example, when the feature information indicates "login", the driver unit refers to the dictionary data and performs each operation "display login content", "enter user ID", and "enter password" associated with "login". , And "press the login button". Then, the driver unit may generate step information for causing the generation unit to generate application information for causing the terminal device 200 to execute each specified process. Further, the driver unit refers to the dictionary data and specifies the condition associated with each specified operation. Then, the driver unit may generate assistant information for causing the generation unit to generate application information for causing the terminal device 200 to execute whether or not the specified condition is satisfied. As long as it is a program that performs such processing, the information providing device 10 can adopt any program as a driver unit.

When the information providing device 10 receives the operation information, the driver unit is used to generate step information and assistant information indicating each operation indicated by the operation information in natural language. Then, the information providing device 10 generates application information from the generated step information and assistant information by using the generation unit. As a result of such processing, the information providing device 10 can apply various operation contents as test cases to each terminal device 200.

[2. Judgment process]
As described above, the information providing device 10 tests whether the application installed in each terminal device 200 operates properly. On the other hand, among the services realized by the application, there are those that provide common services via various types of terminal devices 200, such as payment using a QR code.

However, even for the same application, the appearance may differ depending on the terminal device 200. For example, even if the terminal device 201 and the terminal device 202 are terminals that execute the same OS, the appearance may change if the screen resolutions are different. When the appearance of the screen provided by the application (hereinafter, may be referred to as "content") changes in this way, it is necessary to give unnecessary anxiety when using the application. Further, for example, when the arrangement of various objects is changed, the user may not be able to perform a correct operation.

Here is a more specific example. For example, enter a text box for an application to enter a user ID for login, a text indicating that the user ID will be entered in the text box of the text (for example, "user ID"), and a password. A text box for this purpose and a text (for example, "password") indicating that a password is to be entered in the text box of the text shall be displayed.

Here, the terminal device 201 displays a text box for inputting a user ID and a text such as "user ID" side by side, and displays a text box for inputting a password and a text such as "password". When displayed side by side, it is considered that the user of the terminal device 201 can correctly input the user ID and the password in the text box. However, for example, if the terminal device 202 does not have a screen resolution for arranging the text and the text box side by side, the arrangement of the text and the text box changes, and the user ID is entered in the text box or the password is entered. There is a risk that it will not be possible to tell the user correctly whether to enter.

Further, depending on the OS and screen settings executed by the terminal device 200 and the characteristics of the hardware, the shapes and colors of the services displayed by the application and the corporate logo, and the colors commonly used in the content provided by the application (so-called). There is a risk that the appearance of the theme color) will be different from what was expected.

The problem of the appearance identity of the content provided by such an application may occur not only at the time of developing the application but also at the time of updating the application, changing the service provided by the application, adding the content, and the like. In other words, the content provided by the application may not be able to guarantee the appearance identity even if the application has the same code due to various reasons such as the platform such as the hardware or software of the terminal device 200 that executes the application. ..

Therefore, the information providing device 10 executes a determination process for determining whether or not the appearance identity of the contents is guaranteed in the above-mentioned application test. That is, when the user browses each content, it is determined whether or not it is determined that the content is for realizing the same configuration service by the same application.

For example, the information providing device 10 uses a predetermined terminal device prepared separately from the terminal device 200 as a reference terminal, and the terminal device 200 as a determination device. Then, the information providing device 10 acquires the reference content displayed when the reference terminal executes the predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application. .. Then, the information providing device 10 determines whether or not the similarity between the feature of the reference content and the feature of the comparative content satisfies a predetermined condition.

For example, when the information providing device 10 determines the identity of the content displayed on each terminal device 200, the information providing device 10 provides not only an application that provides the same service to the reference terminal and the comparison terminal (for example, an application having the same code). , Applications that provide the same service and run on different OSs) are installed. Subsequently, the information providing device 10 acquires the reference content, which is the content displayed on the reference terminal by the application, and the comparison content, which is the content displayed on the comparison terminal by the application.

Then, the information providing device 10 determines the similarity between the reference content and the comparison content, and when the similarity satisfies a predetermined condition, the content displayed as the comparison content is provided to the normal content, that is, the user. When it is done, it is determined that the content can be determined to be the content of the same service as the reference content. As a result of such processing, the information providing device 10 automatically determines whether or not appropriate content is provided, for example, when developing an application that provides a service provided on a multi-platform or when updating an application. This can be done, and the labor of the test can be further reduced.

[2-1. About an example of the flow of judgment processing]
Hereinafter, an example of the flow of the determination process executed by the information providing device 10 will be described with reference to FIG. FIG. 4 is a diagram showing an example of a flow of determination processing executed by the information providing device according to the embodiment. The process shown in FIG. 4 is performed in parallel with the test shown in FIG. 1, and it is assumed that the application to be tested is already installed in each terminal device 200. Further, in the example shown in FIG. 4, an example in which the operator terminal 300 provides an application to the information providing device 10 will be described. For example, the operator terminal 300 is in addition to the application actually installed in the terminal device 200. , For example, you may provide image files that can be used as reference content, such as mockups of full-screen apps created by the design team, icons, components included in design guidelines, and so on.

First, the operator terminal 300 installs the application to be tested on the reference terminal BT (step S20). In such a case, the operator acquires a snapshot of the content displayed by the application on the reference terminal BT on the reference terminal BT. Then, the operator terminal 300 acquires the snapshot as the reference content (step S21).

Subsequently, the operator terminal 300 registers the acquired reference content as a baseline in a predetermined cloud server CS (step S22). Then, the operator terminal 300 provides the application to be tested to the information providing device 10 (step S23). Note that step S23 corresponds to step S2 shown in FIG.

In such a case, the information providing device 10 executes the determination process in parallel with each process (steps S3 to S11) shown in FIG. For example, the information providing device 10 acquires a baseline from the cloud server CS (step S24). Then, the information providing device 10 executes steps S3 to S7 shown in FIG. 1 and outputs application information as shown in FIG. 4 (step S25). Note that step S25 corresponds to step S8 shown in FIG.

Here, the information providing device 10 acquires, as application information, a snapshot of the content displayed on the screen by the application as comparison content, and generates application information for operating the terminal device 200 so as to be included in the application result. Such application information can be realized by generating application information by the above-mentioned application process from a natural language such as taking a screenshot when the screen is displayed. For example, the information providing device 10 acquires operation information for acquiring and outputting a snapshot to the terminal device 200 to be tested in step S6 shown in FIG. 1, and acquires a snapshot in step S7. The application information to be output as the application result may be generated.

Then, the information providing device 10 acquires the comparative content displayed by each terminal device 200 as an application result (step S26). Note that step S26 corresponds to step S10 shown in FIG. In this way, the information providing device 10 acquires the comparison content displayed when the comparison terminal executes the same application as the reference terminal.

Subsequently, the information providing device 10 creates a report prior to step S11 shown in FIG. At this time, the information providing device 10 executes a determination process for determining the similarity between the reference content and the comparison content. For example, the information providing device 10 determines whether or not the appearance and / or functional features of the reference content are similar to the appearance and / or functional features of the comparative content (step S27).

For example, FIG. 5 is a flowchart showing an example of a flow of determination processing executed by the information providing apparatus according to the embodiment. For example, when the determination process is started, the information providing device 10 determines whether or not the baseline corresponding to the comparison content, that is, the reference content is registered (step S101). For example, the information providing device 10 determines whether or not the reference content displayed by the application on the reference terminal is registered at the same timing as the timing when the application displays the comparison content on the comparison terminal. Here, the displayed timing is, for example, when the application is started, when a predetermined operation is performed, or the like, and is different from the actual time. That is, the information providing device 10 makes a comparison target between the reference content and the comparison content displayed when the same configuration service is provided.

Subsequently, when the corresponding baseline is registered (step S101: Yes), the information providing device 10 extracts information on various features and objects from the baseline (step S102). For example, the information providing device 10 extracts the appearance characteristics of the entire screenshot of the reference content that serves as a baseline. It should be noted that such processing can be adopted by, for example, various feature quantity extraction techniques. Further, the information providing device 10 extracts information of each object included in the reference content. For example, the information providing device 10 uses OCR (Optical Character Recognition) technology or the like to extract the content, color, font type, display size, etc. of each text included in the reference content.

Further, the information providing device 10 uses various image recognition techniques to extract information on various objects such as buttons and text boxes included in the reference content. For example, the information providing device 10 extracts what kind of appearance (color, display size, etc.) an object is included, and what kind of function an object is included. The information providing device 10 may specify the function of each object by, for example, analyzing the code included in the application instead of the snapshot.

Further, the information providing device 10 may acquire the arrangement position of each object in the reference content. For example, the information providing device 10 may acquire the position on the screen where the text box is displayed in the reference content. In this way, the information providing device 10 extracts the characteristics of the reference content as a whole, and the appearance characteristics and functional characteristics of various objects included in the reference content.

Subsequently, the information providing device 10 acquires the comparison content (step S103). Then, the information providing device 10 extracts information on various features and objects from the comparison content in the same manner as when extracting external features and functional features of various objects from the reference content (step S104). Then, the information providing device 10 determines the similarity of the features and objects extracted from each content (step S105).

The information providing device 10 determines whether or not the similarity between the characteristics of the appearance of the reference content and the characteristics of the appearance of the comparative content satisfies a predetermined condition. To give a more specific example, the information providing device 10 may calculate the degree of similarity between the feature amount of the reference content and the feature amount of the comparative content.

Further, the information providing device 10 acquires the color, size, and arrangement position of various objects such as texts and buttons arranged in the reference content and the comparison content as external features. Further, the information providing device 10 calculates the degree of matching of the character information indicated by the text, the amount of appearance features, and the degree of similarity of the appearance features of various contents. Further, the information providing device 10 may calculate the degree of similarity depending on whether or not the functions of the various objects are the same.

Then, the information providing device 10 determines whether or not the similarity satisfies a predetermined condition (step S106). For example, the information providing device 10 determines whether or not the calculated similarity exceeds a predetermined threshold value. Then, it is presumed that the information providing device 10 can recognize each content as a content that provides the same configuration service when the similarity satisfies a predetermined condition (step S106: Yes). If possible, register that the comparison content is the reference content (step S107). On the other hand, it can be presumed that the information providing device 10 cannot recognize each content as a content providing the same configuration service when the similarity does not satisfy a predetermined condition (step S106: No). If so, register that the comparison content is the reference content (step S108).

If the corresponding baseline is not registered (step S101), the information providing device 10 performs the baseline registration process (step S109). For example, the information providing device 10 uses the comparison content acquired by one of the terminal devices 200 as a baseline, and realizes a comparison between the comparison content acquired by the other terminal device 200 and the baseline. You may. The information providing device 10 may, for example, prompt the operator to register the baseline.

Here, an example of a process in which the information providing device 10 compares the reference content with the comparison content will be described with reference to FIG. FIG. 6 is a diagram showing an example of processing for comparing each content. In the example shown in FIG. 6, an example of comparing the contents displayed by the application and providing the login configuration service in order to receive the service is shown.

For example, the standard content BC shown in FIG. 6 includes a logo indicating a service, a banner, a text box in which texts such as "mobile phone number" and "password" are arranged, various terms of use, and "new registration by agreeing to the above". Buttons with characters such as "New registration with an external account", buttons for realizing login using each external account, etc. are arranged. Further, although the same object as the reference content BC is arranged in the comparison content CC shown in FIG. 6, for example, among the contents included in the reference content BC, the upper part of the text box for inputting the mobile phone number and the password. It is assumed that the banner placed in is not displayed.

In such a case, the information providing device 10 uses various OCR techniques to extract texts from the reference content BC and the comparison content CC, respectively. Then, the information providing device 10 determines whether or not the appearance and the like of the text extracted from the reference content BC and the text extracted from the comparative content CC match. Further, the information providing device 10 determines the presence / absence and similarity of various contents such as banners and text boxes. For example, in the example shown in FIG. 6, among the various objects included in the reference content BC, the banner arranged at the top of the text box for inputting the mobile phone number and the password is not included in the comparison content CC. Therefore, the information providing device 10 extracts the banner arranged at the upper part of the text box for inputting the mobile phone number and the password as the difference DF, and determines the presence / absence of the banner as the difference DF, the amount of identity of other texts, and the like. Based on this, the similarity of each content is determined.

Returning to FIG. 4, the explanation will be continued. The information providing device 10 provides a report according to the determination result (step S28). Note that step S28 corresponds to step S11 shown in FIG. For example, the information providing device 10 provides a report showing whether or not the appearance characteristics of each content are similar to the reference content as comparative content, whether or not the similarity exceeds a predetermined threshold value, or the similarity itself. do. As a result of such processing, the information providing device 10 determines whether or not the application to be tested provides the content that looks as expected in each terminal device 200, in other words, the user makes each terminal. It is possible to determine whether or not to provide content that can be recognized as content related to the same service when the device 200 is used.

[2-2. About comparison contents]
As described above, the information providing device 10 determines the similarity between the reference content and the comparison content. At this time, the information providing device 10 determines the external features of each content and / or the functional similarity of the objects arranged in each content. Hereinafter, an example of a target for which the information providing device 10 determines similarity will be described.

For example, the information providing device 10 may extract various marks and images arranged in each content as objects and calculate a score based on the similarity of the appearance features of the extracted marks and images. good. For example, the information providing device 10 may calculate the score based on the similarity in appearance such as the color and display size of each mark and image.

Further, the information providing device 10 may extract the characters displayed in each content as an object. Here, the information providing device 10 may extract characters from each content as a screenshot by using the OCR technique, or may specify what kind of characters are displayed from the code of the application. Then, the information providing device 10 may calculate the score based on the similarity of appearance features such as character color, display size, font, etc. of characters in each content. Further, the information providing device 10 may calculate the score based on the identity of the character string. Further, when different languages are selected for the reference content and the comparison content, the information providing device 10 may calculate a score based on the similarity of the intention indicated by the character string.

Further, the information providing device 10 may calculate a score based on the similarity in appearance of functional objects that exert various functions at the time of operation such as buttons and text boxes among the objects arranged in each content. .. For example, in each content, objects that do not function when a user's operation such as text or image is performed (in other words, an object that is static to the operation), buttons, text boxes, etc. are used. It may include objects (in other words, objects that are dynamic to the operation) that perform certain functions such as displaying other content or accepting text input when selected by a person. Therefore, the information providing device 10 identifies an object that exerts various functions when selected by such a user as a functional object, and the presence / absence of the specified functional object, functional similarity, and / or appearance. Determine the similarity of each content based on the similarity. That is, the information providing device 10 determines the similarity of the appearance of the object and the information indicated by the object (for example, meaning that there is no character string or image) for the static object, and for the dynamic object, the information providing device 10 determines the similarity. The similarity of each content is determined based not only on the appearance of the object but also on the similarity of the functions exhibited by the object.

For example, the information providing device 10 may specify functional objects that are presumed to be the same from the appearance, code, or the like from each content, and compare the functional objects with each other on a pixel-by-pixel basis. You may calculate the score according to the similarity of. That is, the information providing device 10 may calculate a score based on the appearance similarity of not only an object such as a text but also a functional object that exerts some function.

The information providing device 10 confirms the existence or nonexistence of the same object, confirms the similarity of the functions of the objects, and then calculates the score according to the stepwise comparison result such as performing the comparison in pixel units. You may. When executing such a process, the information providing device 10 performs a subsequent process such as a pixel-by-pixel comparison when the same object (not only a functional object but also another object may be included) does not exist. You do not have to do. On the other hand, when the same object exists, the information providing device 10 may perform the comparison result in pixel units only for the objects presumed to be the same. That is, the information providing device 10 does not have to make a pixel-by-pixel comparison for the entire screen, and may reduce the processing load, improve the efficiency, and reduce the processing time.

Further, the information providing device 10 may determine whether or not the similarity between the functional object arranged in the reference content and the functional object arranged in the comparison content satisfies a predetermined condition. That is, the information providing device 10 may calculate a score based on the similarity of the features of the functions of each content. For example, the information providing device 10 may calculate a score according to the similarity of the functions exhibited by the functional contents included in each content.

For example, the information providing device 10 may calculate a score according to the similarity between the type of the functional object arranged in the reference content and the type of the functional object arranged in the comparison content. More specifically, the information providing device 10 may calculate a score indicating higher similarity when functional objects exhibiting the same function are included in each content. That is, the information providing device 10 may calculate a higher value score as the functions exhibited by the functional objects in each content are similar.

Further, the information providing device 10 may calculate a score based on the similarity of the positions where the objects are arranged. For example, the information providing device 10 specifies the placement position of each text or functional object (for example, coordinates based on the upper left pixel that is the origin of the content), and calculates a score based on the similarity of the specified placement position. You may.

Further, the information providing device 10 determines the similarity between the relative arrangement position of the plurality of objects arranged in the reference content and the relative arrangement position of the plurality of objects arranged in the comparison content. It may be determined whether or not the condition of is satisfied. For example, the information providing device 10 has a relative positional relationship between the text and the text box in the reference content (for example, whether it is arranged vertically or horizontally), and a relative position between the text boxes. Specify the relative positional relationship between various objects, such as relationships and positional relationships between buttons and other objects. Similarly, the information providing device 10 identifies the relative positional relationship of various objects in the comparison content.

Then, the information providing device 10 calculates a score based on the similarity of the relative positional relationships between the objects in each content. For example, the information providing device 10 calculates a score indicating higher similarity when a predetermined text and a predetermined text box are arranged in the vertical direction in each content. On the other hand, in the information providing device 10, the predetermined text and the predetermined text box are arranged vertically in the reference content, but the predetermined text and the predetermined text box are arranged horizontally in the comparison content. If so, calculate a score that indicates lower similarity. The information providing device 10 may specify a relative positional relationship with respect to a predetermined set of arbitrary objects, and determine the similarity of the specified positional relationship.

Further, the information providing device 10 determines whether or not the similarity between the order in which a plurality of objects are arranged in the reference content and the order in which the plurality of objects are arranged in the comparison content satisfies a predetermined condition. You may judge. For example, the information providing device 10 specifies the order in which each object is arranged in the reference content, and also specifies the order in which each object is arranged in the comparison object. For example, the information providing device 10 identifies in what order the predetermined text, the predetermined text box, and the button are displayed from the top of the content. Then, the information providing device 10 may calculate a higher value score as the similarity of the specified order is higher.

Further, the information providing device 10 may determine the similarity based on the presence or absence of each object. For example, the information providing device 10 has an object common to each content (for example, an object having the same appearance or a similarity exceeding a predetermined threshold value, an object presumed to be displayed by the same code, and the same function. (An object that is presumed to exhibit) is included. Then, the information providing device 10 calculates a score indicating higher similarity when an object common to each content is included, and calculates a score indicating lower similarity when the object is not included. You may.

Further, the information providing device 10 may calculate a score based on the similarity of the features of the attributes of each content. For example, the information providing device 10 may determine whether or not each content includes audio or moving images, and calculate a score according to the determination result.

The information providing device 10 calculates a plurality of scores based on some or all of the above-mentioned calculation criteria for various scores, and the total value of the calculated scores may be used as the final score. good. Further, the information providing device 10 may set different weights for each calculation standard and calculate the total value of the scores. Then, the information providing device 10 determines whether or not the reference content and the comparison content are similar based on whether or not the total value of the scores exceeds a predetermined threshold value, in other words, the similarity of each content is a predetermined condition. Judge whether or not the condition is satisfied.

The information providing device 10 may determine whether or not the similarity of each content satisfies a predetermined condition based on the number of matching points and differences without calculating the score. Further, the information providing device 10 may count the number of criteria satisfying a predetermined condition among various calculation criteria, and determine the similarity depending on whether the counted number exceeds a predetermined threshold value. ..

[2-3. About using the model]
As described above, the information providing device 10 determines the similarity of each content according to the appearance similarity between the contents, the similarity of objects, and the like. At this time, the information providing device 10 may determine the similarity of each content by using a model in which the user has learned whether or not the content is determined to provide the same configuration service.

For example, the information providing device 10 generates changed content in which the appearance and arrangement position of each object included in the reference content are randomly changed. Further, the information providing device 10 provides the changed content and the reference content to the user, and inquires whether each content is the same or similar content, that is, inquires about the label of the content set. The inquiry for such a label may be realized by, for example, cloud sourcing. Then, the information providing device 10 outputs a label such as whether or not the reference content and the modified content are similar to each other when the screenshot or code of the reference content and the modified content is input to the model such as DNN (Deep Neural Netrowk). Train the model.

Then, at the time of the test, the information providing device 10 may input the reference content and the comparison content into the above-mentioned model and realize the determination of whether or not they are similar. As described above, the information providing device 10 may determine the similarity of the appearance and function of each content on a rule basis, or may directly determine using a model.

[2-4. About the range of content to be compared]
Here, the information providing device 10 compares the screenshots as the reference content and the comparison content. However, depending on the screen size of the reference terminal or the comparison terminal, there is a risk that the entire content may not be displayed at once. As a result, when a single content is compared with each other, it may be determined that the similarity is low even though the reference content and the comparison content include the same object.

Therefore, the information providing device 10 may compare the similarity between an arbitrary number of reference contents and the comparison contents. For example, the information providing device 10 acquires a plurality of screenshots when the reference terminal displays the reference content and a plurality of screenshots when the comparison terminal displays the comparison content, and uses the comparison results as the respective screenshots. Based on this, the similarity of each content may be determined. Further, the information providing device 10 acquires a screenshot of the reference terminal and a screenshot of the comparison terminal, which is different from the screenshot of the reference terminal, and each of them is based on the comparison result. Content similarity may be determined.

For example, the information providing device 10 has a plurality of screens so that when the application displays the content on the reference terminal BT, if only a part of the content to be displayed is displayed, the entire content is captured. Take shots and use these multiple screenshots as reference content. Similarly, the information providing device 10 has a plurality of information providing devices 10 so that when the application displays the content on the terminal device 200, if only a part of the content to be displayed is displayed, the entire content is captured. Take screenshots and use these multiple screenshots as comparison content.

Then, the information providing device 10 may determine the similarity between the entire reference content and the entire comparison content by comparing each of the reference content and the comparison content. Here, the information providing device 10 makes a one-to-one comparison, such as comparing one screenshot serving as reference content with one of the screenshots serving as comparison content. It is also possible to make a one-to-many comparison, for example, one screenshot serving as reference content and a plurality of screenshots serving as comparison content. Further, the information providing device 10 may determine the similarity of each screenshot by brute force, and may determine the similarity between the entire reference content and the entire comparison content based on each determination result.

Further, in the information providing device 10, when the reference terminal BT displays the entire content and the terminal device 200 displays only a part of the content, one screenshot as the reference content and a plurality of screenshots as the comparison content. And may be compared to determine the similarity between the entire reference content and the entire comparison content. Further, in the information providing device 10, when the reference terminal BT displays a part of the content and the terminal device 200 displays the entire content, a plurality of screenshots as the reference content and one screenshot as the comparison content. And may be compared to determine the similarity between the entire reference content and the entire comparison content.

[2-5. Applicable target]
In the above-mentioned example, the information providing device 10 determines the similarity of each content displayed on the reference terminal BT and the terminal device 200 by the application providing the same service. Here, the information providing device 10 may determine the similarity of each content displayed by the application composed of the same code, or determine the similarity of each content displayed by the application composed of different codes. You may.

For example, when the reference terminal BT and the comparison terminal execute the same OS, the information providing device 10 installs an application composed of the same code in each, and the content similar to the content displayed on each terminal device by the application. The sex may be determined. On the other hand, when the reference terminal BT and the comparison terminal execute different OSs, the information providing device 10 is an application composed of different codes, and an application for providing the same service on different OSs. Install on the corresponding terminal. Then, the information providing device 10 may determine the similarity of the contents displayed by each terminal.

Further, the information providing device 10 may notify the user of information indicating the relationship between the application executed by the reference terminal and the application executed by the comparison terminal based on the determination result. For example, the information providing device 10 can be recognized by the user that each application provides the same service when the similarity between the content displayed by the reference terminal and the content displayed by the comparison terminal satisfies a predetermined condition. It may be determined that the application is a good one. When performing such processing, the information providing device 10 executes the above-mentioned determination processing for each content displayed by the application, and the ratio of the content whose similarity satisfies the predetermined condition satisfies the predetermined condition. In some cases, it may be determined that the application can be recognized by the user if each application provides the same service.

Further, the information providing device 10 may provide the operator with information about the content for which the similarity is determined not to satisfy a predetermined condition. By providing such information, the operator can easily recognize which content should be modified in the development or update of the application.

Further, the information providing device 10 may determine the robustness of the service provided by a certain application. For example, the information providing device 10 executes the above-mentioned determination process for each content displayed by the application, and when the ratio of the contents having the similarity satisfying the predetermined condition satisfies the predetermined condition, the robustness of the application. May determine that the robustness of the application does not satisfy the predetermined condition when the proportion of the content whose similarity satisfies the predetermined condition does not satisfy the predetermined condition. Further, the information providing device 10 may notify the operator of the determination result of such robustness.

Further, the information providing device 10 may determine an unauthorized application by using the determination result of the similarity of each content. For example, the information providing device 10 prepares a terminal device of the same type as a reference terminal and a comparison terminal, installs a legitimate application in the reference terminal, and installs an application to be determined in the comparison terminal. The application to be determined may be a web application running on a browser.

Then, the information providing device 10 determines the similarity between the reference content displayed on the reference terminal and the comparison content displayed on the comparison terminal, and the application installed on the comparison terminal is invalid according to the determination result. It may be determined whether it is an application. For example, if the information providing device 10 has high similarity of each content but does not completely match, an application installed on the comparison terminal imitates a legitimate application and acquires personal information. It may be determined that the application is a good one.

Here, the information providing device 10 may guide to a phishing site and determine a fake web application that steals a user's login ID or password. For example, the information providing device 10 is not only the content displayed by a formal web application that provides various services to the user via the web (that is, the content provided by the application independently, but also the application is a web API. (Various web contents provided in cooperation with) and various contents displayed by fake web applications are compared. At this time, in the information providing device 10, is the web application to be compared a fake web application based on, for example, the analogy of the appearance characteristics of each content and the similarity of attributes such as the domain of each content? May be determined.

[2-6. About applications that provide multiple services]
Here, an application called a super application or the like may provide not only a single service but also a plurality of services. However, in such an application, instead of providing a service by a single application program, the main application often calls a plurality of applications called mini-apps to provide various services. Therefore, when the operation of the super application is confirmed, the information providing device 10 may compare the reference content and the comparison content for each mini application constituting the super application.

[2-7. About language]
Here, when the reference terminal and the comparison terminal display contents in different languages, the information providing device 10 may regard texts having the same contents as the same object. For example, when the reference content includes a text such as "password" and the comparison content includes a content such as "Password", the information providing device 10 regards these contents as the same content and of each content. Similarity may be determined.

[2-8. About the terminal]
Here, the information providing device 10 uses any terminal as a reference terminal other than the terminal device 200 such as a smartphone, the terminal device CT on the cloud, and the terminal device ET emulated on the predetermined information processing device. Or as a comparison terminal. For example, the information providing device 10 may determine the similarity of each content by using a device (so-called hardware) having an actual condition such as a PC, a server device, various sensors, or digital signage as a reference terminal or a comparison terminal. Further, the information providing device 10 generates reference content and comparative content for each OS and application executed on various virtual machines and PCs, and determines the robustness of the application based on the similarity of each content. You may.

[2-9. About an example of the effect of the embodiment]
As described above, the information providing device 10 determines the similarity between the reference content displayed on the reference terminal and the comparison content displayed on the comparison terminal. Therefore, the information providing device 10 can automatically determine the identity and robustness of the same or different application displaying the reference content and the comparison content, and can facilitate the test of the application and the like.

Even if it is necessary to generate reference content and comparison content for each terminal device and each OS in this way, the information providing device 10 generates operation information indicating the operation content in natural language, and the generated operation. Automatically generate application information based on the information. Therefore, even if the information providing device 10 has a plurality of versions in the application to be tested, or even if there are a plurality of variations of the terminal device to be the reference terminal or the comparison terminal, these reference terminals and comparisons are made. The combination of terminals and applications can be automatically specified, and the similarity of each content can be automatically determined for each specified combination. Therefore, the information providing device 10 can reduce the time and effort of manually operating the application and checking the screen.

[3. Configuration of information providing device]
Hereinafter, an example of the functional configuration of the above-mentioned information providing device 10 will be described. FIG. 7 is a diagram showing a configuration example of the information providing device according to the embodiment. As shown in FIG. 7, the information providing device 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

The communication unit 20 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 20 is connected to the network N by wire or wirelessly, and transmits / receives information to / from the information providing device 10, the terminal device 200, and the operator terminal 300.

The storage unit 30 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. Further, the storage unit 30 stores the operation database 31, the combination database 32, and the content database 33.

In the operation database 31, various functions exhibited by the operation target and user operations performed to exert the functions are registered in association with each other. For example, FIG. 8 is a diagram showing an example of information registered in the operation database according to the embodiment. In the example shown in FIG. 8, the operation database 31 is registered with information having items such as a domain, a corresponding function, an operation content, and a natural language expression. In addition to the information shown in FIG. 8, the operation database 31 may contain information about user operations applicable to the operation target.

For example, the domain is information indicating the type of service provided by the operation target to the user. Further, the corresponding function is a function provided by an operation target belonging to the associated domain, and is information indicating a function provided via a predetermined screen. Further, the operation content is information indicating the content of the operation that the user can perform on the screen that exerts the associated function. The natural language expression is information indicating the associated operation in natural language.

For example, in the example shown in FIG. 8, the domain "payment", the corresponding function "login", the operation content "operation # 1-1", and the natural language expression "click" are registered in association with each other. For such information, the operation target whose service is "payment" exerts the "login" function on a certain screen, and the operation that the user can perform on such a screen is "operation # 1-". 1 ”indicates that the natural language expression of such an operation is a“ click ”. In the example shown in FIG. 8, a plurality of operations such as "operation # 1-1" and "operation # 1-2" are associated with the corresponding function "login". Such information indicates that a plurality of operations such as "operation # 1-1" (for example, click) and "operation # 1-2" (for example, user ID input) can be performed on the screen for performing "login". show.

In the example shown in FIG. 8, a conceptual value such as "operation # 1-1" is described, but in reality, various information (for example, an instance or the like) indicating the operation content is registered. Further, for example, the scroll amount, the number of clicks, the character string constituting the user ID and the password, and the like may be further registered in the operation database 31.

Information indicating a combination of operations, that is, operation information is registered in the combination database 32. For example, FIG. 9 is a diagram showing an example of information registered in the combination database according to the embodiment. In the example shown in FIG. 9, the corresponding domain, the corresponding function, and the operation combination are registered in the combination database 32 in association with each other. In addition to the information shown in FIG. 9, various information related to the combination of operations may be registered in the combination database 32.

Here, the operation combination is information indicating a plurality of operations determined according to the function of the operation target. Further, the corresponding function is information indicating which function the operation indicated by the operation combination is a test case for. Further, the corresponding domain is information indicating to which domain the associated operation combination belongs to the test case of the operation target.

For example, in the example shown in FIG. 9, the corresponding domain "payment", the corresponding function "login", the operation combination "combination # 1", "combination # 2", and the like are registered in association with each other. Such information is an operation indicated by "combination # 1", "combination # 2", etc. in the test case performed on the screen providing the "login" function of the operation target for providing the service related to "payment". Show that. In the example shown in FIG. 9, a conceptual value such as "combination # 1" is described, but in reality, information indicating a plurality of operations and the order in which each operation is performed (for example, an instance) is registered. The Rukoto.

Data as reference content and data as comparison content are registered in the content database 33. For example, FIG. 10 is a diagram showing an example of information registered in the content database according to the embodiment. In the example shown in FIG. 10, information having items such as a content ID, a content type, an acquisition source terminal, a screenshot, and object information is registered in the content database 33. In addition to the information shown in FIG. 10, various information related to the reference content and the comparison content, such as the version of the application and the time stamp such as the date and time when the content was acquired, may be registered in the content database 33.

Here, the content ID is an identifier of the content. The content type is information indicating whether the content is reference content or comparative content. Further, the acquisition source terminal is information indicating a terminal device that has acquired a screenshot of the content, that is, a terminal device that executes an application to be tested. Further, the screenshot is the data of the screenshot acquired as each content. Further, the object information is information about various objects included in the content.

In the example shown in FIG. 10, conceptual values such as "SSBC1" and "OOBC1" are described, but in reality, various image data acquired as screenshots, the content, type, appearance, etc. of the object are described. Various data indicating the above will be registered in the content database 33.

Returning to FIG. 7, the explanation will be continued. The control unit 40 is, for example, a controller, and is an example of various programs (notification programs) stored in a storage device inside the information providing device 10 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. Is realized by executing RAM as a work area. Further, the control unit 40 is a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 7, the control unit 40 includes a specific unit 41, a determination unit 42, a generation unit 43, an application unit 44, an acquisition unit 45, a determination unit 46, and a notification unit 47.

The specific unit 41 specifies a function exhibited by the user's operation target. For example, the specifying unit 41 identifies a function exhibited by the operation target based on a screen displayed on a predetermined terminal device by the operation target application. That is, the specifying unit 41 specifies the function provided via the screen. For example, the specifying unit 41 specifies the type of service provided by the operation target, that is, the domain. Then, the specifying unit 41 specifies the function provided by the operation target on each screen for each screen based on the specified domain and the object on each screen provided by the operation target.

The determination unit 42 determines a combination of user operations performed on the operation target according to the specified function. For example, the determination unit 42 refers to the operation database 31 in which the function is associated with the operation of the user performed when the operation target exerts the function, and the operation corresponding to the function specified by the specific unit 41. The combination is determined based on.

The generation unit 43 generates operation information expressing the determined combination of operations in natural language. For example, the generation unit 43 generates operation information for each screen provided by the operation target. For example, the generation unit 43 acquires from the combination database 32 the operation combination associated with the function provided on the screen on which the operation information is generated and the domain to be operated. Subsequently, the generation unit 43 acquires a natural language expression indicating each operation included in each operation combination from the operation database 31. Then, the generation unit 43 generates operation information indicating the operation content indicated by the operation combination in natural language for each operation combination according to the acquired natural language expression and the order of the operations included in the operation combination. .. As a result of such processing, the information providing device 10 can automatically generate operation information as a test case.

The application unit 44 tests the subject according to the operation information generated by the generation unit 43. For example, when the application unit 44 acquires the operation information, it executes the process illustrated in FIG. 3 to generate application information for each combination of the operation information and the target, and applies the generated application information to the terminal device 200. By doing so, the test is realized. As for the application process executed by the application unit 44, it is assumed that the various processes described above can be applied, and detailed description thereof will be omitted.

When executing the above-mentioned determination process, the application unit 44 uses each terminal device 200 as a comparison terminal, and generates application information for acquiring and transmitting a snapshot to the comparison terminal 200. It should be noted that such processing may be realized, for example, based on the operation information generated by the generation unit 43.

The acquisition unit 45 acquires the reference content displayed when the reference terminal executes the predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application. For example, the acquisition unit 45 causes the reference terminal BT to install the application to be tested, displays all the contents displayed by the application, or displays a part of the predetermined contents, and acquires a screenshot of each content. .. Then, the acquisition unit 45 acquires the screenshot as the reference content from the reference terminal BT.

Further, the acquisition unit 45 acquires a screenshot of the content displayed by the application from each terminal device 200 as the comparison content. That is, the acquisition unit 45 acquires the comparison content displayed when the comparison terminal executes the same predetermined application as the reference terminal. The acquisition unit 45 may acquire a plurality of screenshots when the reference terminal displays the reference content and a plurality of screenshots when the comparison terminal displays the comparison content. That is, the acquisition unit 45 may acquire a screenshot when displaying a part of one content as a reference content or a comparison content. Further, the acquisition unit 45 may acquire different numbers of screenshots of the same content as reference content and comparison content.

Further, the acquisition unit 45 identifies various objects from the screenshots of each content, extracts information about the specified objects as objects, and registers the screenshots and the objects in the content database 33. The acquisition unit 45 may calculate the feature amount of the appearance of each content and register it in the content database 33.

The determination unit 46 determines whether or not the similarity between the feature of the reference content and the feature of the comparative content satisfies a predetermined condition. Here, if the determination unit 46 determines the similarity of each content to the extent that the user can recognize that the content provides the same service, each determination unit 46 is based on an arbitrary criterion. Content similarity may be determined. The determination criteria described below are merely examples, and the embodiments are not limited thereto.

For example, the determination unit 46 determines whether or not the similarity between the characteristics of the appearance of the reference content and the characteristics of the appearance of the comparative content satisfies a predetermined condition. For example, the determination unit 46 determines the similarity in appearance of each content by comparing various feature quantities and pixel units.

Further, the determination unit 46 may determine whether or not the similarity between the arrangement position of the object arranged in the reference content and the arrangement position of the object arranged in the comparison content satisfies a predetermined condition. .. For example, the determination unit 46 may specify the position where the text, the image, various functional objects, and the like are displayed on the screen, and determine the similarity of each content based on the similarity of the specified position.

Further, the determination unit 46 has a predetermined similarity between the relative arrangement positions of the plurality of objects arranged in the reference content and the relative arrangement positions of the plurality of objects arranged in the comparison content. It may be determined whether or not the condition is satisfied. For example, the determination unit 46 has a relative positional relationship (for example, vertically arranged or horizontally arranged) between a predetermined text such as “password” and a predetermined functional object such as a text box for inputting a password. You may specify each content (such as password, distant direction, distant distance, etc.) and determine the similarity of each content according to whether or not the specified positional relationship is similar.

Further, the determination unit 46 determines whether or not the similarity between the order in which the plurality of objects are arranged in the reference content and the order in which the plurality of objects are arranged in the comparison content satisfies a predetermined condition. You may. For example, the determination unit 46 determines the similarity of each content according to whether or not the predetermined objects are arranged in a predetermined order in each content, the similarity of the order in which the objects are arranged, and the like. May be good.

Further, the determination unit 46 may determine whether or not the similarity between the appearance characteristics of the characters included in the reference content and the appearance characteristics of the characters included in the comparison content satisfies a predetermined condition. .. For example, the determination unit 46 may determine the similarity of each content based on the similarity of the text content, color, font size, itself, etc. in each content.

Further, the determination unit 46 has similarities between the features of the reference content and the features of the comparison content, based on whether or not an object similar to the object placed in the reference content is placed in the comparison content. May determine whether or not satisfies a predetermined condition. For example, the determination unit 46 determines whether or not the same or similar objects (for example, buttons having slightly different display sizes but the same text are arranged) are arranged in each content, and the same or similar objects are arranged. The similarity of each content may be determined according to the number of arrangements.

Further, among the objects arranged in the reference content, the determination unit 46 has a predetermined similarity between the functional object that exerts a predetermined function by the operation from the user and the functional object arranged in the comparison content. It may be determined whether or not the condition is satisfied. Further, the determination unit 46 determines whether or not the similarity between the function exhibited by the functional object arranged in the reference content and the function exhibited by the functional object arranged in the comparison content satisfies a predetermined condition. You may. For example, the determination unit 46 may determine that the contents are similar when the function objects exhibiting the common functions are arranged in common for each content.

Further, the determination unit 46 may determine whether or not the similarity between the appearance of the functional object arranged in the reference content and the appearance of the functional object arranged in the comparison content satisfies a predetermined condition. .. For example, the determination unit 46 determines the similarity of the appearance of the functional objects included in each content for each pixel or by comparing the features, and determines the similarity of the appearances of the functional objects arranged in each content according to the determination result. , You may judge the similarity of each content.

Further, the determination unit 46 may determine whether or not the similarity between the attribute characteristics of the reference content and the attribute characteristics of the comparison content satisfies a predetermined condition. Further, the determination unit 46 may determine whether or not the similarity between the feature of the function of the reference content and the feature of the function of the comparison content satisfies a predetermined condition.

The determination unit 46 calculates a score indicating similarity based on the various criteria described above. However, the embodiments are not limited to this. The determination unit 46 is information set by the user when the reference feature information indicating the characteristics of the reference content and the comparison feature information indicating the features of the comparison content are input, and is the information set by the user, and the reference content and the comparison content. Whether or not the similarity between the characteristics of the reference content and the characteristics of the comparative content satisfies a predetermined condition using a model trained to output information indicating whether or not they are similar to each other. May be determined. That is, the determination unit 46 may determine the commonality between the reference content and the comparison content by using a model that learns the characteristics of the content that the user has determined to be the content that provides the common configuration service.

The above-mentioned process is executed with reference to the information registered in the content database 33. For example, the determination unit 46 refers to the content database 33 and acquires screenshots and object information of the reference content and the comparison content with respect to the reference content. Then, the determination unit 46 determines the similarity between the reference content and the comparison content based on the acquired information.

The notification unit 47 notifies the user of information indicating the relationship between the application executed by the reference terminal and the application executed by the comparison terminal according to the determination result. For example, when the ratio of the content determined to be similar to the reference content corresponding to each comparison content among the comparison contents displayed on the comparison terminal by a certain application exceeds a predetermined threshold, the notification unit 47 is the reference terminal. May notify that the identity between the application executed by the user and the application executed by the comparison terminal is guaranteed. Further, the notification unit 47 may notify information about each content determined to be dissimilar as a result of comparison.

[4. Processing procedure]
Next, the procedure of the determination process by the information providing apparatus 10 according to the embodiment will be described with reference to FIG. FIG. 11 is a flowchart showing an example of the flow of the determination process according to the embodiment.

For example, the information providing device 10 determines whether or not the application to be determined has been accepted (step S201). Then, if the information providing device 10 has not accepted the application (step S201: No), the information providing device 10 waits until the application is accepted (step S201). On the other hand, when the information providing device 10 receives the application (step S201: Yes), the information providing device 10 causes the reference terminal to execute the application and acquires the reference content (step S202). If the reference content is registered in a predetermined database, the information providing device 10 may acquire the reference content.

Further, the information providing device 10 causes the comparison terminal to execute the application and acquires the comparison content (step S203). The information providing device 10 may execute steps S202 and S203 in any order.

Then, the information providing device 10 determines whether or not the similarity between the reference content and the comparison content satisfies a predetermined condition (step S204), provides information according to the determination result (step S205), and processes. To finish.

[5. Modification example]
The information providing device 10 described above may be implemented in various different forms other than the above embodiment. Therefore, another embodiment of the information providing device 10 will be described below.

[5-1. About operation]
In the above example, the information providing device 10 compares the contents displayed by the application. Here, the information providing device 10 may compare the contents displayed as a result of the operation automatically applied in the test by the generation process or the application process generated by the application process. For example, the information providing device 10 acquires the content displayed when the user taps a certain button, pays for the QR code, or performs an operation to open the menu as reference content or comparison content. , The similarity of the acquired reference content or comparison content may be determined.

In this way, when the display mode of the reference content or the comparison content changes due to various operations, the information providing device 10 acquires the content before the change and the content after the change as individual reference content or comparison content, respectively. Similarity may be determined. It should be noted that such processing can be realized, for example, by acquiring a screenshot each time an operation is performed and generating application information for acquiring the acquired screenshot as reference content or comparison content. Further, when performing such processing, the information providing device 10 makes a comparison target, for example, depending on what kind of operation the reference content or the comparison content is displayed. You can select reference content and comparison content. That is, the information providing device 10 may determine the similarity of the contents displayed by the application when the same operation is performed on the reference terminal or the comparison terminal.

The information providing device 10 is operated every time an operation is performed, for example, when an arbitrary action such as posting a content such as a predetermined character string or a photograph to an SNS or the like, inputting to a game or the like, or viewing a video is performed. Screenshots may be acquired as reference content or comparison content. Further, when the information providing device 10 performs various hardware-dependent actions such as inputting various operations by voice, such as changing the direction of the terminal device 200, shaking the terminal device 200, and changing the volume. Each time the operation of is performed, a screenshot may be acquired as reference content or comparison content.

Further, the information providing device 10 may execute the various processes described above for any application. For example, the information providing device 10 may generate operation information and compare reference contents and comparison contents with respect to an application that provides various payment services using the terminal device 200. For example, in the payment service, the user reads the QR code (registered trademark) installed in the store, accepts the input of the payment amount, and uses the electronic money of the received payment amount for the account of the store corresponding to the QR code. There are services such as sending money from your account. In addition, some payment services display a barcode that identifies a user's account or credit card, and the terminal of the store reads the displayed barcode to charge the user.

The information providing device 10 may generate operation information and compare reference contents and comparison contents for applications related to such various payment services. In this case, the information providing device 10 compares the reference content and the comparison content, for example, when a predetermined remittance amount is input, when the QR code is read, when the bar code indicating the user's account is displayed, and the like. By doing so, it may be determined whether or not an appropriate amount or information is displayed.

[5-2. About the purpose]
Here, the information providing device 10 may perform the above-mentioned determination process for any purpose other than the test of the target application. For example, RPA (Robotic Process Automation) technology is known in which software-type robots and AI (Artificial Intelligence) act on behalf of various alliance work and automate it. The information providing device 10 may apply the above-mentioned determination process in such RPA technology. For example, the information providing device 10 uses the operation result of the robot serving as the reference terminal (for example, a moving image obtained by photographing the appearance during operation) as the reference content, and the operation result of the robot serving as the comparison terminal as the comparison content. Then, the information providing device 10 may determine whether the robot serving as the comparison terminal operates appropriately based on the identity of each content.

Further, the information providing device 10 may execute a comparison process of various software in order to test QAAS (Quality Assurance as a Service). For example, the information providing device 10 may determine that the quality of the application executed on the comparison terminal satisfies the predetermined condition when the similarity between the reference content and the comparison content exceeds a predetermined threshold value.

In this way, the information providing device 10 uses not only the test for determining the similarity of the displayed images but also the determination result as to whether the application operates properly or the quality of the application exceeds a predetermined threshold value. You may. Further, the information providing device 10 may generate application information for any purpose other than the above-mentioned purpose as long as it generates operation information indicating an operation for the target.

[5-3. About processing mode]
Of the processes described in the above embodiments, all or part of the processes described as being automatically performed can be performed manually, and conversely, all the processes described as being performed manually. Alternatively, a part thereof can be automatically performed by a known method. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown in the figure.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in any unit according to various loads and usage conditions. Can be integrated and configured.

In addition, each of the above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

[5-4. program〕
Further, the information providing device 10 according to the above-described embodiment is realized by, for example, a computer 1000 having a configuration as shown in FIG. FIG. 12 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to the output device 1010 and the input device 1020, and the arithmetic unit 1030, the primary storage device 1040, the secondary storage device 1050, the output IF (Interface) 1060, the input IF 1070, and the network IF 1080 are connected by the bus 1090. Have.

The arithmetic unit 1030 operates based on a program stored in the primary storage device 1040 or the secondary storage device 1050, a program read from the input device 1020, or the like, and executes various processes. The primary storage device 1040 is a memory device that temporarily stores data used by the arithmetic unit 1030 for various operations such as RAM. Further, the secondary storage device 1050 is a storage device in which data used by the calculation device 1030 for various calculations and various databases are registered, such as a ROM (Read Only Memory), an HDD (Hard Disk Drive), and a flash memory. Is realized by.

The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various information such as a monitor and a printer. For example, USB (Universal Serial Bus), DVI (Digital Visual Interface), and the like. It is realized by a connector of a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Further, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, a keyboard, a scanner, and the like, and is realized by, for example, USB.

The input device 1020 is, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), or a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Further, the input device 1020 may be an external storage medium such as a USB memory.

The network IF 1080 receives data from another device via the network N and sends it to the arithmetic unit 1030, and also transmits the data generated by the arithmetic unit 1030 to the other device via the network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 10, the arithmetic unit 1030 of the computer 1000 realizes the function of the control unit 40 by executing the program loaded on the primary storage device 1040.

[6. effect〕
As described above, the information providing device 10 includes the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application. To get. Then, the information providing device 10 determines whether or not the similarity between the feature of the reference content and the feature of the comparative content satisfies a predetermined condition.

For example, the information providing device 10 acquires the comparison content displayed when the comparison terminal executes the same predetermined application as the reference terminal. As a result of such processing, the information providing device 10 can determine the similarity of the contents displayed on different terminal devices by the same application, so that appropriate development is performed at the time of application development or update. It can be automatically determined whether or not it is. Further, when the comparison terminal and the reference terminal execute different applications, the information providing device 10 automatically determines whether the application executed on the comparison terminal can be identified with the application executed on the reference terminal. be able to. As a result of automating such a determination, the information providing device 10 can automate the test of the UI provided by various applications and reduce the time and effort.

Further, the information providing device 10 determines whether or not the similarity between the characteristics of the appearance of the reference content and the characteristics of the appearance of the comparative content satisfies a predetermined condition. Further, the information providing device 10 determines whether or not the similarity between the arrangement position of the object arranged in the reference content and the arrangement position of the object arranged in the comparison content satisfies a predetermined condition. , The information providing device 10 can automatically determine whether the appearances are similar to the extent that the user can determine that the contents provide the same service.

Further, the information providing device 10 determines the similarity between the relative arrangement position of the plurality of objects arranged in the reference content and the relative arrangement position of the plurality of objects arranged in the comparison content. Judge whether or not the condition of is satisfied. Further, the information providing device 10 determines whether or not the similarity between the order in which a plurality of objects are arranged in the reference content and the order in which the plurality of objects are arranged in the comparison content satisfies a predetermined condition. judge. Therefore, the information providing device 10 can determine the similarity of each content according to the similarity given to the user by the relative positional relationship and the order of the objects.

Further, the information providing device 10 determines whether or not the similarity between the appearance characteristics of the characters included in the reference content and the appearance characteristics of the characters included in the comparison content satisfies a predetermined condition. Therefore, the information providing device 10 can determine the similarity of each content according to the similarity given to the user by the appearance of the characters such as the font.

Further, the information providing device 10 is similar to the feature of the reference content and the feature of the comparison content based on whether or not an object similar to the object arranged in the reference content is arranged in the comparison content. It is determined whether or not the sex satisfies a predetermined condition. Further, among the objects arranged in the reference content, the information providing device 10 determines the similarity between the functional object that exerts a predetermined function by the operation from the user and the functional object arranged in the comparison content. Judge whether or not the condition of is satisfied. Therefore, the information providing device 10 can determine the similarity of each content according to the presence / absence and similarity of various contents.

Further, the information providing device 10 determines whether or not the similarity between the function exhibited by the functional object arranged in the reference content and the function exhibited by the functional object arranged in the comparison content satisfies a predetermined condition. judge. Therefore, the information providing device 10 can determine the similarity of each content based on the similarity of the functions provided by each content.

Further, the information providing device 10 determines whether or not the similarity between the appearance of the functional object arranged in the reference content and the appearance of the functional object arranged in the comparison content satisfies a predetermined condition. Therefore, the information providing device 10 can determine the similarity of each content according to the similarity that the appearance gives to the user.

Further, the information providing device 10 determines whether or not the similarity between the attribute characteristics of the reference content and the attribute characteristics of the comparison content satisfies a predetermined condition. Further, the information providing device 10 determines whether or not the similarity between the functional feature of the reference content and the functional feature of the comparative content satisfies a predetermined condition. Therefore, the information providing device 10 can appropriately determine the similarity of each content and determine the similarity of each content.

Further, the information providing device 10 is information set by the user when the reference feature information indicating the feature of the reference content and the comparative feature information indicating the feature of the comparison content are input, and the reference thereof. Using a model trained to output information indicating whether the content and the comparison content are similar, the similarity between the characteristics of the reference content and the characteristics of the comparison content is a predetermined condition. Determine if it meets or not. Therefore, the information providing device 10 can automatically determine whether or not the user determines that each content is content that provides the same function.

Further, the information providing device 10 acquires a screenshot of the reference content displayed by the reference terminal and a screenshot of the comparison content displayed by the comparison terminal. Further, the information providing device 10 acquires a plurality of screenshots when the reference terminal displays the reference content and a plurality of screenshots when the comparison terminal displays the comparison content. Further, the information providing device 10 acquires a screenshot of the reference terminal and a screenshot of the comparison terminal, which is different from the screenshot of the reference terminal. Therefore, the information providing device 10 automatically automatically identifies the identity of the content appropriately even when the entire single content is displayed on the reference terminal or the comparison terminal, or when only a part of the content is displayed. Can be determined.

Further, the information providing device 10 notifies the user of information indicating the relationship between the application executed by the reference terminal and the application executed by the comparison terminal according to the determination result. Therefore, the information providing device 10 can provide various information such as whether or not the application needs to be modified and whether or not the application is a legitimate application.

The embodiments of the present application have been described in detail with reference to the drawings, but this is an example, and various modifications and improvements are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to carry out the present invention in other forms described above.

Further, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the acquisition unit can be read as an acquisition means or an acquisition circuit.

10 Information provider 20 Communication unit 30 Storage unit 31 Operation database 32 Combination database 33 Content database 40 Control unit 41 Specific unit 42 Decision unit 43 Generation unit 44 Acquisition unit 46 Judgment unit 47 Notification unit 200 Terminal device 300 Operator terminal BT reference Terminal CS cloud server

Claims (27)

An acquisition unit that acquires the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application.
Whether or not the similarity between the order in which the plurality of objects arranged in the reference content are arranged and the order in which the plurality of objects arranged in the comparison content are arranged satisfies a predetermined condition. A determination device characterized by having a determination unit for determination. An acquisition unit that acquires the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application.
Among the objects arranged in the reference content, the function exhibited by the functional object that exerts a predetermined function by the operation from the user and the function exhibited by the functional object arranged in the comparison content are similar to each other. A determination device comprising a determination unit for determining whether or not a predetermined condition is satisfied. The determination unit is characterized in that it determines whether or not the similarity between the appearance of the functional object arranged in the reference content and the appearance of the functional object arranged in the comparison content satisfies a predetermined condition. The determination device according to claim 2 . An acquisition unit that acquires the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application.
Information set by the user when the standard feature information indicating the characteristics of the reference content and the comparative feature information indicating the characteristics of the comparison content are input, and the reference content and the comparison content are similar. Using a model trained to output information indicating whether or not, it is determined whether or not the similarity between the feature of the reference content and the feature of the comparative content satisfies a predetermined condition. A determination device characterized by having a determination unit and a determination unit. An acquisition unit that acquires the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application.
A determination unit for determining whether or not the similarity between the character string arranged in the reference content and the character string arranged in the comparison content satisfies a predetermined condition.
Have,
When the determination unit indicates that the first character string of the first language arranged in the reference content and the second character string of the second language arranged in the comparison content indicate the same content. , Judge that the first character string and the second character string are the same or similar
A judgment device characterized by the fact that. An application unit that applies an operation performed to exert a predetermined function to a predetermined application executed by the reference terminal and the predetermined application executed by the comparison terminal.
Have more
The acquisition unit acquires the reference content displayed by the reference terminal when the operation is applied and the comparison content displayed by the comparison terminal when the operation is applied .
The determination device according to any one of claims 1 to 5, wherein the determination device is characterized by the above. The determination unit is characterized in that it determines whether or not the similarity between the feature of the function of the reference content and the feature of the function of the comparison content satisfies a predetermined condition . The determination device according to any one of them . The acquisition unit according to any one of claims 1 to 7 , wherein the acquisition unit acquires the comparison content displayed when the comparison terminal executes the same predetermined application as the reference terminal. Judgment device. The determination unit is characterized in that it determines whether or not the similarity between the appearance feature of the reference content and the appearance feature of the comparison content satisfies a predetermined condition. The determination device according to any one of them. The determination unit is characterized in that it determines whether or not the similarity between the arrangement position of the object arranged in the reference content and the arrangement position of the object arranged in the comparison content satisfies a predetermined condition. The determination device according to claim 9 . The determination unit has a predetermined similarity between the relative placement positions of the plurality of objects placed in the reference content and the relative placement positions of the plurality of objects placed in the comparison content. The determination device according to claim 10 , wherein the determination device determines whether or not the conditions are satisfied. The determination unit determines whether or not the similarity between the order in which the plurality of objects are arranged in the reference content and the order in which the plurality of objects are arranged in the comparison content satisfies a predetermined condition. The determination device according to claim 11, further comprising determining. The determination unit is characterized in that it determines whether or not the similarity between the appearance characteristics of the characters included in the reference content and the appearance characteristics of the characters included in the comparison content satisfies a predetermined condition. The determination device according to any one of claims 9 to 12. The determination unit has a feature of the reference content and a feature of the comparison content based on whether or not an object similar to the object placed in the reference content is placed in the comparison content. The determination device according to any one of claims 1 to 13, wherein it is determined whether or not the similarity satisfies a predetermined condition. Claims 1 to 14 are characterized in that the determination unit determines whether or not the similarity between the attribute characteristics of the reference content and the attribute characteristics of the comparison content satisfies a predetermined condition. The determination device according to any one of them. One of claims 1 to 15 , wherein the acquisition unit acquires a screenshot of the reference content displayed by the reference terminal and a screenshot of the comparison content displayed by the comparison terminal. The determination device according to one. The acquisition unit is characterized in that it acquires a plurality of screenshots when the reference terminal displays the reference content and a plurality of screenshots when the comparison terminal displays the comparison content. Item 16. The determination device according to item 16 . 16 is characterized in that the acquisition unit acquires a screenshot of the reference terminal and a screenshot of the comparison terminal, which is different from the screenshot of the reference terminal. The determination device described. Claim 1 is characterized by having a notification unit for notifying a user of information indicating a relationship between an application executed by the reference terminal and an application executed by the comparison terminal according to a determination result by the determination unit. The determination device according to any one of 18 to 18. It is a judgment method executed by the judgment device.
An acquisition process for acquiring the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application .
It includes a determination step of determining whether or not the similarity between the character string arranged in the reference content and the character string arranged in the comparison content satisfies a predetermined condition.
In the determination step, when the first character string of the first language arranged in the reference content and the second character string of the second language arranged in the comparison content indicate the same content, the determination step is performed. , Judge that the first character string and the second character string are the same or similar
Judgment method characterized by that. An acquisition procedure for acquiring the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes a predetermined application, and an acquisition procedure.
It is a determination program for causing a computer to execute a determination procedure for determining whether or not the similarity between the character string arranged in the reference content and the character string arranged in the comparison content satisfies a predetermined condition. hand,
In the determination procedure, when the first character string of the first language arranged in the reference content and the second character string of the second language arranged in the comparison content indicate the same content, the determination procedure is performed. , Judge that the first character string and the second character string are the same or similar
A judgment program characterized by that . It is a judgment method executed by the judgment device.
An acquisition process for acquiring the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application.
Whether or not the similarity between the order in which the plurality of objects arranged in the reference content are arranged and the order in which the plurality of objects arranged in the comparison content are arranged satisfies a predetermined condition. A determination method including a determination determination step. An acquisition procedure for acquiring the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes a predetermined application, and an acquisition procedure.
Whether or not the similarity between the order in which the plurality of objects arranged in the reference content are arranged and the order in which the plurality of objects arranged in the comparison content are arranged satisfies a predetermined condition. Judgment A judgment program for making a computer execute the judgment procedure. It is a judgment method executed by the judgment device.
An acquisition process for acquiring the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application.
Among the objects arranged in the reference content, the function exhibited by the functional object that exerts a predetermined function by the operation from the user and the function exhibited by the functional object arranged in the comparison content are similar to each other. A determination method including a determination step of determining whether or not a predetermined condition is satisfied. An acquisition procedure for acquiring the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes a predetermined application, and an acquisition procedure.
Among the objects arranged in the reference content, the function exhibited by the functional object that exerts a predetermined function by the operation from the user and the function exhibited by the functional object arranged in the comparison content are similar to each other. A judgment program for causing a computer to execute a judgment procedure for judging whether or not a predetermined condition is satisfied. It is a judgment method executed by the judgment device.
An acquisition process for acquiring the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application.
Information set by the user when the standard feature information indicating the characteristics of the reference content and the comparative feature information indicating the characteristics of the comparison content are input, and the reference content and the comparison content are similar. Using a model trained to output information indicating whether or not, it is determined whether or not the similarity between the feature of the reference content and the feature of the comparative content satisfies a predetermined condition. Judgment method characterized by including a determination step to be performed. An acquisition procedure for acquiring the reference content displayed when the reference terminal executes a predetermined application and the comparison content displayed when the comparison terminal to be compared executes a predetermined application, and an acquisition procedure.
Information set by the user when the standard feature information indicating the characteristics of the reference content and the comparative feature information indicating the characteristics of the comparison content are input, and the reference content and the comparison content are similar. Using a model trained to output information indicating whether or not, it is determined whether or not the similarity between the feature of the reference content and the feature of the comparative content satisfies a predetermined condition. Judgment program to make the computer execute the judgment procedure to be performed.

Images