JP2022122740A - Determination apparatus, determination method, and determination program - Google Patents

Abstract

To reduce burdens and labor relating to an UI examination.SOLUTION: A determination apparatus according to the present invention is characterized by having an acquiring unit for acquiring reference content displayed upon execution of a predetermine application by a reference terminal and comparison content displayed upon execution of the predetermined application by a comparison terminal as a comparison target, and a determination unit for determining whether or not similarity between characteristics of the reference content and characteristics of the comparison content satisfies a predetermined condition.SELECTED DRAWING: Figure 1

Description

The present invention relates to a determination device, a determination method, and a determination program.

Conventionally, various services have been provided to users. In providing such services, contents having a UI (User Interface) set in advance are distributed to users, and information provision to users and acceptance of operations from users are realized.

On the other hand, when providing services or changing the UI, it is necessary to test whether the UI can function properly. Techniques for reducing the time and effort involved in testing such UIs include techniques for generating new test scripts based on test case execution results and techniques for performing rule-based UI tests. .

Japanese Patent No. 6635963 Japanese translation of PCT publication No. 2013-518321 U.S. Patent No. 9841956

However, in the conventional technology described above, there is room for reducing the labor involved in UI testing.

For example, when an application that provides a certain service is executed by a plurality of information processing apparatuses having different platforms such as OS (Operating System), screen size, screen resolution, etc., that is, information processing apparatuses of different types, each information There is a risk that the UI of the content provided via the processing device will differ. For this reason, in the conventional technology described above, it takes time and effort to prepare test cases and rules for testing the UI for each type of information processing device that executes an application.

It should be noted that such a UI test is troublesome not only when developing an application, but also when changing the code of the application due to various updates.

Moreover, in the conventional technology described above, it is necessary to prepare in advance test cases and rules indicating what kind of test should be performed for each UI. However, when test cases and rules are manually prepared in this way, there is a risk of errors such as omissions of the content to be tested.

The present application has been made in view of the above, and an object thereof is to reduce the labor involved in UI testing.

A determination device according to the present application acquires reference content displayed when a reference terminal executes a predetermined application and comparison content displayed when a comparison terminal to be compared executes a predetermined application. and a determination unit that determines whether the similarity between the feature of the reference content and the feature of the comparison content satisfies a predetermined condition.

According to one aspect of the embodiments, it is possible to reduce the effort involved in testing the UI.

FIG. 1 is a diagram illustrating an example of an information providing device according to an embodiment. FIG. 2 is a diagram illustrating an implementation example of generation processing executed by the information providing apparatus according to the embodiment. FIG. 3 is a diagram illustrating an implementation example of application processing executed by the information providing apparatus according to the embodiment; FIG. 4 is a diagram illustrating an example of the flow of determination processing executed by the information providing device according to the embodiment. FIG. 5 is a flowchart showing an example of the flow of determination processing executed by the information providing device according to the embodiment. FIG. 6 is a diagram showing an example of processing for comparing each content. FIG. 7 is a diagram illustrating a configuration example of an information providing apparatus according to the embodiment; FIG. 8 is a diagram illustrating an example of information registered in an operation database according to the embodiment; FIG. 9 is a diagram illustrating an example of information registered in a combination database according to the embodiment; FIG. 10 is a diagram illustrating an example of information registered in a content database according to the embodiment; FIG. 11 is a flowchart illustrating an example of the flow of determination processing according to the embodiment. FIG. 12 is a diagram illustrating an example of a hardware configuration;

EMBODIMENT OF THE INVENTION Below, the form (it is hereafter described as "embodiment".) for implementing the determination apparatus, determination method, and determination program which concerns on this application is demonstrated in detail, referring drawings. Note that the determination device, determination method, and determination program according to the present application are not limited to this embodiment. Further, in each of the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

[Embodiment]
[1-1. About an example of an information providing device]
First, an example of an information providing device 10, which is an example of a determination device according to an embodiment, will be described using FIG. FIG. 1 is a diagram illustrating an example of an information providing device according to an embodiment. Note that FIG. 1 describes, as an example of a function executed by the information providing apparatus 10, an example of processing for generating operation information for testing an application installed in various terminal devices in order to debug the application. did. As will be described later, in parallel with the processing shown in FIG. 1, the information providing device 10 executes determination processing for determining whether the screen on which the application is displayed on the terminal device in the test is an appropriate screen. .

In the following explanation, the flow of processing realized by the operation information generated by the generation processing will be explained using FIGS. A test to determine whether the screen on which the application is displayed is an appropriate screen, that is, the content of the determination process will be described.

An information providing apparatus 10 illustrated in FIG. 1 is an information processing apparatus that executes a generation process, and is realized by, for example, a server apparatus, a cloud system, or the like. For example, the information providing apparatus 10 generates operation information for automatically testing the operation of various software such as an OS (Operating System) installed in the terminal devices 201 to 203 and various applications running on the OS. Operation information will be described later.

In addition to the above-described generation processing, the information providing device 10 has a function of testing the applications installed in the terminal devices 201 to 203 according to the generated operation information. It should be noted that such a function need not be executed by the information providing apparatus 10 itself, and may be realized by another information processing apparatus (for example, an application apparatus) realized by a server apparatus, a cloud system, or the like. For example, the information providing device 10 automatically tests the operation of various software installed in the terminal devices 201-203.

Terminal devices 201 to 203 (hereinafter sometimes collectively referred to as “terminal device 200”) are terminal devices used by users U1 to U3, and are information processing terminals such as smart phones, for example. For example, the terminal device 200 can communicate with the information providing device 10 via a network N such as the Internet. The terminal device 200 may be an information processing device such as a PC (Personal Computer). Also, in the example shown in FIG. 1, three terminal devices 201 to 203 are illustrated, but the embodiment is not limited to this. As will be apparent from the description below, the information providing device 10 may test any number of terminal devices 200 in parallel. Note that in the example shown in FIG. 1, the terminal device 201 is a smartphone, the terminal device 202 is a tablet, and the terminal device 203 is a PC. The information providing device 10 can simultaneously test the operation of software executed in such various types of terminal devices 200 .

The operator terminal 300 is a terminal device used by a user who wishes to test the terminal device 200 or an operator who receives a request from the user who desires the test. It is implemented by an information processing device.

[1-2. About an example of the test flow]
Here, before describing the generation process performed by the information providing apparatus 10, the flow of the test performed by the information providing apparatus 10 will be described. For example, when checking the operation of each terminal device 200 or debugging software, various processes are repeatedly input to each terminal device 200 in various orders, and each operation is performed to check whether an erroneous operation is performed. A judgment will be made. However, when such a test is performed manually, it takes time and effort. In addition, in manual testing, there is a risk that erroneous operations will be performed, making it impossible to achieve proper testing.

Therefore, the information providing apparatus 10 acquires operation information indicating, in natural language, a user's action performed on a predetermined target, in other words, a function provided by a predetermined target such as an application. For example, when an application provides a login function to a user, the user performs an action such as login by exerting the function. In addition, when the application is an application that exhibits various payment functions, the user performs actions such as payment by exerting the payment functions provided by the application. Then, the information providing apparatus 10 generates, from the acquired operation information, application information for applying the operation performed on the target when the action indicated by the operation information is performed to the target.

For example, when testing whether or not login to certain software can be properly performed, the information providing apparatus 10 acquires an instruction indicating the user's action at the time of login in natural language, such as "Login". In such a case, the information providing apparatus 10 identifies an operation for logging in the software to be operated, and generates application information for implementing the identified operation on the software.

For example, in order to perform an action of "login", the information providing apparatus 10 performs an operation of "displaying login content for logging in", then an operation of "entering a user ID" and an operation of "entering a password". If the operation "tap the login button for logging in" is required, each of these operations is specified. It should be noted that the identification of the various operations required to perform such actions is based on dictionary data generated for each application to be operated, and includes expected actions and when the actions are performed. It is identified by a rule base using dictionary data in which operations are associated with each other.

Then, the information providing device 10 generates application information for applying each identified operation to the terminal device 200 . For example, the information providing apparatus 10 provides the terminal device 200 with various known techniques for reproducing various tap operations and text input operations on the terminal device 200 (for example, various UI (User Interface) tests such as Appium). Application information for performing each specified operation on the terminal device 200 is generated using automation framework technology).

[1-3. About generation process]
As described above, the information providing apparatus 10, based on an instruction that generalizes the function to be tested, that is, an instruction that indicates, in natural language, the user's behavior performed on the application and the function that the user uses, You can test your application. Therefore, the operator can conduct the test using only the information expressing the behavior and functions of the user to be tested in natural language.

However, it cannot be said that an appropriate test can be performed only by such processing. For example, in a test of login provided by an application, it is necessary to know not only whether the login was performed properly, but also what kind of operation failed the login when the login was performed. However, it takes time and effort to manually set such detailed operation contents. Moreover, when various operation contents are set manually, there is a possibility that due to human error, the operation contents to be tried are not set, and bugs may be overlooked.

Therefore, the information providing apparatus 10 executes the following generation processing. First, the information providing apparatus 10 identifies the function that the user's operation target exhibits. Subsequently, the information providing apparatus 10 determines a combination of user's operations to be performed on the operation target according to the specified function. Then, the information providing apparatus 10 generates operation information expressing the determined combination of operations in a natural language.

For example, the information providing apparatus 10 analyzes an application to be operated by the user and identifies various functions of the application. For example, the information providing apparatus 10 includes functions provided by applications such as login, various functions related to payment such as QR code (registered trademark) reading and transmission, and functions for applying various contents such as music contents and map contents. Identify the functions that can be used by the user, which functions are exhibited by the Such function specification can be realized by using a model that has learned the relationship between the features of an application and the functions provided by the application.

Next, for each specified function, the information providing apparatus 10 determines a combination of operations to be performed on the application when the function is exhibited. For example, when the specified function is "login", the information providing apparatus 10 performs operations such as "input of user ID", "input of password", and "selection of login button" as operations to be performed on the application. Identify. That is, the information providing apparatus 10 specifies an operation that can be received from the user (in other words, an operation that the user can perform) within the screen for exhibiting the specified function. Note that such a specific process can be realized by using, for example, a table or the like in which the function of the application and the operation for exhibiting the function are associated in advance.

Then, the information providing apparatus 10 determines a combination of operations by appropriately combining the identified operations. For example, the information providing apparatus 10 can perform combinations such as "input of user ID", "input of user ID", and "selection of login button", and "input of user ID", "input of password", and "input of user ID". A predetermined number of combinations are generated by arranging a random number of specified operations in a random order, such as combinations such as "input". In addition, the information providing apparatus 10 determines a combination of operations to be actually applied to the application from among the generated combinations by a process described later.

Then, the information providing apparatus 10 generates operation information expressing the determined combination of operations in a natural language, that is, an operation scenario (in other words, a test scenario) as operation information. For example, when the information providing apparatus 10 determines a combination of operations such as "entering a user ID", "entering a user ID", and "selecting a login button" for the "login" function, the information providing apparatus 10 performs "on the login screen, Input the user ID, then input the user ID, and then tap the login button."

[1-4. An example of generation processing and application processing]
An example of the flow of the generation process executed by the information providing apparatus 10 and the flow of the application process executed by the information providing apparatus 10 will be described below with reference to FIG. For example, the operator installs an application to be operated on each terminal device 200 (step S1). Here, the applications installed in each terminal device 200 may be of different versions. Also, the applications installed in each terminal device 200 may be applications that operate on individual OSs, or may be applications for providing different services. Also, each terminal device 200 may be a different type of hardware. That is, the terminal device 200 to be tested may have a different combination of hardware and software.

Subsequently, the information providing device 10 receives an application to be tested from the operator terminal 300 (step S2). That is, the information providing device 10 accepts an AUT (Application Under Test). For example, the information providing device 10 receives data of applications installed in each terminal device 200 . In the example shown in FIG. 1, the information providing apparatus 10 receives data of applications installed in each terminal device 200 by the operator, but the embodiment is not limited to this. For example, when receiving an application, the information providing device 10 may transmit the received application to each terminal device 200 and cause the terminal device 200 to install the received application. Further, the information providing apparatus 10 may receive an application that the operator terminal 300 has installed in each terminal device 200 .

Subsequently, the information providing device 10 identifies the service provided by the application as a domain (step S3). As described above, the information providing apparatus 10 determines a combination of operation contents based on the functions of the applications to be operated, but such application functions differ depending on the services provided by the applications. For example, it is considered that a login function is essential for an application for making payments, but it is thought that many applications such as a photo viewer do not provide a login function. In this way, it is conceivable that the function exhibited by the operation target changes according to the type of service provided by the operation target.

Therefore, the information providing apparatus 10 identifies the service provided by the application to be operated as a domain. For example, the content of text displayed on the screen by an application can serve as an index that accurately indicates the domain of the service provided by the application. For example, an application that displays a text such as "playback" is presumed to be an application that provides services for playing back various contents such as music. Applications that display text such as "payment" are presumed to be applications that provide various payment services. Therefore, the information providing apparatus 10 identifies the domain of the service provided by the operation target based on the text displayed on the screen by the content.

Subsequently, the information providing apparatus 10 identifies functions provided on each screen of the application according to the domain (step S4). For example, the information providing apparatus 10 identifies, as a function, a constituent service that constitutes a service provided via an application, such as login. In the following description, such a configuration service may be expressed as a "function" of an application.

For example, if two text boxes are included on a certain screen for an application whose domain is "settlement service", the information providing apparatus 10 provides the function "login" on this screen. I judge. Further, when two text boxes are included on a certain screen for an application whose domain is "SNS (Social Networking Service)", the information providing apparatus 10 sets the function provided on this screen to " post”.

As a result of the above-described processing, the information providing apparatus 10 identifies the function provided through each screen displayed by the application. Therefore, the information providing device 10 determines a combination of operations for exhibiting the specified function (step S5). For example, the information providing apparatus 10 identifies operations pre-associated with the identified function, and generates candidate combinations of operations by combining a random number of the identified operations in random order. Then, the information providing apparatus 10 identifies, from among the generated combination candidates, combination candidates that are estimated to be suitable as test targets.

Subsequently, the information providing device 10 generates operation information indicating the determined combination of operations in natural language (step S6). For example, the information providing apparatus 10 refers to dictionary data in which each operation is associated with a natural language indicating the operation, and generates operation information indicating the determined operation content in natural language. For example, when the combination of operations is "input user ID in text box" and "click button", the information providing apparatus 10 performs "input user ID in text box and then click button". , to generate an operation content indicating each operation content and the order in which each operation is to be executed in a natural language. Such operation information may be expressed as, for example, test cases, test scenarios, scenario outlines, feature files, or user flows. In other words, the information providing apparatus 10 generates operation information indicating, in natural language, operations that can be input by the user when the configuration service is provided.

[1-5. About an example of application processing]
Next, an example of the providing process executed by the information providing apparatus 10 will be described. First, the information providing device 10 generates application information for causing each terminal device 200 to provide the operation indicated by the operation information (step S7). For example, the information providing apparatus 10 identifies operation details from the operation information for each OS running on each terminal device 200 and software to be logged in. Subsequently, the information providing device 10 generates application information for applying the identified operation to each terminal device 200 . For example, the information providing device 10 generates test code for executing the identified operation for each terminal device 200 . In other words, the information providing apparatus 10 generates various scripts as application information for causing each terminal device 200 to virtually input the specified operation.

Then, the information providing apparatus 10 outputs the applicable information generated for each terminal device 200 to each terminal device 200 (step S8). For example, when the operation information indicates a test case on the login screen, the information providing apparatus 10 distributes application information to the terminal device 201 for applying the operation in each test case to the terminal device 201. . Similarly, the information providing apparatus 10 distributes to the terminal devices 202 and 203 application information for applying the operations in each test case to the terminal devices 202 and 203 .

In such a case, each terminal device 200 assumes that an operation has been performed according to the application information received from the information providing device 10, and operates the respective application (step S9). Here, the application information output from the information providing device 10 includes information for causing each terminal device 200 to collect the results of application of the operation. As a result, the terminal device 200 collects the operation application results and transmits the collected application results to the information providing device 10 (step S10). In such a case, the information providing apparatus 10 generates a report of the application result and provides the generated report to the operator terminal 300 (step S11). For example, the information providing apparatus 10 determines which terminal device 200 the login succeeded or failed, how many terminal devices 200 the login succeeded or failed, along with the type of OS and application being executed in each terminal device 200. provide such reports. As a result of such processing, the information providing apparatus 10 can promote easy understanding of test results such as debugging.

[1-6. Implementation example of generation process]
As described above, the information providing apparatus 10 generates test cases for testing applications based on the analysis results of the applications. In other words, the information providing apparatus 10 generates operation information indicating, in natural language, a combination of user's operations to be performed on the operation target. In such a generation process, the information providing apparatus 10 generates operation information by a method that is an improved version of IAGA (Improved Adaptive Genetic Algorithm).

An implementation example of the generation process executed by the information providing apparatus 10 will be described below with reference to FIG. FIG. 2 is a diagram illustrating an implementation example of generation processing executed by the information providing apparatus according to the embodiment. Although FIG. 2 describes an example of the flow of processing in which the information providing apparatus 10 generates operation information, the embodiment is not limited to this. The information providing apparatus 10 can generate operation information by arbitrary processing other than the processing shown in FIG. you can go

For example, in the example shown in FIG. 2, the information providing apparatus 10 generates operation information using information registered in the operation database and the combination database. In the operation database, all operation contents (for example, instruc- tion ) that the user can perform on the application to be operated and information indicating each operation contents in natural language are associated and registered. . Note that the operation database may be rephrased as a user action database.

The combination database registers combinations of user operations applied to applications. Here, in the combination database, for example, not only combinations of operations for appropriately exhibiting the functions provided by the application, but also combinations of operations that cannot appropriately exhibit the functions are registered. becomes. That is, the combination database registers information indicating combinations of operations applied during application testing, in other words, test scenarios. Note that the combination database may be rephrased as a population database.

In the following description, combinations of user operations may be collectively referred to as combination information. In addition, the combination information may or may not be registered in the combination database in the initial state.

First, the information providing device 10 extracts text from the application. For example, the information providing apparatus 10 analyzes the code of the application to be tested and extracts all the text displayed by the application on the screen. The information providing device 10 then identifies the domain of the application based on the extracted text. For example, the information providing device 10 uses the domain model to identify the domain of the function provided by the application from the text displayed by the application.

Subsequently, the information providing apparatus 10 identifies all screens displayed by the application. For example, the information providing apparatus 10 analyzes the code of the application and identifies all screens displayed by the application. Note that the information providing apparatus 10 executes the processing described below for each specified screen.

Subsequently, the information providing apparatus 10 identifies functions to be applied via the identified screen. For example, for each specified screen, the information providing apparatus 10 specifies the function provided on that screen based on the type of object placed on that screen and the details of the operation that can be applied on that screen. Here, when specifying the function provided by each screen, the information providing apparatus 10 uses the screen model to specify the function provided by the application based on the domain of the application and the characteristics of each screen. conduct.

For example, the information providing apparatus 10 indicates the domain of the application and the characteristics of a certain screen among the screens displayed by the application (for example, the type, number, appearance of objects, various operations that can be executed on the screen, etc.). A model that has been trained to output information indicating the function provided on the screen when information is input is held as a screen model. Then, the information providing apparatus 10 inputs information indicating the domain of the application to be tested and the characteristics of the screen displayed by the application to the screen model, thereby specifying (estimating) the function provided on the screen. do.

Subsequently, the information providing device 10 refers to the operation database and the combination database, and generates and obtains combination candidates that combine operations for exhibiting the specified function. For example, in the operation database, a function performed by an application and one or more operations performed by a user to perform the function are registered in association with each other. The information providing device 10 refers to the operation database and identifies an operation associated with the identified function. Then, the information providing device 10 generates a predetermined number of combination candidates in which a random number of combinations of the specified operations are combined in random order.

For example, in the operation database, three operations such as "enter user ID", "enter password", and "click login button" are associated with the function "login" and registered in the operation database. shall be In such a case, the information providing apparatus 10 performs predetermined operations such as "input of user ID", "input of user ID", and "input of user ID" as combination candidates corresponding to "login". Candidate combinations that repeat the number of times, candidates that do not perform some operations such as "enter user ID" and "click the login button", and candidates that change the order of execution. , a plurality of combination candidates, etc. with different numbers of operations to be executed are generated. For example, the information providing apparatus 10 may generate combination candidates by randomly combining up to 10 operations.

Also, in the combination database, combination information generated by processing described later is registered in association with functions. The information providing apparatus 10 refers to the combination database and acquires, as combination candidates, combination information associated with functions specified as functions exhibited by the application to be tested.

Subsequently, the information providing device 10 counts the number of steps until the function is exhibited for each combination candidate. For example, the information providing apparatus 10 executes an operation included in each combination candidate on a screen to be tested, and determines whether or not the function is appropriately exhibited before the screen transitions to another screen. , the number of operations executed until the screen transitions to another screen is taken as the number of steps. It should be noted that the coefficient of the number of steps may be rephrased as fitness calculation.

Subsequently, the information providing device 10 determines whether or not the processing has been looped a predetermined number of times. Then, if the information providing apparatus 10 has not looped the processing a predetermined number of times, it selects a pair of parent combination candidates from among the combination candidates determined to exhibit the function. Then, the information providing device 10 generates new combination candidates by randomly combining the operations included in the combination candidates selected as parents. The information providing device 10 also generates new combination candidates by adding or changing operations at random to the newly generated combination candidates. Then, the information providing apparatus 10 re-executes the coefficient of the number of steps until the function is properly exhibited for each combination candidate to determine whether the function is appropriately exhibited.

That is, for each combination candidate, the information providing apparatus 10 generates a coefficient of steps until the function is exhibited, generates a new combination candidate by crossing over the operation, and generates a combination candidate by mutating the operation. Repeat the generation of . By performing such processing, the information providing apparatus 10 can generate combination candidates by combining various operations for each screen of the application to be tested.

Further, when the processing is looped a predetermined number of times, the information providing apparatus 10 registers in the combination database, as combination information, combination candidates whose number of steps satisfies a predetermined condition. For example, the information providing apparatus 10 specifies the distribution of the number of steps required to exhibit the function when the operations included in each combination candidate are executed, and based on the specified distribution, the number of steps satisfies a predetermined condition. Combination candidates are registered. At this time, the information providing apparatus 10 may select a combination candidate with a relatively large number of steps instead of preferentially selecting a combination candidate with a small number of steps.

Then, the information providing device 10 generates operation information indicating the content of the operation in natural language from the combination information registered in the combination database. For example, the information providing device 10 identifies each operation included in the combination information, and identifies a natural language indicating the identified operation from the operation database. Then, based on the specified natural language, the information providing apparatus 10 generates operation information indicating, in natural language, a combination of operations indicated by the operation information.

[1-8. About an example of the flow of application processing]
Next, an implementation example of application processing executed by the information providing apparatus 10 will be described with reference to FIG. FIG. 3 is a diagram illustrating an implementation example of application processing executed by the information providing apparatus according to the embodiment; Here, in addition to the operation information provided by the information providing apparatus 10, the information providing apparatus 10 has a function of generating application information from information indicating, in natural language, a function to be tested among the functions exhibited by the application. have. Therefore, in the example shown in FIG. 3, the flow of the function for generating the application information from the information indicating the function to be tested in natural language is described. The information providing apparatus 10 uses some of these functions to generate application information from operation information and execute a test.

For example, the information providing apparatus 10 acquires from the operator terminal 300 natural language as operation information. For example, the information providing apparatus 10 acquires information indicating the function of the application in the test, such as "the top screen is displayed after logging in", and the result when the function is provided.

Here, the "login" is, for example, a service that constitutes a service provided via the terminal device 200 or an application executed by the terminal device 200, such as an SNS, that is, a constituent service. It can be said that it is a service provided when a predetermined operation is performed. Such constituent services include, for example, not only "login" but also functions such as reception of various posted information posted by users, and a function for capturing a QR code for payment using a QR code. Become.

In such a case, the information providing apparatus 10 uses the extraction unit to extract words that indicate the action performed on the terminal device 200 and the results that occur when the action is performed, from among the words contained in the natural language. and generate feature information that summarizes the extracted words. In other words, the information providing apparatus 10 extracts words indicating actions of the user from the operation information for each action.

For example, the information providing apparatus 10 converts the user's behavior (i.e., behavior to be operated) such as "login" from the operation information such as "top content is displayed when you log in", A word indicating the state of the terminal device 200 when the action is performed (that is, the result of the action) is extracted. The information providing device 10 then generates feature information indicating the extracted words. For such processing, for example, various techniques related to a test description language format called Gherkin such as Cucumber and Turnip can be adopted.

Subsequently, the information providing apparatus 10 uses the driver unit to obtain step information (step function) used to convert the natural language into a device command, and polling the target before sending the command to obtain the step information. Generate assistant information (assistant functions) that are used to support functions.

For example, when generating step information, the information providing apparatus 10 identifies the details of operations occurring in the terminal device 200, the order of the operations, the results of the operations, and the like from the characteristic information. In other words, the information providing apparatus 10 executes a process of translating the behavior of the user into operations on hardware such as the terminal apparatus 200 to be operated and various software executed in the hardware.

For example, when the characteristic information indicates an action of "login", the information providing apparatus 10 first executes "login content display", second executes "user ID input", third executes " step information (step function) indicating a plurality of operations such as "enter password" and "press login button" to be executed fourth, and the order of each operation. That is, the information providing device 10 identifies various operations performed on the terminal device 200 and the order in which each operation is performed when the action indicated by the word "login" is performed, and step information indicating the identified operation and order. to generate

Further, when the feature information includes an action result such as "top content is displayed", the information providing apparatus 10 generates assistant information (assistant function) for determining whether or not such a result has occurred. do. For example, when an action result such as "top screen is displayed" is included, the information providing apparatus 10 generates assistant information indicating processing such as "determine whether or not top content is displayed". Note that the information providing apparatus 10 may generate assistant information for determining whether or not a predetermined result occurs when various operations indicated by the step information are performed, for example. For example, if the step information includes an operation such as "display login content", the information providing apparatus 10 may generate assistant information indicating a process such as "determine whether or not login content has been displayed".

That is, the information providing apparatus 10 generates step information indicating various operations and the order of the operations for realizing the action indicated by the operation information based on the operation information described in a natural language. In addition, the information providing apparatus 10 generates assistant information indicating details of processing performed to determine whether or not a result that occurs when an operation indicated by each step information is performed has actually occurred.

Note that the step information and the assistant information are generated for each OS type or application type executed by the target terminal device 200, or for each combination thereof. For this reason, the driver unit acquires various inputs that occur in user actions (behaviors) indicated by feature information generated from operation information, and identifies devices and applications that are targets of user actions and elements that are targets of operation. conduct. Then, the driver unit generates step information based on the identification result.

Similarly, the information providing device 10 generates assistant information for collecting (polling and querying) the state of the target at regular intervals to ensure that each operation indicated by the step information has been performed. do. More specifically, the information providing apparatus 10 collects, for each target, the state of the target that occurs when the operation indicated by the step information is performed, and determines whether the collected state satisfies a predetermined condition. Generate assistant information for In other words, the assistant information determines whether the result of performing a certain operation indicated by the step information is appropriate, and if it is determined to be appropriate, the assistant information sets the conditions for performing the next operation indicated by the step information. can be regarded as information.

Subsequently, the information providing apparatus 10 uses the generation unit to generate application information for causing the target to perform operations and various determinations indicated by the step information and the assistant information. In other words, the information providing device 10 generates application information for applying the operation performed on the terminal device 200 when the action indicated by the word extracted by the extraction unit is performed.

Such a generation unit can be realized by various frameworks such as Appium for automatically testing various terminal devices 200 and applications. More specifically, the information providing apparatus 10 applies each operation indicated by the step information to the target in the order indicated by the step information, and determines whether the result of each operation is appropriate according to the conditions indicated by the assistant information. generate a script for

The script generated in this way is a script for causing the terminal device 200 to apply the actions of the user that can be performed when providing the configuration service such as "login", that is, the operation corresponding to the configuration service. . As described above, the information providing apparatus 10 generates action information indicating, in natural language, a constituent service that constitutes a service provided by a predetermined target and that is provided when a user performs a predetermined operation. Once acquired, application information for applying a predetermined operation corresponding to the configuration service indicated by the acquired behavior information to the object is generated from the acquired behavior information.

Then, the information providing device 10 outputs the applicable information to the target terminal device 200 . For example, the information providing apparatus 10 provides a virtual terminal device ET reproduced in a predetermined information processing device, a terminal device RT such as a smartphone, and various terminal devices CT existing on the cloud. 200 generated application information is transmitted. As a result, the information providing device 10 can test the operation of the terminal device 200 when the action indicated by the operation information written in natural language is performed.

In the above example, a driver unit may be provided for each combination of an extraction unit and a generation unit. For example, the driver unit may generate step information and assistant information in a format identifiable by the generation unit from the test description language in the format adopted by the implemented extraction unit. Such a driver unit includes, for example, dictionary data that associates actions indicated by the test description language generated by the extraction unit with the operation contents and the order of operations for realizing the actions, each operation, and each operation. An arbitrary program that operates on a rule basis can be adopted using dictionary data that associates conditions for determining whether the results of the above are appropriate.

For example, when the feature information indicates "login", the driver unit refers to the dictionary data and performs operations "display login content", "enter user ID", and "enter password" associated with "login". ”, and “press login button”. Then, the driver unit may generate step information for causing the generation unit to generate application information for causing the terminal device 200 to execute each specified process. Also, the driver unit refers to the dictionary data to identify conditions associated with each identified operation. Then, the driver unit may generate assistant information for causing the generation unit to generate application information for causing the terminal device 200 to determine whether the specified condition is satisfied. The information providing apparatus 10 can employ any program as a driver unit as long as it is a program that performs such processing.

When receiving the operation information, the information providing apparatus 10 uses the driver unit to generate step information and assistant information indicating each operation indicated by the operation information in natural language. Then, the information providing apparatus 10 uses the generation unit to generate application information from the generated step information and assistant information. As a result of such processing, the information providing device 10 can apply various operation contents as test cases to each terminal device 200 .

[2. About judgment processing]
As described above, the information providing device 10 tests whether the application installed in each terminal device 200 operates properly. On the other hand, among services realized by applications, there are services that provide common services via various types of terminal devices 200, such as settlement using QR codes.

However, even the same application may look different for each terminal device 200 . For example, even if the terminal device 201 and the terminal device 202 are terminals that run the same OS, if the screen resolutions are different, the appearance may change. When the appearance of the screen provided by the application (hereinafter sometimes referred to as "content") changes in this way, it is necessary to cause unnecessary anxiety when using the application. Further, for example, when the arrangement of various objects changes, the user may not be able to perform correct operations.

A more specific example will be given. For example, for logging in, an application inputs a text box for entering a user ID, a text indicating that the user ID is to be entered in the text box (for example, "user ID"), and a password. and a text indicating that a password is to be entered in the text box (for example, "password").

Here, the terminal device 201 displays a text box for entering a user ID and a text "user ID" side by side, and displays a text box for entering a password and a text "password". When displayed side by side, it is considered that the user of the terminal device 201 can correctly input the user ID and password into the text boxes. However, for example, if the terminal device 202 does not have a screen resolution that allows the text and text boxes to be arranged side by side, the arrangement of the text and the text boxes will change, and it will be difficult to enter the user ID or password in the text box. There is a risk that it will not be possible to correctly tell the user what to enter.

In addition, depending on the OS and screen settings executed by the terminal device 200 and the characteristics of the hardware, the shapes and colors of the logos of the services and companies displayed by the applications, and the colors commonly used in the content provided by the applications (so-called theme color) may appear differently than expected.

Such a problem of appearance consistency of content provided by an application may occur not only when the application is developed, but also when the application is updated, or when the service provided by the application is changed or added. In other words, the content provided by the application may not be identical in appearance even if the application has the same code for various reasons such as the hardware or software platform of the terminal device 200 that executes the application. .

Therefore, the information providing apparatus 10 executes determination processing for determining whether or not the content is guaranteed to be identical in appearance in the application test described above. That is, when the user views each content, it is determined whether or not it is determined that the content is for realizing the same configuration service by the same application.

For example, the information providing device 10 uses a predetermined terminal device prepared separately from the terminal device 200 as a reference terminal, and the terminal device 200 as a determination device. Then, the information providing apparatus 10 acquires the reference content displayed when the reference terminal executes the predetermined application and the comparison content displayed when the comparison terminal to be compared executes the predetermined application. . Then, the information providing apparatus 10 determines whether or not the similarity between the feature of the reference content and the feature of the comparison content satisfies a predetermined condition.

For example, when an application determines the identity of content displayed on each terminal device 200, the information providing device 10 uses an application (for example, not only an application with the same code but also an application that provides the same service to the reference terminal and the comparison terminal). , code that provides the same service, but applications that run on different OSs). Subsequently, the information providing apparatus 10 acquires the reference content, which is the content displayed on the reference terminal by the application, and the comparison content, which is the content displayed on the comparison terminal by the application.

Then, the information providing apparatus 10 determines the similarity between the reference content and the comparison content, and if the similarity satisfies a predetermined condition, the content displayed as the comparison content is normal content, that is, is provided to the user. It is determined that the content can be determined to be the content of the same service as the reference content. As a result of such processing, the information providing apparatus 10 automatically determines whether appropriate content is being provided, for example, when developing an application that provides services provided on multiple platforms or when updating an application. can be done, and the labor of the test can be further reduced.

[2-1. About an example of the flow of determination processing]
An example of the flow of determination processing executed by the information providing apparatus 10 will be described below with reference to FIG. FIG. 4 is a diagram illustrating an example of the flow of determination processing executed by the information providing device according to the embodiment. It is assumed that the processing shown in FIG. 4 is performed in parallel with the test shown in FIG. 1, and an application to be tested is already installed in each terminal device 200 . Further, in the example shown in FIG. 4, an example in which the operator terminal 300 provides an application to the information providing apparatus 10 will be described. For example, you may provide image files that can be used as reference content, such as full-screen app mockups created by your design team, icons, and components to include in your design guidelines.

First, the operator terminal 300 installs an application to be tested in the reference terminal BT (step S20). In such a case, the operator acquires, at the reference terminal BT, a snapshot of the content displayed by the application on the reference terminal BT. Then, the operator terminal 300 acquires the snapshot as the reference content (step S21).

Subsequently, the operator terminal 300 registers the acquired reference content as a baseline in a predetermined cloud server CS (step S22). Then, the operator terminal 300 provides the application to be tested to the information providing device 10 (step S23). Note that step S23 corresponds to step S2 shown in FIG.

In such a case, the information providing device 10 executes determination processing in parallel with each processing (steps S3 to S11) shown in FIG. For example, the information providing device 10 acquires the baseline from the cloud server CS (step S24). Then, the information providing apparatus 10 executes steps S3 to S7 shown in FIG. 1, and outputs application information as shown in FIG. 4 (step S25). Note that step S25 corresponds to step S8 shown in FIG.

Here, the information providing apparatus 10 acquires, as application information, a snapshot of the content displayed on the screen by the application as comparison content, and generates application information for operating the terminal device 200 so as to include it in the application result. Such application information can be realized, for example, by generating the application information by the application process described above from a natural language such as acquiring a screenshot when the screen is displayed. For example, in step S6 shown in FIG. 1, the information providing device 10 acquires a snapshot in the terminal device 200 to be tested, generates operation information for outputting the snapshot, acquires a snapshot in step S7, It is sufficient to generate application information for outputting as an application result.

Then, the information providing device 10 acquires the comparison content displayed by each terminal device 200 as the application result (step S26). Note that step S26 corresponds to step S10 shown in FIG. In this way, the information providing apparatus 10 acquires comparison content displayed when the comparison terminal executes the same application as the reference terminal.

Subsequently, the information providing device 10 creates a report prior to step S11 shown in FIG. At this time, the information providing apparatus 10 executes determination processing for determining similarity between the reference content and the comparison content. For example, the information providing apparatus 10 determines whether or not the external and/or functional features of the reference content and the external and/or functional features of the comparison content are similar (step S27).

For example, FIG. 5 is a flowchart showing an example of the flow of determination processing executed by the information providing device according to the embodiment. For example, when starting the determination process, the information providing apparatus 10 determines whether or not the baseline corresponding to the comparison content, that is, the reference content is registered (step S101). For example, the information providing apparatus 10 determines whether or not the reference content displayed by the application on the reference terminal is registered at the same timing as when the application displays the comparison content on the comparison terminal. Here, the displayed timing is, for example, when an application is started or when a predetermined operation is performed, and is different from the actual time. That is, the information providing apparatus 10 compares the reference content and the comparison content displayed when providing the same constituent service.

Subsequently, when the corresponding baseline is registered (step S101: Yes), the information providing apparatus 10 extracts various features and object information from the baseline (step S102). For example, the information providing apparatus 10 extracts the appearance features of the entire screenshot of the reference content serving as the baseline. Note that such processing can be employed by, for example, various feature amount extraction techniques. Also, the information providing apparatus 10 extracts information of each object included in the reference content. For example, the information providing apparatus 10 extracts the content, color, font type, display size, and the like of each text included in the reference content using OCR (Optical Character Recognition) technology or the like.

The information providing apparatus 10 also uses various image recognition techniques to extract information on various objects such as buttons and text boxes included in the reference content. For example, the information providing apparatus 10 extracts what kind of appearance (color, display size, etc.) objects are included, and what kind of function objects are included. Note that the information providing apparatus 10 may specify the function of each object by analyzing the code included in the application instead of the snapshot.

Furthermore, the information providing apparatus 10 may acquire the arrangement position of each object in the reference content. For example, the information providing apparatus 10 may acquire the position on the screen where the text box is displayed in the reference content. In this way, the information providing apparatus 10 extracts the features of the reference content as a whole and the external features and functional features of various objects included in the reference content.

Subsequently, the information providing device 10 acquires comparison content (step S103). Then, the information providing apparatus 10 extracts various features and object information from the comparative content in the same manner as when extracting the external features and functional features of various objects from the reference content (step S104). Then, the information providing apparatus 10 determines similarity between features and objects extracted from each content (step S105).

The information providing apparatus 10 determines whether or not the similarity between the appearance features of the reference content and the appearance features of the comparison content satisfies a predetermined condition. As a more specific example, the information providing apparatus 10 may calculate the degree of similarity between the feature amount of the reference content and the feature amount of the comparison content.

Further, the information providing apparatus 10 acquires the color, size, and arrangement position of various objects such as texts and buttons arranged in the reference content and the comparison content as appearance features. In addition, the information providing apparatus 10 calculates the matching degree of character information indicated by the text, the appearance feature quantity, and the similarity of the appearance features of various contents. Further, the information providing apparatus 10 may calculate the degree of similarity according to whether or not the functions of various objects are the same.

The information providing device 10 then determines whether or not the similarity satisfies a predetermined condition (step S106). For example, the information providing apparatus 10 determines whether the calculated similarity exceeds a predetermined threshold. Then, when the similarity satisfies a predetermined condition (step S106: Yes), the information providing apparatus 10 presumes that the user can recognize each content as content providing the same constituent service. If yes, it registers that the comparison content is the reference content (step S107). On the other hand, if the similarity does not satisfy the predetermined condition (step S106: No), the information providing apparatus 10 can presume that the user cannot recognize each content as content that provides the same constituent service. If so, it registers that the comparison content is the reference content (step S108).

If the corresponding baseline is not registered (step S101), the information providing apparatus 10 performs baseline registration processing (step S109). For example, the information providing apparatus 10 uses the comparison content acquired by one of the terminal devices 200 among the terminal devices 200 as the baseline, and compares the comparison content acquired by the other terminal devices 200 with the baseline. may Note that the information providing apparatus 10 may prompt the operator to register the baseline, for example.

Here, an example of processing in which the information providing apparatus 10 compares the reference content and the comparison content will be described with reference to FIG. FIG. 6 is a diagram showing an example of processing for comparing each content. In the example shown in FIG. 6, an example is shown in which, among the contents displayed by the application, comparison is made between the contents that provide the login configuration service in order to receive the provision of the service.

For example, the reference content BC shown in FIG. 6 includes a logo indicating a service, a banner, text boxes in which texts such as "mobile phone number" and "password" are arranged, various terms of use, and "new registration after agreeing to the above". , text such as "new registration with an external account", buttons for logging in using each external account, and the like. Also, in the comparison content CC shown in FIG. 6, although objects similar to those of the reference content BC are arranged, for example, among the contents included in the reference content BC, the upper part of the text box for entering the mobile phone number and password There shall be no display of banners placed on

In such a case, the information providing apparatus 10 uses various OCR techniques to extract text from the reference content BC and the comparison content CC. Then, the information providing apparatus 10 determines whether or not the text extracted from the reference content BC and the text extracted from the comparison content CC match in appearance. In addition, the information providing apparatus 10 determines the presence or absence of various contents such as banners and text boxes and similarity. For example, in the example shown in FIG. 6, among the various objects included in the reference content BC, the comparison content CC does not include the banner placed above the text box for entering the mobile phone number and password. For this reason, the information providing apparatus 10 extracts the banner placed above the text box for entering the mobile phone number and password as the difference DF, and determines the presence or absence of the banner as the difference DF and the amount of text identity. Based on this, the similarity of each content is determined.

Returning to FIG. 4, the description is continued. The information providing device 10 provides a report according to the determination result (step S28). Note that step S28 corresponds to step S11 shown in FIG. For example, the information providing apparatus 10 provides a report or the like indicating whether the external features of each content are similar to the reference content as the comparison content, whether the degree of similarity exceeds a predetermined threshold, or the degree of similarity itself. do. As a result of such processing, the information providing apparatus 10 determines whether or not the application to be tested provides content that looks as expected on each terminal device 200. When using the device 200, it is possible to determine whether or not to provide content that can be recognized as content related to the same service.

[2-2. About comparison content]
As described above, the information providing device 10 determines the similarity between the reference content and the comparison content. At this time, the information providing apparatus 10 determines the external features of each content and/or the functional similarity of objects arranged in each content. An example of a target for similarity determination by the information providing apparatus 10 will be described below.

For example, the information providing apparatus 10 may extract various marks and images arranged in each content as objects, and calculate a score based on the similarity of the external features of the extracted marks and images. good. For example, the information providing apparatus 10 may calculate the score based on the similarity in appearance such as the color and display size of each mark or image.

Further, the information providing apparatus 10 may extract characters displayed in each content as objects. Here, the information providing apparatus 10 may extract characters from each content serving as a screen shot using OCR technology, or may specify what characters are displayed from the application code. Then, the information providing apparatus 10 may calculate a score based on the similarity of appearance features such as character color, display size, font, etc. of each content. Further, the information providing apparatus 10 may calculate a score based on the identity of character strings. Further, when different languages are selected for the reference content and the comparison content, the information providing apparatus 10 may calculate a score based on the similarity of intention indicated by the character strings.

Further, the information providing apparatus 10 may calculate a score based on the similarity in appearance of functional objects, such as buttons and text boxes, that perform various functions when operated, among the objects arranged in each content. . For example, in each content, there are objects such as text and images that do not exhibit their functions when the user operates (in other words, objects that are static with respect to operations), buttons, text boxes, etc. Objects that, when selected by the user, perform predetermined functions such as displaying other content or accepting text input (in other words, objects that are dynamic with respect to operation) may be included. Therefore, the information providing apparatus 10 identifies such objects that perform various functions when selected by the user as functional objects, and determines the presence or absence of the identified functional objects, functional similarity, and/or external appearance. The similarity of each content is determined based on the similarity. That is, for static objects, the information providing apparatus 10 determines the similarity between the appearance of the object and the information indicated by the object (for example, the meaning of characters without an image), and for dynamic objects, The similarity of each content is determined based not only on the appearance of the object but also on the similarity of the functions exhibited by the object.

For example, the information providing apparatus 10 may identify functional objects that are presumed to be the same from each content, such as appearance and code, and perform pixel-by-pixel comparison of the functional objects. You may calculate the score according to the similarity of . That is, the information providing apparatus 10 may calculate a score based on the external similarity of not only objects such as text but also functional objects that perform some function.

Note that the information providing apparatus 10 confirms the presence or absence of the same object, confirms the similarity of the functions of each object, and then calculates a score according to the step-by-step comparison result of performing pixel-by-pixel comparison. You may When executing such processing, for example, if the same object (not only a functional object but also other objects may be included) does not exist, the information providing apparatus 10 performs subsequent processing such as pixel-by-pixel comparison. may not be performed. On the other hand, when the same object exists, the information providing apparatus 10 may compare the result of pixel-by-pixel comparison only for the objects that are estimated to be the same. That is, the information providing apparatus 10 does not need to perform pixel-by-pixel comparison for the entire screen, and may reduce the processing load, improve efficiency, and reduce processing time.

Further, the information providing apparatus 10 may determine whether or not the similarity between the functional object arranged in the reference content and the functional object arranged in the comparative content satisfies a predetermined condition. That is, the information providing apparatus 10 may calculate a score based on the similarity of the features of the functions of each content. For example, the information providing apparatus 10 may calculate a score according to the similarity of functions exhibited by functional content included in each content.

For example, the information providing apparatus 10 may calculate a score according to the similarity between the type of functional object placed in the reference content and the type of functional object placed in the comparison content. More specifically, the information providing apparatus 10 may calculate a score indicating higher similarity when functional objects that exhibit the same function are included in each content. In other words, the information providing apparatus 10 may calculate a higher score as the functions exhibited by the functional objects in each content are more similar.

The information providing device 10 may also calculate a score based on the similarity of the positions where each object is arranged. For example, the information providing apparatus 10 identifies the arrangement position of each text or functional object (for example, coordinates based on the upper left pixel that is the origin of the content), and calculates a score based on the similarity of the identified arrangement position. may

Further, the information providing apparatus 10 determines the similarity between the relative placement positions of the plurality of objects placed in the reference content and the relative placement positions of the plurality of objects placed in the comparison content. It may be determined whether or not the conditions of are satisfied. For example, the information providing apparatus 10 determines the relative positional relationship between text and textboxes in the reference content (for example, whether they are aligned vertically or horizontally), the relative positions of the textboxes, and so on. The relative positional relationship of various objects, such as relationship, positional relationship between buttons and other objects, etc., is specified. Similarly, the information providing apparatus 10 identifies relative positional relationships of various objects within the comparison content.

Then, the information providing apparatus 10 calculates a score based on the similarity of the relative positional relationship between the objects in each content. For example, the information providing apparatus 10 calculates a score indicating higher similarity when a predetermined text and a predetermined text box are arranged vertically in each content. On the other hand, in the information providing apparatus 10, the predetermined text and the predetermined text box are arranged vertically in the reference content, but the predetermined text and the predetermined text box are arranged horizontally in the comparison content. If so, calculate the score that indicates the lower similarity. Note that the information providing apparatus 10 may specify the relative positional relationship for a predetermined set of arbitrary objects, and determine the similarity of the specified positional relationship.

Further, the information providing apparatus 10 determines whether or not the similarity between the order in which the plurality of objects are arranged in the reference content and the order in which the plurality of objects are arranged in the comparison content satisfies a predetermined condition. You can judge. For example, the information providing apparatus 10 identifies the order in which each object is arranged in the reference content and also identifies the order in which each object is arranged in the comparison object. For example, the information providing apparatus 10 identifies the order in which the predetermined text, predetermined text boxes, and buttons are displayed from the top of the content. Then, the information providing apparatus 10 may calculate a score with a higher value as the similarity of the specified order is higher.

Further, the information providing apparatus 10 may determine similarity based on the presence or absence of each object. For example, the information providing apparatus 10 may identify objects common to each content (for example, objects that match in appearance or whose similarity exceeds a predetermined threshold, objects that are presumed to be displayed by the same code, objects that have the same function object) is included. Then, the information providing apparatus 10 calculates a score indicating higher similarity when an object common to each content is included, and calculates a score indicating lower similarity when it is not included. may

Further, the information providing apparatus 10 may calculate a score based on the similarity of the features of the attributes of each content. For example, the information providing apparatus 10 may determine whether or not each content includes audio or moving images, and calculate a score according to the determination result.

Note that the information providing apparatus 10 may calculate a plurality of scores based on some or all of the above-described score calculation standards, and use the total value of the calculated scores as the final score. good. Further, the information providing apparatus 10 may set a different weight for each calculation criterion and calculate the total value of the scores. Then, the information providing apparatus 10 determines whether or not the reference content and the comparison content are similar based on whether or not the total value of the scores exceeds a predetermined threshold. It is determined whether or not the condition is satisfied.

Note that the information providing apparatus 10 may determine whether or not the similarity of each content satisfies a predetermined condition based on the number of matching points and different points without calculating the score. Further, the information providing apparatus 10 may count the number of criteria satisfying a predetermined condition among various calculation criteria, and determine similarity depending on whether the counted number exceeds a predetermined threshold. .

[2-3. About the use of the model]
As described above, the information providing apparatus 10 determines the similarity of each content according to the similarity of appearance between the respective contents, the similarity of objects, and the like. At this time, the information providing apparatus 10 may determine the similarity of each content using a model that has learned whether or not the user determines that the content provides the same constituent service.

For example, the information providing apparatus 10 generates changed content by randomly changing the appearance and arrangement position of each object included in the reference content. Further, the information providing apparatus 10 provides the changed content and the reference content to the user, and inquires whether each content is the same or similar content, that is, inquires about the label of the set of content. Inquiries about such labels may be realized by, for example, crowdsourcing. Then, the information providing apparatus 10 outputs a label indicating whether or not the content is similar to a model such as a DNN (Deep Neural Network) when screenshots or codes of the reference content and the change content are input. Train the model.

Then, the information providing apparatus 10 may input the reference content and the comparison content into the above-described model during the test, and implement determination of whether or not they are similar. In this way, the information providing apparatus 10 may determine the similarity of the appearance and functions of each content based on a rule, or directly using a model.

[2-4. Regarding the range of content to be compared]
Here, the information providing apparatus 10 compared screen shots as the reference content and the comparison content. However, depending on the screen size of the reference terminal or the comparison terminal, there is a possibility that the entire content may not be displayed at once. As a result, when single contents are compared, it may be determined that the similarity is low even though the reference contents and the comparison contents include the same object.

Therefore, the information providing apparatus 10 may compare the similarity between any number of reference contents and comparison contents. For example, the information providing apparatus 10 acquires a plurality of screen shots when the reference terminal displays the reference content and a plurality of screen shots when the comparison terminal displays the comparison content. Based on this, the similarity of each content may be determined. Further, the information providing apparatus 10 acquires screenshots of the reference terminal and screenshots of the comparison terminal, the number of which is different from that of the screenshots of the reference terminal, and based on the respective comparison results, each Content similarity may be determined.

For example, when an application causes content to be displayed on the reference terminal BT, if only part of the content to be displayed is being displayed, the information providing device 10 may display the entire content on multiple screens. Let the shots be taken, and take these multiple screenshots as the reference content. Similarly, when an application causes content to be displayed on the terminal device 200, if only part of the content to be displayed is being displayed, the information providing device 10 displays a plurality of images so that the entire content is captured. Screen shots are taken, and these multiple screen shots are used as comparison content.

Then, the information providing apparatus 10 may determine the similarity between the reference content as a whole and the comparison content as a whole by comparing the reference content and the comparison content. Here, the information providing apparatus 10 performs a one-to-one comparison, such as comparing one screenshot as reference content with one of the screenshots as comparison content. Alternatively, comparison may be made in a one-to-many correspondence, such as one screenshot as reference content and a plurality of screenshots as comparison content. Further, the information providing apparatus 10 may determine the similarity of each screen shot in a round-robin manner, and based on each determination result, determine the similarity between the reference content as a whole and the comparison content as a whole.

Further, when the reference terminal BT displays the entire content and the terminal device 200 displays only a part of the content, the information providing apparatus 10 displays one screen shot as the reference content and a plurality of screen shots as the comparison content. may be compared to determine similarity between the overall reference content and the overall comparison content. Further, when the reference terminal BT displays a part of the content and the terminal device 200 displays the entire content, the information providing apparatus 10 displays a plurality of screen shots as the reference content and one screen shot as the comparison content. may be compared to determine similarity between the overall reference content and the overall comparison content.

[2-5. Applicable target]
In the example described above, the information providing device 10 determined the similarity of each content displayed on the reference terminal BT and the terminal device 200 by the application providing the same service. Here, the information providing apparatus 10 may determine the similarity of each content displayed by an application configured with the same code, or determine the similarity of each content displayed by an application configured with different codes. may

For example, when the reference terminal BT and the comparison terminal run the same OS, the information providing device 10 installs an application composed of the same code on each, and the application causes the content displayed on each terminal to resemble each other. Gender may be determined. On the other hand, when the reference terminal BT and the comparison terminal run different OSs, the information providing apparatus 10 may use applications configured with different codes to provide the same service on different OSs. Install on compatible devices. Then, the information providing apparatus 10 may determine the similarity of the content displayed by each terminal.

Further, the information providing apparatus 10 may notify the user of information indicating the relationship between the application executed by the reference terminal and the application executed by the comparison terminal based on the determination result. For example, when the similarity between the content displayed by the reference terminal and the content displayed by the comparison terminal satisfies a predetermined condition, the information providing apparatus 10 allows the user to recognize that each application provides the same service. may be determined to be a valid application. Note that when performing such processing, the information providing apparatus 10 executes the above-described determination processing for each content displayed by the application, and the proportion of content satisfying the predetermined condition for similarity satisfies the predetermined condition. In this case, it may be determined that each application provides the same service as an application that can be recognized by the user.

Further, the information providing apparatus 10 may provide the operator with information about the content whose similarity has been determined not to satisfy a predetermined condition. If such information is provided, the operator can easily recognize which content should be modified in developing or updating the application.

The information providing apparatus 10 may also determine the robustness of services provided by certain applications. For example, the information providing apparatus 10 executes the determination process described above for each content displayed by an application, and if the proportion of content that satisfies a predetermined similarity condition satisfies the predetermined condition, the robustness of the application. satisfies a predetermined condition, and if the percentage of content whose similarity satisfies the predetermined condition does not satisfy the predetermined condition, it may be determined that the robustness of the application does not satisfy the predetermined condition. Further, the information providing apparatus 10 may notify the operator of such robustness determination results.

Further, the information providing apparatus 10 may determine unauthorized applications using the similarity determination result of each content. For example, the information providing apparatus 10 prepares terminal devices of the same type as a reference terminal and a comparison terminal, and installs a valid application in the reference terminal and an application to be judged in the comparison terminal. Note that the application to be determined may be a web application that runs on a browser.

Then, the information providing apparatus 10 determines the similarity between the reference content displayed on the reference terminal and the comparison content displayed on the comparison terminal, and determines whether the application installed on the comparison terminal is illegal according to the determination result. You may determine whether it is an application. For example, if the content is highly similar but not completely matched, the information providing apparatus 10 may be configured to prevent an application installed in the comparison terminal from imitating a legitimate application to obtain personal information. may be determined to be a valid application.

Here, the information providing apparatus 10 may guide the user to a phishing site and determine a fake web application that steals the user's login ID and password. For example, the information providing apparatus 10 can display content displayed by a formal web application that provides various services to users via the web (that is, not only content independently provided by the application, but also web API Various web contents provided in cooperation with ) and various contents displayed by the fake web application will be compared. At this time, the information providing apparatus 10 determines whether the web application to be compared is a fake web application based on the similarity of the appearance features of each content and the similarity of attributes such as the domain of each content. may be determined.

[2-6. Applications that provide multiple services]
Here, an application called a super application or the like may provide not only a single service but also multiple services. However, in many cases, such applications do not provide services by a single application program, but the main application calls multiple applications called mini-apps to provide various services. Therefore, when checking the operation of the super application, the information providing apparatus 10 may compare the reference content and the comparison content for each mini application that constitutes the super application.

[2-7. About language]
Here, when the reference terminal and the comparison terminal display contents in different languages, the information providing apparatus 10 may regard texts with the same contents as the same object. For example, when the reference content includes a text such as "password" and the comparison content includes a content such as "password", the information providing apparatus 10 considers these contents to be the same content, and Similarity may be determined.

[2-8. About terminal]
Here, the information providing apparatus 10 can use any terminal as a reference terminal other than the terminal device 200 such as a smartphone, the terminal device CT on the cloud, or the terminal device ET emulated on a predetermined information processing device. or a comparison terminal. For example, the information providing apparatus 10 may determine the similarity of each content using physical devices (so-called hardware) such as PCs, server devices, various sensors, and digital signage as reference terminals and comparison terminals. In addition, the information providing apparatus 10 generates reference content and comparison content for each OS and application executed on various virtual machines and PCs, and determines the robustness of the application based on the similarity of each content. You may

[2-9. About an example of the effect of the embodiment]
As described above, the information providing apparatus 10 determines the similarity between the reference content displayed on the reference terminal and the comparison content displayed on the comparison terminal. Therefore, the information providing apparatus 10 can automatically determine the identity and robustness of the same or different applications that display the reference content and the comparison content, and can facilitate application testing and the like.

Note that even if the reference content and the comparison content must be generated for each terminal device and each OS, the information providing device 10 generates operation information indicating operation details in a natural language, and performs the generated operation. Automatically generate application information based on information. Therefore, even if there are multiple versions of the application to be tested, or multiple variations of the terminal device to be used as the reference terminal or the comparison terminal, the information providing device 10 can It is possible to automatically identify combinations of terminals and applications, and automatically determine the similarity of each content for each identified combination. Therefore, the information providing apparatus 10 can reduce the trouble of manually operating the application and checking the screen.

[3. Configuration of Information Providing Device]
An example of the functional configuration of the information providing apparatus 10 described above will be described below. FIG. 7 is a diagram illustrating a configuration example of an information providing apparatus according to the embodiment; As shown in FIG. 7, the information providing device 10 has a communication section 20, a storage section 30, and a control section 40. FIG.

The communication unit 20 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 20 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the information providing device 10 , the terminal device 200 and the operator terminal 300 .

The storage unit 30 is implemented by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 30 also stores an operation database 31 , a combination database 32 and a content database 33 .

In the operation database 31, various functions performed by the operation target are registered in association with user operations performed to perform the functions. For example, FIG. 8 is a diagram showing an example of information registered in the operation database according to the embodiment. In the example shown in FIG. 8, the operation database 31 registers information having items such as domains, corresponding functions, operation details, and natural language expressions. In addition to the information shown in FIG. 8, the operation database 31 may also register information related to user operations applicable to the operation target.

For example, the domain is information indicating the type of service provided to the user by the operation target. Further, the corresponding function is a function provided by an operation target belonging to the associated domain, and is information indicating a function provided via a predetermined screen. Further, the operation content is information indicating the content of the operation that the user can perform on the screen that exhibits the associated function. A natural language expression is information indicating the associated operation in a natural language.

For example, in the example shown in FIG. 8, the domain "settlement", the corresponding function "login", the operation content "operation #1-1", and the natural language expression "click" are associated and registered. Such information indicates that an operation target whose service to be provided is "settlement" has a function of "login" on a certain screen, and that an operation that a user can perform on such a screen is "operation #1- 1”, indicating that the natural language expression for such an operation is “click”. In the example shown in FIG. 8, a plurality of operations such as "operation #1-1" and "operation #1-2" are associated with the corresponding function "login". Such information indicates that multiple operations such as "operation #1-1" (eg, click) and "operation #1-2" (eg, input of user ID) can be performed on the "login" screen. show.

In the example shown in FIG. 8, a conceptual value such as "operation #1-1" is described, but in reality, various types of information (for example, instance) indicating the content of the operation are registered. Further, the operation database 31 may further register, for example, the amount of scrolling, the number of clicks, character strings constituting user IDs and passwords, and the like.

Information indicating combinations of operations, ie, operation information, is registered in the combination database 32 . For example, FIG. 9 is a diagram showing an example of information registered in the combination database according to the embodiment. In the example shown in FIG. 9, corresponding domains, corresponding functions, and operation combinations are registered in the combination database 32 in association with each other. In addition to the information shown in FIG. 9, the combination database 32 may also register various types of information relating to combinations of operations.

Here, the operation combination is information indicating a plurality of operations determined according to the functions to be operated. The corresponding function is information indicating which function the operation indicated by the operation combination is a test case for. Further, the corresponding domain is information indicating to which domain the associated operation combination belongs to the operation target test case.

For example, in the example shown in FIG. 9, the corresponding domain "settlement", the corresponding function "login", and the operation combinations "combination #1", "combination #2", etc. are registered in association with each other. Such information is the operation indicated by the test case "combination #1", "combination #2", etc. performed on the screen providing the function of "login" of the operation target providing the service related to "payment". indicate to the effect. Note that in the example shown in FIG. 9, a conceptual value such as "combination #1" is described, but in reality, information (for example, instances, etc.) indicating a plurality of operations and the order in which each operation is performed is registered. The Rukoto.

In the content database 33, data serving as reference content and data serving as comparison content are registered. For example, FIG. 10 is a diagram showing an example of information registered in the content database according to the embodiment. In the example shown in FIG. 10, information having items such as content ID, content type, acquisition source terminal, screen shot, and object information is registered in the content database 33 . In addition to the information shown in FIG. 10, the content database 33 may also register various types of information related to the reference content and the comparison content, such as application versions, time stamps such as the date and time of content acquisition, and the like.

Here, the content ID is an identifier of content. The content type is information indicating whether the content is reference content or comparison content. Also, the acquisition source terminal is information indicating the terminal device that acquired the screenshot of the content, that is, the terminal device that executes the application to be tested. A screenshot is data of a screenshot acquired as each content. Object information is information about various objects included in the content.

In the example shown in FIG. 10, conceptual values such as "SSBC1" and "OOBC1" are described, but in reality, various image data obtained as screenshots, object contents, types, appearances, etc. are registered in the content database 33.

Returning to FIG. 7, the description is continued. The control unit 40 is, for example, a controller, and various programs (an example of a notification program) stored in a storage device inside the information providing apparatus 10 are controlled by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. ) is executed using the RAM as a work area. Also, the control unit 40 is a controller, and is implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 7 , the control unit 40 has an identification unit 41 , a determination unit 42 , a generation unit 43 , an application unit 44 , an acquisition unit 45 , a determination unit 46 and a notification unit 47 .

The specifying unit 41 specifies the function that the user's operation target exhibits. For example, the identification unit 41 identifies the function that the operation target exhibits based on the screen displayed on the predetermined terminal device by the application to be the operation target. That is, the identifying unit 41 identifies the function provided via the screen. For example, the identifying unit 41 identifies the type of service provided by the operation target, that is, the domain. Then, the specifying unit 41 specifies, for each screen, the function provided by the operation target on each screen based on the specified domain and the object on each screen provided by the operation target.

The determining unit 42 determines a combination of user operations to be performed on the operation target according to the specified function. For example, the determining unit 42 refers to the operation database 31 that associates a function with a user's operation performed when an operation target is caused to exhibit the function, and determines whether the operation corresponding to the function specified by the specifying unit 41 is performed. Determine the combination based on

The generation unit 43 generates operation information expressing the determined combination of operations in a natural language. For example, the generation unit 43 generates operation information for each screen provided by the operation target. For example, the generation unit 43 acquires from the combination database 32 an operation combination associated with a function provided on a screen for which operation information is to be generated and an operation target domain. Subsequently, the generation unit 43 acquires from the operation database 31 natural language expressions indicating each operation included in each operation combination. Then, the generation unit 43 generates, for each operation combination, operation information indicating the operation content indicated by the operation combination in natural language according to the acquired natural language expression and the order of the operations included in the operation combination. . As a result of such processing, the information providing device 10 can automatically generate operation information that serves as a test case.

The application unit 44 performs the target test according to the operation information generated by the generation unit 43 . For example, when the application unit 44 acquires the operation information, the application unit 44 generates application information for each combination of the operation information and the target by executing the processing illustrated in FIG. By doing so, the test is realized. It should be noted that the application processing executed by the application unit 44 is assumed to be applicable to the various types of processing described above, and detailed description thereof will be omitted.

Note that, when executing the determination process described above, the application unit 44 uses each terminal device 200 as a comparison terminal, acquires a snapshot from the comparison terminal 200, and generates application information for transmitting the snapshot. Note that such processing may be realized based on the operation information generated by the generation unit 43, for example.

The acquisition unit 45 acquires reference content displayed when the reference terminal executes a predetermined application, and comparison content displayed when the comparison terminal to be compared executes the predetermined application. For example, the acquisition unit 45 installs an application to be tested on the reference terminal BT, displays all the content displayed by the application or a predetermined part of the content, and acquires a screen shot of each content. . Then, the acquiring unit 45 acquires the screen shot as the reference content from the reference terminal BT.

In addition, the acquisition unit 45 acquires a screen shot of the content displayed by the application from each terminal device 200 as comparison content. That is, the acquisition unit 45 acquires the comparison content displayed when the comparison terminal executes the same predetermined application as the reference terminal. Note that the acquisition unit 45 may acquire multiple screen shots when the reference terminal displays the reference content and multiple screen shots when the comparison terminal displays the comparison content. In other words, the acquiring unit 45 may acquire a screen shot when a part of one content is displayed as the reference content or the comparison content. Also, the acquisition unit 45 may acquire different numbers of screen shots for the same content as the reference content and the comparison content.

Also, the acquisition unit 45 identifies various objects from the screen shots of each content, extracts information about the identified objects as objects, and registers the screen shots and the objects in the content database 33 . Note that the acquisition unit 45 may calculate the feature amount of the appearance of each content and register it in the content database 33 .

The determination unit 46 determines whether or not the similarity between the feature of the reference content and the feature of the comparison content satisfies a predetermined condition. Here, if the determination unit 46 determines the similarity of each content to the extent that the user can recognize that the content provides the same service, based on an arbitrary criterion, each Content similarity may be determined. The criteria described below are merely examples, and the embodiments are not limited to these.

For example, the determination unit 46 determines whether or not the similarity between the feature of the appearance of the reference content and the feature of the appearance of the comparison content satisfies a predetermined condition. For example, the determining unit 46 determines the similarity of the appearance of each content based on various feature amounts and pixel-by-pixel comparison.

Further, the determination unit 46 may determine whether or not the similarity between the placement position of the object placed in the reference content and the placement position of the object placed in the comparison content satisfies a predetermined condition. . For example, the determination unit 46 may identify the positions where text, images, various functional objects, etc. are displayed on the screen, and determine the similarity of each content based on the similarity of the identified positions.

Further, the determination unit 46 determines that the similarity between the relative placement positions of the plurality of objects placed in the reference content and the relative placement positions of the plurality of objects placed in the comparison content is a predetermined value. It may be determined whether or not the conditions are satisfied. For example, the determination unit 46 determines the relative positional relationship between a predetermined text such as "password" and a predetermined functional object such as a text box for entering a password (for example, vertically aligned or horizontally aligned). location, direction of separation, distance of separation, etc.) may be specified for each content, and the similarity of each content may be determined according to whether or not the specified positional relationship is similar.

Further, the determination unit 46 determines whether or not the similarity between the order in which the plurality of objects are arranged in the reference content and the order in which the plurality of objects are arranged in the comparison content satisfies a predetermined condition. You may For example, the determination unit 46 determines the similarity of each content based on whether or not predetermined objects are arranged in a predetermined order in each content, and the similarity of the order in which the objects are arranged. good too.

Further, the determination unit 46 may determine whether or not the similarity between the appearance features of the characters included in the reference content and the appearance features of the characters included in the comparison content satisfies a predetermined condition. . For example, the determination unit 46 may determine the similarity of each content based on the similarity of text content, color, font size, itself, etc. in each content.

Further, the determining unit 46 determines the similarity between the features of the reference content and the features of the comparison content based on whether or not an object similar to the object arranged in the reference content is arranged in the comparison content. may determine whether satisfies a predetermined condition. For example, the determination unit 46 determines whether or not the same or similar objects (for example, buttons with slightly different display sizes but the same text is placed) are arranged in each content, and whether the same or similar objects are arranged. The similarity of each content may be determined according to the number arranged.

Further, the determining unit 46 determines that the similarity between the functional object that performs a predetermined function by the user's operation among the objects arranged in the reference content and the functional object arranged in the comparison content is determined to be a predetermined value. It may be determined whether or not the conditions are satisfied. Further, the determination unit 46 determines whether or not the similarity between the function exhibited by the functional object arranged in the reference content and the function exhibited by the functional object arranged in the comparative content satisfies a predetermined condition. You may For example, the determination unit 46 may determine that each content is similar when a functional object that exhibits a common function is commonly arranged in each content.

The determination unit 46 may also determine whether or not the similarity between the appearance of the functional object placed in the reference content and the appearance of the functional object placed in the comparison content satisfies a predetermined condition. . For example, the determination unit 46 determines the similarity of the appearance of the functional objects included in each content for each pixel or by comparing the feature amount, and determines the similarity of the appearance of the functional objects arranged in each content. , the similarity of each content may be determined.

Further, the determination unit 46 may determine whether or not the similarity between the attribute feature of the reference content and the attribute feature of the comparison content satisfies a predetermined condition. The determination unit 46 may also determine whether or not the similarity between the feature of the function of the reference content and the feature of the function of the comparison content satisfies a predetermined condition.

Note that the determination unit 46 calculates a score indicating similarity based on the various criteria described above. However, embodiments are not so limited. When the reference feature information indicating the feature of the reference content and the comparison feature information indicating the feature of the comparison content are input, the determination unit 46 determines whether the reference content and the comparison content are information set by the user. Whether the similarity between the feature of the reference content and the feature of the comparison content satisfies a predetermined condition using a model trained to output information indicating whether or not the content is similar to may be determined. That is, the determining unit 46 may determine the commonality between the reference content and the comparison content using a model that has learned the characteristics of the content determined by the user to be content that provides a common constituent service.

It should be noted that the above-described processing is executed with reference to information registered in the content database 33 . For example, the determination unit 46 refers to the content database 33 and acquires screenshots and object information of the reference content and the comparison content with respect to the reference content. Then, the determination unit 46 determines the similarity between the reference content and the comparison content based on the acquired information.

The notification unit 47 notifies the user of information indicating the relationship between the application executed by the reference terminal and the application executed by the comparison terminal according to the determination result. For example, the notification unit 47, among the comparison contents displayed on the comparison terminal by a certain application, when the ratio of contents determined to be similar to the reference content corresponding to each comparison content exceeds a predetermined threshold, the notification unit 47 The application executed by the comparison terminal and the application executed by the comparison terminal are guaranteed to be identical. In addition, the notification unit 47 may notify information about each content determined to be dissimilar as a result of the comparison.

[4. Processing procedure]
Next, the procedure of determination processing by the information providing apparatus 10 according to the embodiment will be described with reference to FIG. 11 . FIG. 11 is a flowchart illustrating an example of the flow of determination processing according to the embodiment.

For example, the information providing apparatus 10 determines whether or not an application to be determined has been received (step S201). If the information providing apparatus 10 has not received the application (step S201: No), it waits until it receives the application (step S201). On the other hand, when the application is accepted (step S201: Yes), the information providing apparatus 10 causes the reference terminal to execute the application and acquires the reference content (step S202). If the reference content has already been registered in a predetermined database, the information providing apparatus 10 may acquire it.

Further, the information providing apparatus 10 causes the comparison terminal to execute an application and acquires comparison content (step S203). The information providing apparatus 10 may perform steps S202 and S203 in any order.

Then, the information providing apparatus 10 determines whether or not the similarity between the reference content and the comparison content satisfies a predetermined condition (step S204), provides information according to the determination result (step S205), and performs processing. exit.

[5. Modification]
The information providing device 10 described above may be implemented in various different forms other than the above embodiment. Therefore, another embodiment of the information providing device 10 will be described below.

[5-1. About operation]
In the example described above, the information providing apparatus 10 compares the content displayed by the application. Here, the information providing apparatus 10 may compare the content displayed as a result of the operation automatically applied in the test by the application process generated by the generation process or the application process. For example, the information providing apparatus 10 acquires the content displayed when the user taps a certain button, pays with a QR code, or opens a menu as reference content or comparison content. , the similarity of the acquired reference content and comparison content may be determined.

In this way, when the display mode of the reference content or the comparison content changes due to various operations, the information providing apparatus 10 acquires the content before the change and the content after the change as separate reference content and comparison content, respectively. Similarity may be determined. Such processing can be realized, for example, by acquiring a screen shot each time an operation is performed and generating application information for acquiring the acquired screen shot as reference content or comparison content. Further, when performing such processing, the information providing apparatus 10, for example, selects the reference content and the comparison content as objects of comparison according to what kind of operation was performed when they were displayed. What is necessary is just to select a reference|standard content and a comparison content. That is, the information providing apparatus 10 may determine the similarity of the content displayed by the application when the same operation is performed on the reference terminal and comparison terminal.

It should be noted that the information providing apparatus 10, for example, post a content such as a predetermined character string or a photo on SNS, input to a game, or view a video, every time an operation is performed. Screenshots may be acquired as reference content or comparison content. In addition, the information providing apparatus 10 performs various actions dependent on hardware, such as changing the direction of the terminal device 200, shaking the terminal device 200, changing the volume, and inputting various operations by voice. Screen shots may be obtained as reference content or comparison content each time the operation of is performed.

In addition, the information providing device 10 may execute various types of processing described above for any application. For example, the information providing apparatus 10 may generate operation information and compare reference content and comparison content for applications that provide various payment services using the terminal device 200 . For example, a payment service reads a QR code (registered trademark) installed at a store, accepts input of a payment amount, and sends electronic money for the accepted payment amount to an account at the store corresponding to the QR code. There are services such as remittances from bank accounts. In addition, there is a payment service that displays a bar code that identifies a user's account or credit card, and a store terminal reads the displayed bar code, thereby realizing billing for the user.

The information providing apparatus 10 may generate operation information and compare reference content and comparison content for applications related to such various payment services. In this case, the information providing apparatus 10 compares the reference content and the comparison content, for example, when a predetermined remittance amount is input, when a QR code is read, when a barcode indicating the user's account is displayed, etc. may be used to determine whether an appropriate amount or information is displayed.

[5-2. About purpose]
Here, the information providing apparatus 10 may perform the determination process described above for any purpose other than testing the target application. For example, there is known a technology of RPA (Robotic Process Automation) in which software-type robots or AI (Artificial Intelligence) perform and automate various collaborative work. The information providing apparatus 10 may apply the determination process described above in such RPA technology. For example, the information providing apparatus 10 uses the operation results of the robot serving as the reference terminal (for example, a moving image of the appearance during operation) as reference content, and the operation results of the robot serving as the comparison terminal as comparison content. Then, the information providing apparatus 10 may determine whether the robot serving as the comparison terminal operates appropriately based on the identity of each content.

In addition, the information providing apparatus 10 may perform comparison processing of various software in order to test QAAS (Quality Assurance as a Service). For example, the information providing apparatus 10 may determine that the quality of the application executed on the comparison terminal satisfies a predetermined condition when the similarity between the reference content and the comparison content exceeds a predetermined threshold.

In this way, the information providing apparatus 10 uses not only the test to determine the similarity of the displayed images, but also whether the application operates properly and whether the quality of the application exceeds a predetermined threshold. You may Further, the information providing apparatus 10 may generate application information for any purpose other than the purpose described above as long as it generates operation information indicating an operation on a target.

[5-3. About processing mode]
Of the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, and conversely, all of the processes described as being manually performed can be performed manually. Alternatively, some can be done automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Moreover, each of the embodiments described above can be appropriately combined within a range in which the processing contents are not inconsistent.

[5-4. program〕
Also, the information providing apparatus 10 according to the above-described embodiment is implemented by a computer 1000 configured as shown in FIG. 12, for example. FIG. 12 is a diagram illustrating an example of a hardware configuration; A computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected via a bus 1090. have

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device such as a RAM that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used for various calculations by the arithmetic device 1030 and various databases are registered. It is realized by

The output IF 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various types of information such as a monitor and a printer. It is realized by a connector conforming to a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Also, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, scanner, etc., and is realized by, for example, USB.

Note that the input device 1020 includes, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), PDs (Phase change rewritable discs), magneto-optical recording media such as MOs (Magneto-Optical discs), and tapes. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Also, the input device 1020 may be an external storage medium such as a USB memory.

Network IF 1080 receives data from other devices via network N and sends the data to arithmetic device 1030, and also transmits data generated by arithmetic device 1030 via network N to other devices.

The arithmetic device 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070 . For example, arithmetic device 1030 loads a program from input device 1020 or secondary storage device 1050 onto primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 10 , the arithmetic device 1030 of the computer 1000 implements the functions of the control unit 40 by executing programs loaded on the primary storage device 1040 .

[6. effect〕
As described above, the information providing apparatus 10 displays the reference content displayed when the reference terminal executes the predetermined application, and the comparison content displayed when the comparison terminal to be compared executes the predetermined application. to get Then, the information providing apparatus 10 determines whether or not the similarity between the feature of the reference content and the feature of the comparison content satisfies a predetermined condition.

For example, the information providing apparatus 10 acquires comparison content displayed when the comparison terminal executes the same predetermined application as the reference terminal. As a result of such processing, the information providing apparatus 10 can determine the similarity of content displayed on different terminal devices by the same application. It can automatically determine whether Further, when the comparison terminal and the reference terminal are caused to execute different applications, the information providing apparatus 10 automatically determines whether the application executed by the comparison terminal can be regarded as the same as the application executed by the reference terminal. be able to. As a result of automating such a determination, the information providing apparatus 10 can automate testing of UIs provided by various applications and reduce labor.

Further, the information providing apparatus 10 determines whether or not the similarity between the feature of the appearance of the reference content and the feature of the appearance of the comparison content satisfies a predetermined condition. Further, the information providing apparatus 10 determines whether or not the similarity between the placement position of the object placed in the reference content and the placement position of the object placed in the comparison content satisfies a predetermined condition. , the information providing apparatus 10 can automatically determine whether the appearances are similar enough to allow the user to determine that the content provides the same service.

Further, the information providing apparatus 10 determines the similarity between the relative placement positions of the plurality of objects placed in the reference content and the relative placement positions of the plurality of objects placed in the comparison content. It is determined whether or not the conditions of are satisfied. Further, the information providing apparatus 10 determines whether or not the similarity between the order in which the plurality of objects are arranged in the reference content and the order in which the plurality of objects are arranged in the comparison content satisfies a predetermined condition. judge. Therefore, the information providing apparatus 10 can determine the similarity of each content according to the similarity given to the user by the relative positional relationship and order of the objects.

The information providing apparatus 10 also determines whether or not the similarity between the appearance features of the characters included in the reference content and the appearance features of the characters included in the comparison content satisfies a predetermined condition. Therefore, the information providing apparatus 10 can determine the similarity of each content according to the similarity given to the user by the appearance of characters such as fonts.

Further, the information providing apparatus 10 determines the similarity between the feature of the reference content and the feature of the comparison content based on whether or not an object similar to the object arranged in the reference content is arranged in the comparison content. It determines whether or not the property satisfies a predetermined condition. Further, the information providing apparatus 10 determines that a functional object, among the objects arranged in the reference content, that exerts a predetermined function in response to an operation by the user and a functional object arranged in the comparison content have a predetermined similarity. It is determined whether or not the conditions of are satisfied. Therefore, the information providing apparatus 10 can determine the similarity of each content according to the presence or absence of various types of content and the similarity.

Further, the information providing apparatus 10 determines whether or not the similarity between the function exhibited by the functional object arranged in the reference content and the function exhibited by the functional object arranged in the comparison content satisfies a predetermined condition. judge. Therefore, the information providing apparatus 10 can determine the similarity of each content based on the similarity of functions provided by each content.

Further, the information providing apparatus 10 determines whether or not the similarity between the appearance of the functional object arranged in the reference content and the appearance of the functional object arranged in the comparison content satisfies a predetermined condition. Therefore, the information providing apparatus 10 can determine the similarity of each content according to the similarity given to the user by the appearance.

Further, the information providing apparatus 10 determines whether or not the similarity between the attribute feature of the reference content and the attribute feature of the comparison content satisfies a predetermined condition. Further, the information providing apparatus 10 determines whether or not the similarity between the feature of the function of the reference content and the feature of the function of the comparison content satisfies a predetermined condition. Therefore, the information providing apparatus 10 can appropriately determine the similarity of each content and determine the similarity of each content.

Further, when the reference feature information indicating the feature of the reference content and the comparison feature information indicating the feature of the comparison content are input, the information providing apparatus 10 is the information set by the user, which is the reference. Using a model trained to output information indicating whether or not the content and the comparison content are similar, the similarity between the features of the reference content and the features of the comparison content satisfies a predetermined condition. Determine whether or not the conditions are met. Therefore, the information providing apparatus 10 can automatically determine whether or not the user determines that each content provides the same function.

Further, the information providing apparatus 10 acquires a screenshot of the reference content displayed by the reference terminal and a screenshot of the comparison content displayed by the comparison terminal. The information providing apparatus 10 also acquires multiple screen shots when the reference terminal displays the reference content and multiple screen shots when the comparison terminal displays the comparison content. In addition, the information providing apparatus 10 acquires the screen shots of the reference terminal and the screen shots of the comparison terminal, the number of which is different from that of the screen shots of the reference terminal. For this reason, the information providing apparatus 10 can automatically and appropriately check the identity of a single content even when the entire single content is displayed on the reference terminal or the comparison terminal, or when only a part of the content is displayed. can be determined.

Further, the information providing apparatus 10 notifies the user of information indicating the relationship between the application executed by the reference terminal and the application executed by the comparison terminal according to the determination result. Therefore, the information providing apparatus 10 can provide various types of information such as whether the application needs to be modified and whether the application is legitimate.

As described above, the embodiments of the present application have been described in detail with reference to the drawings, but these are examples, and various modifications and improvements, including the aspects described in the section of the disclosure of the invention, can be made based on the knowledge of those skilled in the art. It is possible to implement the invention in other forms.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the acquisition unit can be read as acquisition means or an acquisition circuit.

10 information providing device 20 communication unit 30 storage unit 31 operation database 32 combination database 33 content database 40 control unit 41 identification unit 42 determination unit 43 generation unit 44 acquisition unit 46 determination unit 47 notification unit 200 terminal device 300 operator terminal BT reference Terminal CS Cloud server

Claims (20)

an acquisition unit for acquiring reference content displayed when a reference terminal executes a predetermined application and comparison content displayed when a comparison terminal to be compared executes a predetermined application;
A determination device, comprising: a determination unit that determines whether or not a similarity between the feature of the reference content and the feature of the comparison content satisfies a predetermined condition. The determination device according to claim 1, wherein the acquisition unit acquires comparison content displayed when the comparison terminal executes the same predetermined application as that of the reference terminal. 3. The method according to claim 1, wherein the determining unit determines whether or not similarity between the feature of the appearance of the reference content and the feature of the appearance of the comparison content satisfies a predetermined condition. Determination device as described. The determination unit determines whether or not a similarity between an arrangement position of an object arranged in the reference content and an arrangement position of an object arranged in the comparison content satisfies a predetermined condition. 4. The determination device according to claim 3. The determination unit determines that the similarity between the relative placement positions of the plurality of objects placed in the reference content and the relative placement positions of the plurality of objects placed in the comparison content is predetermined. 5. The determination device according to claim 4, wherein it is determined whether or not a condition is satisfied. The determination unit determines whether or not similarity between an order in which the plurality of objects are arranged in the reference content and an order in which the plurality of objects are arranged in the comparison content satisfies a predetermined condition. 6. The determination device according to claim 5, characterized by: The determination unit determines whether or not similarity between appearance features of characters included in the reference content and appearance features of characters included in the comparison content satisfies a predetermined condition. The determination device according to any one of claims 3 to 6. The determination unit determines whether or not an object similar to an object arranged in the reference content is arranged in the comparison content, and determines the characteristics of the reference content and the characteristics of the comparison content. The determination device according to any one of claims 1 to 7, wherein it is determined whether or not the similarity satisfies a predetermined condition. The determination unit determines that a functional object, among the objects arranged in the reference content, that exerts a predetermined function in response to an operation by a user and a functional object arranged in the comparison content have a predetermined similarity. The determination device according to any one of claims 1 to 8, wherein it is determined whether or not a condition is satisfied. The determination unit determines whether a similarity between a function exhibited by the functional object arranged in the reference content and a function exhibited by the functional object arranged in the comparison content satisfies a predetermined condition. The determination device according to claim 9, characterized in that: The determination unit determines whether or not similarity between an appearance of the functional object arranged in the reference content and an appearance of the functional object arranged in the comparison content satisfies a predetermined condition. 11. The determination device according to claim 9 or 10. The determination unit determines whether or not a similarity between an attribute feature of the reference content and an attribute feature of the comparison content satisfies a predetermined condition. The determination device according to any one of them. The determination unit determines whether or not similarity between the feature of the function of the reference content and the feature of the function of the comparison content satisfies a predetermined condition. The determination device according to any one of them. When the reference feature information indicating the feature of the reference content and the comparison feature information indicating the feature of the comparison content are input, the determination unit compares the information set by the user with the reference content. The similarity between the feature of the reference content and the feature of the comparison content satisfies a predetermined condition using a model trained to output information indicating whether or not the content is similar to the content. The determination device according to any one of claims 1 to 13, characterized in that it determines whether or not. 15. The acquiring unit acquires a screenshot of the reference content displayed by the reference terminal and a screenshot of the comparison content displayed by the comparison terminal. The determination device according to 1. The acquisition unit acquires a plurality of screen shots when the reference terminal displays the reference content and a plurality of screen shots when the comparison terminal displays the comparison content. Item 16. The determination device according to item 15. 16. The acquiring unit according to claim 15, wherein the acquisition unit acquires screenshots of the reference terminal and screenshots of the comparison terminal, the number of which is different from that of the screenshots of the reference terminal. judgment device. 2. A notification unit for notifying a user of information indicating the relationship between the application executed by the reference terminal and the application executed by the comparison terminal according to the determination result of the determination unit. 18. The determination device according to any one of 17. A determination method executed by a determination device,
an acquisition step of acquiring reference content displayed when a reference terminal executes a predetermined application and comparison content displayed when a comparison terminal to be compared executes a predetermined application;
A determination method, comprising: determining whether or not a similarity between the feature of the reference content and the feature of the comparison content satisfies a predetermined condition. an acquisition procedure for acquiring reference content displayed when a reference terminal executes a predetermined application and comparison content displayed when a comparison terminal to be compared executes a predetermined application;
A determination program for causing a computer to execute a determination procedure for determining whether or not the similarity between the features of the reference content and the features of the comparison content satisfies a predetermined condition.

Images