KR102483475B1 - Method and device for automating process with image - Google Patents

Abstract

The present invention relates to a method for automating a process using an image, and more particularly, obtaining a first screen image representing a first screen being output from an output unit of a user terminal, starting work included in the first screen image Acquisition of an image, acquisition of an event associated with a work start image, acquisition of a second screen image representing a second screen being output after the first screen is output from an output unit of a user terminal, Determining whether the screen image includes the job start image, and executing an event associated with the job start image when it is determined that the second screen image includes the job start image.

Description

Process automation method and device using image {Method and device for automating process with image}

The present invention relates to a method for automating a process, and more particularly, to a method for automating a process using an image.

2. Description of the Related Art Computer technology is developing, and computers are becoming widespread. The majority of office work is performed using computers, and the number of tasks processed in repetitive forms or types using computers is also greatly increasing.

Accordingly, there is an increasing demand for an automatic processing system for quickly and efficiently processing tasks using a computer.

An example of a conventional automatic processing system is a macro program. A macro program binds a plurality of frequently used commands so that a plurality of commands are executed together with a single key input operation. The macro program can quickly and conveniently perform the same pattern of repetitive tasks by inputting only one specific command stored in the macro function without the need for an operator to input multiple commands each time when performing a task of a certain pattern that is repeatedly performed.

However, this conventional automatic processing system can be applied to tasks having simple patterns, and thus cannot be utilized for complex computer tasks.

In addition, the conventional automatic processing system has a problem in that it is difficult to utilize it in an actual work environment because the set automatic task processing process is not properly executed on a computer having an environment different from the computer in which the automatic task processing process is set.

In addition to this, the conventional automatic processing system requires the user to set specific details such as a specific command, execution order, execution location, method, etc. in order to set up an automatic task processing process that automatically performs task processing for performing predetermined repetitive tasks. do. The setting of these specific details had a problem in that it was difficult to set up an automatic business processing process for those who were not good at computers.

Republic of Korea Patent Publication No. 10-2020-0046769 Republic of Korea Patent Registration No. 10-2039422

Therefore, the present invention has been proposed to solve the above problems, and to set the automatic business processing process by setting the work start image and event information to process the specific details for setting the automatic business process. It is to provide a business process automation method to increase work convenience and efficiency.

Objects of the present invention are not limited to those mentioned above, and other objects not mentioned above will be clearly understood by those skilled in the art from the description below.

A process automation method using an image according to an embodiment of the present invention includes obtaining a first screen image representing a first screen being output from an output unit of a user terminal; obtaining a work start image included in the first screen image; obtaining an event associated with the work start image; obtaining a second screen image representing a second screen being output after the first screen is output from an output unit of the user terminal; determining whether the second screen image includes the work start image; and when it is determined that the second screen image includes the job start image, executing an event associated with the job start image.

The event information associated with the job start image includes information on a plurality of events and an execution order or execution time of each event, and the executing of the events includes the plurality of events according to the execution order or execution time of each event. run,

The step of determining whether the second screen image includes the work start image may include dividing the second screen image into a plurality of pieces by using an image segmentation filter that divides the second screen image into image segments having a predetermined resolution. segmenting the image into segments; and comparing each of the plurality of image segments with the job start image, and determining whether an image segment corresponding to the job start image exists among the plurality of image slices, and displaying the job start image in the second screen image. The step of determining whether an image is included determines that the second screen image includes the work start image if an image segment corresponding to the work start image exists, and if not, the second screen image includes the work start image. Determines that the start-up image is not included.

The step of determining whether there is an image segment corresponding to the job start image may include calculating a similarity between each image segment and the job start image; and comparing the calculated similarity and threshold similarity, and determining whether an image segment corresponding to the work start image exists based on the comparison result, wherein the image corresponding to the work start image exists based on the comparison result. In the step of determining whether a segment exists, if there exists an image segment whose calculated similarity is equal to or greater than the threshold similarity, it is determined that an image segment corresponding to the work start image exists among the plurality of image segments, and an image whose calculated similarity is equal to or greater than the threshold similarity If the slice does not exist, it is determined that the image slice corresponding to the job start image does not exist among the plurality of image slices.

In the process automation method according to an embodiment of the present invention, when it is determined that there is no image segment corresponding to the job start image in the step of determining whether an image slice corresponding to the job start image exists, the second screen image In the dividing into a plurality of image segments, a predetermined resolution of the image segment divided by the image segmentation filter is changed and the second screen image is divided into a plurality of image segments having the changed resolution.

In the process automation method according to an embodiment of the present invention, the step of dividing the second screen image into a plurality of image segments and the step of determining whether the image segments exist are when an image segment corresponding to the work start image is found. It is performed repeatedly up to or as many times as the number of times set in advance.

The executing of the event may include calculating a position of an image segment corresponding to the work start image; moving a mouse cursor to the calculated position; and executing an event associated with the work start image at the location where the mouse cursor is moved.

A user terminal performing a process automation method using an image according to another embodiment of the present invention includes an input unit for obtaining data input by the user terminal; an output unit outputting data to a user of the user terminal; A first screen image representing a first screen being output from the output unit of the user terminal is obtained, a work start image included in the first screen image is acquired, and the first screen is displayed from the output unit of the user terminal. an image acquisition unit acquiring a second screen image representing a second screen being output after being output; an event acquisition unit acquiring an event associated with the work start image; an image analysis unit determining whether the second screen image includes the work start image; and a processor executing an event associated with the job start image when it is determined that the second screen image includes the job start image.

A method of automating a process using an image sets a work start image and at least one event related thereto in relation to a user's computer work, and when the work start image is retrieved from a screen being executed in a user terminal, at least one work start image related to the work start image The event of can be automatically executed and processed. When a job start image set by the user is detected, even if the user does not operate the computer, at least one event related to the job start image is automatically executed and processed, thereby increasing the convenience and efficiency of the user's computer work. there is.

In addition, according to the present invention, when searching for a job start image set by a user when setting up process automation, it is searched whether the job start image is included in the currently displayed screen while changing the search ratio. The present invention extracts image fragments with various resolutions when searching for whether or not a preset work start image is included in a screen currently being output, and compares the extracted image fragments with the work start image, thereby increasing the detection accuracy of the work start image. can improve

In addition, the present invention sets a work start image by utilizing an image included in a screen output to an output unit of a user terminal. For example, a user can set a specific image as a work start image by dragging the mouse. As such, the present invention can set a work start image with a simple manipulation when setting process automation. Accordingly, even a person without expert knowledge of computers or process automation can easily set up a process automation method.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a block diagram of a user terminal that is a process automation device utilizing an image according to an embodiment of the present invention.
2 is a flowchart of a process automation method using an image according to an embodiment of the present invention.
3 is a diagram showing an example screen for setting a work start image.
FIG. 4 is a detailed flowchart of a step of determining whether a second screen image includes a work start image in the process automation method using image data shown in FIG. 2 .
5 is a diagram illustrating a method of comparing images according to an embodiment of the present invention.
FIG. 6 is a detailed flowchart of a step of executing an event associated with the work start image shown in FIG. 3 .

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

First, terms used in the present specification and claims are general terms in consideration of functions in various embodiments of the present invention. However, these terms may vary depending on the intention of a technician working in the field, legal or technical interpretation, and the emergence of new technologies. In addition, some terms may be terms arbitrarily selected by the applicant. These terms may be interpreted as the meanings defined in this specification, and if there is no specific term definition, they may be interpreted based on the overall content of this specification and common technical knowledge in the art.

In addition, the same reference numerals or numerals in each drawing attached to this specification indicate parts or components that perform substantially the same function. For convenience of explanation and understanding, the same reference numerals or symbols will be used in different embodiments. That is, even if all components having the same reference numerals are shown in a plurality of drawings, the plurality of drawings do not mean one embodiment.

Also, in the present specification and claims, terms including ordinal numbers such as 'first' and 'second' may be used to distinguish between elements. These ordinal numbers are used to distinguish the same or similar components from each other, and the meaning of the term should not be limitedly interpreted due to the use of these ordinal numbers. For example, components combined with such ordinal numbers should not be construed as limiting the order of use or arrangement by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as 'comprise' or 'comprise' are intended to designate that there is a characteristic, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that it does not preclude the possibility of the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof.

Also, in an embodiment of the present invention, when a part is said to be connected to another part, this includes not only a direct connection but also an indirect connection through another medium. In addition, the meaning that a certain part includes a certain component means that it may further include other components rather than excluding other components unless otherwise specified.

Terms described in the detailed description of the embodiments of the present invention below have the following meanings. "Process" means a program or task being executed in a device such as a user terminal. "Event" means any action or event executed in a user terminal, such as mouse input or command input. can be cited as an example. Those of ordinary skill in the art to which the embodiments of the present invention pertain can easily understand the above terms.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

1 is a block diagram of a user terminal that is a process automation device utilizing an image according to an embodiment of the present invention.

Referring to FIG. 1, a user terminal 1000 includes a processor 1100, a storage 1200, an input unit 1300, an output unit 1400, an image acquisition unit 1500, an image analysis unit 1600, and event acquisition. It consists of section 1700.

The processor 1100 is hardware including processing and processing functions for decoding and executing commands in the user terminal 1000 . The processor 1100 is a hardware unit that executes a program in a computer in terms of hardware, and may be a central processing unit (CPU) as a representative example. The processor 1100 processes general tasks performed by the user terminal 1000 .

The storage 1200 is a storage for storing data in an electromagnetic form, and stores data necessary for performing a process automation method using an image in the user terminal 1000 . Data stored in the storage 1200 may include process information, work start images, and related event data.

The input unit 1300 is a component for the user terminal 1000 to obtain data input from a user. Representative examples of the input unit 1300 include a keyboard, a mouse, and a touch screen. A user of the user terminal 1000 may input a command or the like through the input unit 1300 . Data that can be input through the input unit 1300 may be a work start image or data related to an event related to the work start image.

The output unit 1400 is a component for outputting data to the user of the user terminal 1000, and examples thereof include a monitor, a printer, and a speaker. Data that can be output from the output unit 1400 may include a screen representing a process being performed in the user terminal 1000, a work start image, and an event related to the work start image.

The image acquisition unit 1500 is a component for acquiring an image representing a screen currently being displayed on the output unit 1400 of the user terminal 1000 . The image acquisition unit 1500 captures the screen currently being output from the output unit 1400 and generates a screen image representing the screen currently being output. The screen image is an image representing a screen showing a process performed by the user terminal 1000 . Also, the image acquisition unit 1500 acquires work start image data indicating a work start image from the user. The user may input an image to be designated as a work start image into the user terminal 1000 using the input unit 1300 . The image acquisition unit 1500 acquires a work start image input from a user.

The image analysis unit 1600 is a component for analyzing the screen image and the work start image acquired by the image acquisition unit 1500 . More specifically, the image analysis unit 1600 compares the screen image and the work start image, and determines whether the screen image includes the work start image based on the comparison result. The image analysis unit 1600 includes an image division unit 1601, a similarity calculation unit 1602, a comparison unit 1603, and a filter adjustment unit 1064.

The image segmentation unit 1601 divides a screen image into a plurality of image segments having a predetermined resolution using an image segmentation filter. The image segmentation filter divides the screen image into image segments having a predetermined resolution. The image segmentation unit 1601 divides a screen image into a plurality of image segments having a predetermined resolution of an image segmentation filter. Here, the resolution of the image segmentation filter is varied by the user or by the processor 1100 of the user terminal 1000.

The similarity calculator 1602 compares each of the plurality of image segments divided by the image segmentation unit 1601 with the work start image, and calculates a similarity between each image slice and the work start image. Here, the degree of similarity means the degree of similarity between the two images. The similarity calculation unit 1602 calculates the similarity between the two images as a number between 0 and 100. When the similarity between the two images is '0', it means that the two images have no similar parts, and when the similarity between the two images is '100', it means that the two images completely match. The similarity calculation unit 1602 may be an artificial neural network trained in advance to calculate a similarity between two images when two images are input.

The comparison unit 1603 compares the similarity calculated by the similarity calculation unit 1602 with a threshold similarity previously set by the user. The comparator 1603 compares the similarity and threshold similarity between the image segment and the work start image, and determines whether the calculated similarity is greater than or less than the threshold similarity. If the calculated similarity is equal to or greater than the threshold similarity, the comparator 1603 determines that the image segment having the corresponding similarity corresponds to the work start image.

The filter adjusting unit 1064 adjusts the resolution of the image segmentation filter. If there is no image segment corresponding to the work start image among the plurality of image segments divided from the screen image, the filter adjusting unit 1604 adjusts the resolution of the image segmentation filter. Here, the filter adjustment unit 1604 changes the resolution of the image segmentation filter to have a different resolution from the previous image segmentation resolution. The filter adjustment unit 1604 allows the image segmentation unit 1601 to divide the screen image into image segments having various resolutions by changing the resolution of the image segmentation filter.

The event acquisition unit 1700 is a component for acquiring information about an event associated with a work start image input from a user. More specifically, the user may input an event related to the work start image input through the input unit 1300 of the user terminal 1000, and the event information includes mouse cursor movement, mouse cursor operation, text input, and the like. do.

In the user terminal 1000 according to embodiments of the present invention, the image acquisition unit 1500, the image analysis unit 1600, and the event acquisition unit 1700 may be implemented by a processor 1100 and a separate dedicated processor. It may be implemented by executing a computer program executed by the processor 1100.

2 is a flowchart of a process automation method using an image according to an embodiment of the present invention.

Referring to FIG. 2 , in step S1010 , the user terminal 1000 acquires a first image representing a screen being output from the output unit 1400 of the user terminal 1000 . More specifically, the image acquisition unit 1500 of the user terminal 1000 captures a screen being output from a display, which is the output unit 1400 of the user terminal 1000, and displays a first screen currently being output from the display. Acquire first screen image data representing the first screen image. The image data is data representing a screen currently being output by the output unit 1400 of the user terminal 1000 . For example, when a web browser is running on a computer monitor, image data is data on the screen of the web browser. The image data includes the number of pixels constituting the image and the color of each pixel. The image acquisition unit 1500 acquires an image obtained by capturing a screen being output from the output unit 1400 of the user terminal 1000 .

In step S 1020, the user terminal 1000 obtains a work start image from the user of the user terminal 1000. More specifically, the image acquiring unit 1500 of the user terminal 1000 acquires job start image data indicating a job start image input from the user through the input unit 1300 . A user of the user terminal 1000 may set a work start image through the input unit 1300 .

In this regard, FIG. 3 of the present invention is a diagram showing an example screen for setting a work start image. Referring to FIG. 3 , a user can set a work start image using a mouse, which is an input unit 1300. According to the example shown in FIG. 3, a method of dragging an image part to be set as a work start image with a mouse cursor You can set the job start image with .

The image acquisition unit 1500 acquires job start image data indicating a job start image input by a user. The work start image data includes the size of the work start image set by the user, the position of the work start image (eg, coordinates on the screen of the output unit), the number of pixels included in the work start image, and the color of each pixel. The image acquisition unit 1500 stores the acquired work start image data in the storage 1200 .

In step S1030, the user terminal 1000 obtains an event associated with the work start image input from the user. The event acquisition unit 1700 of the user terminal 1000 acquires event data for an event related to a job start image input from the user through the input unit 1300 . More specifically, the user may input an event related to the work start image input in step S1020 through the input unit 1300 of the user terminal 1000 . Here, the associated event is a task executed in the user terminal 1000 in association with the work start image, and includes movement of the mouse cursor, operation of the mouse cursor, and the like. The associated event data refers to data representing an event associated with a job start image input by a user.

In step S1030, the event associated with the work start image may be a plurality of events. The user may set a plurality of events related to the work start image through the input unit 1300 of the user terminal 1000 . A user may input an execution sequence or execution time for each of a plurality of events. A plurality of events are executed in the user terminal 1000 according to the execution order between each event or according to the execution time of each event when a job start image is retrieved from a process running in the user terminal 1000 .

When a job start image is included in any process running in the user terminal 1000 , an event associated with the job start image is automatically executed in the user terminal 1000 . For example, it is assumed that an image representing a notification of a mobile messenger program is set as a job start image, and a task for executing a mobile messenger is set as an event related to the job start image. In this example, when a process running on the user terminal 1000 includes an image representing a notification of a mobile messenger program (ie, when a notification of a mobile messenger program is executed), the user inputs a command to execute the mobile messenger. Even if not, the processor 1100 of the user terminal 1000 moves the position of the mouse cursor in response to the notification of the mobile messenger and executes an event to execute the image indicated by the mouse cursor (ie, the mobile messenger). The user terminal 1000 stores the work start image and related event information in the storage 1200 .

In the example shown in FIG. 3 , the user inputs an image representing a fingerprint set as a work start image (image portion divided by a square box in FIG. 3 ) as a work start image using an input unit 1300 such as a keyboard and a mouse. . When the process running on the user terminal 1000 includes the same image as the job start image, the user terminal 1000 moves the mouse cursor to the same image as the job start image, and the image representing the fingerprint, which is a related event. Executes an event that enlarges a part.

In step S1040, the user terminal 1000 acquires a second screen image representing a second screen being output after the first screen is output from the output unit 1400 of the user terminal 1000. More specifically, The image acquisition unit 1500 of the user terminal 1000 captures a screen currently output from a display, which is the output unit 1400 of the user terminal 1000, and displays a second screen displaying a second screen currently output from the display. Acquire second screen image data representing an image. The image acquisition unit 1500 inputs the obtained second screen image data to the image analysis unit 1600 .

In step S1050, the user terminal 1000 determines whether a work start image is included in the second screen image. The image analyzer 1600 of the user terminal 1000 determines whether a work start image is included in the second image representing the screen being output from the output unit 1400 of the user terminal 1000 . More specifically, the image analysis unit 1600 compares the second image and the work start image. The image analyzer 1600 determines whether the second screen image includes the same image as the image indicated by the work start image data. If the image analysis unit 1600 determines that the second screen image includes the work start image, it proceeds to step S1060, otherwise it proceeds to step S1040. A detailed process for the user terminal 1000 to determine whether or not the second screen image includes an operation start image will be described below with reference to FIG. 4 .

4 is a detailed flowchart of a step of determining whether a second screen image includes a work start image in the process automation method using the image shown in FIG. 2 . Referring to FIG. 4 , in step S1051, the image analysis unit 1600 of the user terminal 1000 uses an image segmentation filter to divide the second screen image of the second screen image data into image segments having a predetermined resolution. The second screen image is divided into image segments having a predetermined resolution representing a portion of the second screen image. The image division unit 1601 of the image analysis unit 1600 divides the second screen image into a plurality of image segments having a predetermined resolution according to a predetermined resolution of the image segmentation filter, and outputs the plurality of divided image segments. Each of the plurality of image segments has a predetermined resolution of the image segmentation filter.

Here, among the plurality of image segments, the similarity of each image segment adjacent to each other is greater than or equal to a predetermined threshold similarity. For example, the threshold similarity of adjacent image slices may be 90 points (90% of all pixels are the same). The images represented by each of the adjacent image slices are substantially similar. The image represented by the first image slice and the image represented by the second image slice overlap each other except for the border portion.

For example, if the second image slice is an image slice immediately adjacent to the right of the first image slice, the first image slice and the second image slice differ only in the left or right outermost pixel column, and the rest of the pixels are identical to each other. have In other words, the images of the first image slice and the second image slice overlap each other.

Here, the threshold similarity of the similarity of the image segments may be preset by a designer to 90 points, 80 points, 70 points, or the like. If the similarity score is 0, it means that the two images do not have identical parts, and if the similarity score is 100, it means that the two images completely match.

In step S1052, the image analyzer 1600 of the user terminal 1000 compares each of the plurality of extracted image segments with the work start image, and determines whether an image corresponding to the work start image exists among the plurality of image slices. The image analyzer 1600 determines that the second screen image includes the job start image when an image segment corresponding to the job start image exists, and if not, determines that the job start image is not included in the second screen image. Decide. A method of determining whether an image segment corresponding to the work start image exists will be described in detail below.

More specifically, the similarity calculation unit 1602 of the image analysis unit 1600 compares each image segment with the job start image, and calculates a similarity between each image segment and the job start image.

Here, the similarity calculation unit 1602 is an artificial neural network pre-trained to calculate the similarity between both images. The similarity calculation unit 1602 is an artificial neural network previously trained by a learning data set including a plurality of images and a similarity between the images. When two images are input, the similarity calculator 1602 according to the present invention calculates and outputs the similarity of each image.

An artificial neural network includes a fully convolutional network (FCN) in which the entire artificial neural network is composed of convolutional operations, a multi-layer perceptron including several perceptrons, and a convolutional neural network (CNN) including multiple convolution layers. network), a recurrent neural network (RNN) having a circular structure, and the like.

The similarity calculation unit 1602 inputs the calculated similarity to the comparison unit 1603.

The comparison unit 1603 of the image analysis unit 1600 compares the calculated similarity and threshold similarity, and determines whether an image segment corresponding to the work start image exists based on the comparison result. More specifically, the comparator 1603 determines whether an image exceeding a threshold similarity exists among the calculated similarities. The comparison unit 1603 of the image analysis unit 1600 compares the threshold similarity among the similarities between each calculated image segment and the work start image. The comparator 1603 determines that an image fragment corresponding to the work start image exists among a plurality of image fragments when there exists an image fragment whose similarity calculated as a comparison result is equal to or greater than the threshold similarity, and the image fragment whose calculated similarity is equal to or greater than the critical similarity If it does not exist, it is determined that there is no image segment corresponding to the work start image among the plurality of image segments. The comparator 1603 proceeds to step S1060 if there is an image segment having a similarity or higher as a result of the comparison, and to step S1053 if not.

In step S1053, the image analyzer 1600 determines whether the number of times of performing step S1051 of dividing the second screen image into a plurality of image segments exceeds a limit number of executions. The image analyzer 1600 determines whether the number of times of performing the dividing into image segments exceeds a limit number of times preset by the user. The image analyzer 1600 determines whether the second screen image is divided into image segments having different resolutions for one second screen image. If the number of times the segmenting of the image segment is performed exceeds the limit number of times, step S1040 is performed, and otherwise, step S1054 is performed.

In step S1054, the image analysis unit 1600 changes the resolution of an image segmentation filter that divides image segments from the second screen image data. The filter adjustment unit 1604 of the image analysis unit 1600 changes the resolution of the image segmentation filter when there is no image segment corresponding to the work start image among the image segments extracted from the second screen image data. The filter adjustment unit 1604 inputs the filter whose resolution is changed to the image segmentation unit 1701 .

In step S1051, the image segmentation unit 1601 of the image analysis unit 1600 divides the second screen image into image segments having an adjusted resolution by using an image segmentation filter whose resolution is adjusted. The image segmentation unit 1601 divides the second screen image into image segments having resolutions different from those of the previous image segment using an image segmentation filter whose resolution is adjusted. The image division unit 1601 outputs a plurality of divided image segments.

The user terminal 1000, particularly the image analyzer 1600, re-performs steps S1052, S1053, and S1054 on a plurality of image segments having resolutions different from those of the previous image segment.

Summarizing the above information, the process automation method according to the embodiments of the present invention repeats steps S1051 to S1054. The image analyzer 1600 divides the second screen image into a plurality of image segments having the first resolution using the first image segmentation filter, and compares each of the image segments having the first resolution with the work start image. As a result of the comparison, when there is no image segment corresponding to the work start image, the image analysis unit 1600 adjusts the resolution of the first image segmentation filter to convert the second screen image into a plurality of second screen images having a second resolution different from the first resolution. Segment the image into slices. The image analyzer 1600 compares the image fragment having the second resolution with the work start image, and determines whether an image fragment corresponding to the work start image exists. A process automation method according to embodiments of the present invention divides a second screen image into image segments having a predetermined resolution, and if there is no image segment corresponding to a work start image among the divided image segments, the second screen image is generated. is divided into image fragments having different resolutions, and the divided image fragments are compared with the work start image. As such, the present invention can improve the accuracy of detecting the work start image by dividing the second screen image into image segments having different resolutions and comparing the divided image segments with the work start image.

5 is a diagram illustrating a method of comparing images according to an embodiment of the present invention. In FIG. 5, "A" is a screen that is currently being output from the output unit 1400 of the user terminal 1000, that is, a second screen image, and "B" is a work start image included in the second image. Referring to FIG. 5 , as described above, the screen being output from the output unit 1400 of the user terminal 1000 is divided into image segments (rectangles with dotted lines in FIG. 5 ) having a predetermined resolution. The image analyzer 1600 compares each of the divided image segments with a work start image stored in the storage 1200 . As a result of comparing the image fragment and the work start image, if an image fragment having a similarity equal to or higher than a threshold similarity previously set by a designer is not found, the image analysis unit 1600 changes the resolution of the image fragment divided from the second screen image. The image analyzer 1600 compares the image fragments divided at the changed resolution with the work start image, and detects an image fragment having a similarity with the work start image equal to or greater than a threshold similarity.

In the process automation method according to an embodiment of the present invention, the user terminal 1000 repeats steps S1051 to S1054 until an image segment corresponding to the work start image is detected or by a limit number of times preset by the user. . Here, the limit number of executions is the maximum number of times steps S1051 to S1054 are performed when an image segment corresponding to the work start image is not detected. When the second screen image does not include an image corresponding to the work start image, the user terminal 1000 adjusts the resolution of the image segmentation filter by the limit number of executions preset by the user to obtain an image segment from the second screen image. Extract, and compare the extracted image fragment with the work start image.

As a result of dividing and comparing the second screen image into image segments by the limit number of executions, if there is no image segment corresponding to the work start image, the user terminal 1000 does not include the work start image in the second screen image. decide not to In other words, the user terminal 1000 determines that the work start image is not displayed on the screen being output from the output unit 1400 of the user terminal 1000 .

If there is an image segment corresponding to the work start image among the image fragments divided from the second screen image before reaching the limit number of executions, the user terminal 1000 determines that the work start image is included in the second screen image. and stop repeating steps S1051 to S1054.

In step S1060, the user terminal 1000 executes an event associated with the work start image. In step S1050, when the user terminal 1000 detects an image fragment similar to the work start image obtained in step S1030 from the second screen image, which is a screen currently being output from the output unit 1400 of the user terminal 100, the user The processor 1100 of the terminal 1000 reads event information associated with the job start image stored in the storage 1200 and executes an event indicated by the associated event information.

Step S1060 of executing an event associated with the work start image consists of the following steps. FIG. 6 is a detailed flowchart of a step of executing an event associated with the work start image shown in FIG. 3 . Referring to FIG. 6 , in step S1061, the user terminal 1000 calculates the position of the image segment corresponding to the work start image detected in the second screen image. More specifically, the image analyzer 1600 of the user terminal 1000 selects an image segment similar to the work start image obtained in step S1030 among the image fragments divided from the second screen image (ie, the similarity with the work start image is critical). The location of the image segment with similarity or higher) is calculated. The image analyzer 1600 of the user terminal 1000 calculates the position of the image slice similar to the work start image by calculating the coordinates of the position where the image slice is placed on the screen currently being displayed.

In step S1062, the user terminal 1000 moves the position of the mouse cursor to the position of the image segment calculated in step S1061. In detail, the processor 1100 of the user terminal 1000 moves the position of the mouse cursor to the position of the image segment whose similarity with the work start image is greater than or equal to the threshold similarity.

In step S1063, the user terminal 1000 executes an event associated with the work start image at the location where the mouse cursor is moved. The processor 1100 of the user terminal 1000 executes the event obtained in step S1040 at the mouse cursor moved to the position of the image segment similar to the work start image.

According to the example shown in FIG. 5 , the user terminal 1000 determines whether the same image as the “B” image set as the work start image is included on the screen being displayed on the user terminal 1000 . When an image similar to the "B" image is found, the processor 1100 of the user terminal 1000 moves the image "B" to the location of the currently displayed screen, and the same event as double-clicking the left button of the mouse cursor is moved. Executes an event that executes a program associated with the in "B" image.

According to the above-described embodiments of the present invention, the method for automating a process using an image sets a work start image and at least one event related thereto in relation to a user's computer work, and sets the work start image to a screen on which the user terminal is performing. If searched in , at least one event related to the work start image can be automatically executed and processed. When a job start image set by the user is detected, even if the user does not operate the computer, at least one event related to the job start image is automatically executed and processed, thereby increasing the convenience and efficiency of the user's computer work. there is.

In addition, according to the present invention, when searching for a job start image set by a user when setting up process automation, it is searched whether the job start image is included in the currently displayed screen while changing the search ratio. The present invention extracts image fragments with various resolutions when searching for whether or not a preset work start image is included in a screen currently being output, and compares the extracted image fragments with the work start image, thereby increasing the detection accuracy of the work start image. can improve

In addition, the present invention sets a work start image by utilizing an image included in a screen output to an output unit of a user terminal. For example, a user can set a specific image as a work start image by dragging the mouse. As described above, when setting process automation according to the present invention, a user can set a work start image with a simple operation. Accordingly, even a person without expert computer or process automation knowledge can easily set a process automation method.

Meanwhile, although preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. Of course, various modifications are possible by those skilled in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

So far, the present invention has been looked at with respect to preferred embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

1000: user terminal
1100: processor 1200: storage
1300: input unit 1400: output unit
1500: image acquisition unit 1600: image analysis unit
1700: event acquisition unit

Claims (9)

In the process automation method using images,
obtaining, by an image acquisition unit of a user terminal, a first screen image representing a first screen being output from an output unit of the user terminal;
obtaining, by the image acquiring unit of the user terminal, a work start image input from a user through an input unit of the user terminal, wherein the first screen image includes the work start image;
Acquiring, in the event acquisition unit of the user terminal, an event associated with the work start image input from the user through the input unit of the user terminal - the event is any operation executed in the user terminal, and the position of the mouse cursor including an event that moves the , an event that runs an arbitrary program;
acquiring a second screen image representing a second screen being output after the first screen is output from an output unit of the user terminal, by an image obtaining unit of the user terminal;
In an image analysis unit of the user terminal, determining whether the second screen image includes the work start image based on whether the same image as the work start image is included in the second screen image; and
When it is determined that the second screen image includes the work start image, executing an event associated with the work start image in a processor of the user terminal ,
The step of determining whether the second screen image includes the work start image
dividing the second screen image into a plurality of image segments using an image segmentation filter that divides the second screen image into image segments having a predetermined resolution; and
Comparing each of the plurality of image segments with the job start image, and determining whether an image segment corresponding to the job start image exists among the plurality of image slices;
In the step of determining whether the second screen image includes the job start image, if an image segment corresponding to the job start image exists, it is determined that the job start image is included in the second screen image; If not, it is determined that the second screen image does not include the work start image,
In the step of determining whether there is an image segment corresponding to the work start image, if it is determined that there is no image segment corresponding to the work start image, the dividing of the second screen image into a plurality of image segments may include dividing the image into a plurality of image segments. Changing a predetermined resolution of an image segment divided by a filter and dividing the second screen image into a plurality of image segments having the changed resolution;
The step of comparing each of the plurality of image segments with the job start image and determining whether there is an image segment corresponding to the job start image among the plurality of image segments is to compare each of the plurality of image segments having the changed resolution to the job start image. A process automation method using an image, characterized in that comparing with the starting image and determining whether an image fragment corresponding to the work start image exists among the plurality of image fragments having the changed resolution. According to claim 1,
The event information associated with the work start image includes information about a plurality of events and an execution order or execution time of each event,
The step of executing the event is a process automation method using an image, characterized in that performing the plurality of events according to the execution order or execution time of each event. delete According to claim 1,
The step of determining whether an image segment corresponding to the work start image exists
Calculating a degree of similarity between each image segment and the work start image - The similarity is a numerical value indicating the degree of similarity between each image slice and the work start image. If the similarity is 0, it means that both images are different images, and the similarity If is 100, it means that both images are identical images -; and
Comparing the calculated similarity and threshold similarity, and determining whether an image segment corresponding to the work start image exists based on the comparison result,
The step of determining whether there is an image segment corresponding to the job start image based on the comparison result is, if an image segment having a calculated similarity equal to or greater than a threshold similarity exists, an image segment corresponding to the job start image among the plurality of image segments is present, and if there is no image segment whose calculated similarity is equal to or greater than a threshold similarity, it is determined that there is no image segment corresponding to the work start image among the plurality of image segments. automation method. delete According to claim 1,
The step of dividing the second screen image into a plurality of image segments and the step of determining whether the image segments exist are repeatedly performed until an image segment corresponding to the work start image is found or by a preset limit number of executions. Process automation method using images characterized in that According to claim 1,
Steps to trigger the event
calculating a position of an image segment corresponding to the work start image;
moving a mouse cursor to the calculated position; and
Process automation method using an image comprising the step of executing an event associated with the work start image at the location where the mouse cursor is moved. delete Claim 1, claim 2, claim 4, claim 6 or claim 7, a computer-readable recording medium on which a program for performing the method according to any one of claims is recorded.

Images