JP7386007B2 - Image processing method, image processing device, and image processing equipment - Google Patents

Description

The present invention relates to the field of image processing, and in particular to an image processing method, an image processing apparatus, an image processing device, and a computer-readable storage medium.

Deep neural networks are large-scale, multi-parameter optimized tools. Deep neural networks rely on large amounts of training data, and through learning, they can determine hidden features in the data that are difficult to summarize, thereby performing multiple complex tasks such as face detection, image semantic segmentation, and object segmentation. It can complete detection, motion tracking, natural language translation, etc.

Deep neural networks are currently being used for complex tasks, such as detecting cracks in the surface of objects, and are expected to achieve higher detection accuracy than the human eye. In order to obtain a neural network model for crack detection, it is necessary to train the neural network using a large number of training images (ie, training data) in which cracks are marked. Therefore, the performance and effectiveness of crack detection based on deep neural networks is highly dependent on the provided training data. In conventional techniques, training data is often obtained by many workers manually marking cracks in images. However, theoretical cognitive or visual errors may exist in training data obtained by manually marking images, both of which may result in inaccuracies in the obtained training data. There is. Also, acquiring large amounts of training data may require a longer time for the operator. It causes a large mental or visual burden on workers, makes incorrect matching more likely, and is not suitable for industrial production.

The present invention has been proposed in view of the above problems. The present invention provides an image processing method, an image processing device, an image processing device, and a computer-readable storage medium.

According to one aspect of the present invention, the steps include: receiving an operation input; determining a target range from an image to be processed based on the operation input; An image processing method is provided that includes: generating a cutout image. Note that the size of the target range is in a predetermined proportion to the size of the cut-out image.

According to one aspect of the present invention, an operation input receiving unit is arranged to receive an operation input, and a target range determining unit is arranged to determine a target range from an image to be processed based on the operation input. and a cut-out image generation unit arranged to generate a predetermined number of cut-out images of a predetermined size within the target range. Note that the size of the target range is in a predetermined proportion to the size of the cut-out image.

According to one aspect of the present disclosure, an image processing apparatus includes a memory arranged to store computer readable instructions and a processor arranged to execute the computer readable instructions stored in the memory. provided. Additionally, when the processor executes the computer readable instructions, it performs the following steps. That is, the steps are: accepting an operational input, determining a target range from an image to be processed based on the operational input, and generating a predetermined number of cut-out images of a predetermined size in the target range. Note that the size of the target range is in a predetermined proportion to the size of the cut-out image.

According to one aspect of the present disclosure, a computer-readable storage medium having computer-readable instructions stored thereon is provided, and when the computer-readable instructions are executed by the computer, the computer performs the following steps. That is, the steps are: accepting an operational input, determining a target range from an image to be processed based on the operational input, and generating a predetermined number of cut-out images of a predetermined size in the target range. Note that the size of the target range is in a predetermined proportion to the size of the cut-out image.

According to the above aspect of the present invention, the user can automatically and accurately generate a cutout image by simply performing a simple operation input, and thereby the training image (training data) marked in the deep neural network can be generated automatically and accurately. ), it is also possible to reduce theoretical cognitive errors and visual errors, and reduce the cost of manual marking.

The above and other objects, features, and advantages of the present disclosure will become more apparent by describing embodiments of the present disclosure in more detail with reference to the accompanying drawings. The drawings provide a further understanding of the embodiments of the disclosure, constitute a part of the specification, and together with the embodiments of the disclosure, illustrate the disclosure and do not limit the disclosure. . In the drawings, like numbers generally indicate like elements or steps.
1 is a flowchart of an image processing method for implementing an embodiment of the present invention. FIG. 6 is a diagram showing an example of determining a target range from an image to be processed according to the first embodiment of the present invention. 12 is a flowchart for determining a target range from an image to be processed according to a second embodiment of the present invention. FIG. 7 is a diagram showing an example of determining a target range from an image to be processed according to the second embodiment of the present invention. FIG. 7 is a diagram showing an example of determining a target range from an image to be processed according to the second embodiment of the present invention. FIG. 3 is an explanatory diagram showing a procedure for generating a cutout image according to the first embodiment of the present invention. FIG. 7 is an explanatory diagram showing a procedure for generating a cutout image according to a second embodiment of the present invention. 1 is a schematic diagram of an image processing device according to an embodiment of the present invention. 1 is a schematic diagram of an image processing device according to an embodiment of the present invention. 1 is a schematic diagram of a computer-readable storage medium according to an embodiment of the invention. FIG.

Hereinafter, technical solutions according to embodiments of the present invention will be clearly and completely explained with reference to drawings according to embodiments of the present invention. Of course, the described embodiments are only some but not all embodiments of the present disclosure. All other embodiments that can be obtained by a person skilled in the art based on the embodiments of the present disclosure without requiring any creative effort fall within the scope of the present disclosure.

First, with reference to FIG. 1, an image processing method for realizing an embodiment of the present invention will be described. This method can be performed by a computer. According to this method, a user can automatically and accurately generate a cutout image by simply inputting a simple operation, and can provide a marked training image to a deep neural network.

As shown in FIG. 1, in step S101, an operation input by the user is accepted. As a method for receiving operation input, for example, it may be accepted by tracking clicks and slides of a corresponding position on an image by a mouse, or by tracking clicks and slides of a stylus connected to a computer. Good too. It should be understood that the method of accepting the operation input described above is only an example, and can be flexibly selected depending on the actual situation.

In step S102, based on this operational input, a target range of a size that is in a predetermined proportion to the size of the cutout image is determined from the image to be processed. In the first embodiment of the present invention, the operation input is used to specify a specific point in the image to be processed, and determining the target range from the image to be processed is centered around the specific point. This includes identifying the target range.

Initially, for example, the operational input may correspond to a user click with a mouse or stylus on the image to be processed, which click specifies a particular point in the image to be processed. For example, when a user wants to mark a crack in an object used for industrial purposes by image processing, this operation input specifies the crack or a specific point near the crack in the image that the user clicks on in the image to be processed. It is used to do things.

Next, a target range of a size that is in a predetermined proportion to the size of the cut-out image is determined around this specific point. The size of the cut-out image is the size desired by the user. For example, in this embodiment, the target range is a circular area having a radius of a specific value centered on this specific point, and the size of the circular area is larger than the size of the cutout image. As another example, a square with a size larger than the size of the cropped image may be determined as the target range centered on this particular point, so that the target range of the generated squares completely covers the cropped image. . As yet another example, a rectangle whose length and width are predetermined proportional values to the length and width of the cropped image is determined as the target range with this particular point as the center, and here, the generated target The range is set such that the predetermined proportional value is greater than 1 so as to completely cover the cropped image. Note that the shape of the target range is merely an example, and a desired shape of the target range can be flexibly selected as necessary.

FIG. 2 is a diagram showing an example of determining a target range from an image to be processed according to the first embodiment of the present invention. As shown in FIG. 2, the computer specifies a specific point 20 (indicated by a dot) in the image to be processed in response to a user's click with a mouse or stylus on the image to be processed, and A rectangular area centered on point 20 is determined as target range 22 . The length and width of the target range 22 have a predetermined proportional value to the length and width of the cutout image 21, and this predetermined proportional value is larger than one. For example, the predetermined proportional value can be any number greater than one. For example, if the size of the cutout image 21 is 96x96 pixels, the target range can be determined to be 128x128 pixels.

Alternatively, in order to select a cutout image desired by the user from the target range, the predetermined proportional value may be set to 3/2, 2, or 3, etc., as necessary. However, if the selected predetermined proportional value is too large, too many unnecessary cut-out images may be selected from the target image, and storage space may be wasted. If the selected predetermined proportional value is about 1, there is a risk that too many overlapping cutout images will be selected from the target image. Therefore, the predetermined proportionality value should be determined flexibly as needed.

In the second embodiment of the present invention, the operation input is used to specify a specific trajectory of the image to be processed, and determining the target range from the image to be processed is based on the specific trajectory. scanning the trajectory range using the size of the cropped image as a step size, and determining a target range corresponding to each step size with each step size as a center. For example, the operational input may correspond to a user's slide of a mouse or stylus over an image to be processed. This slide specifies a particular trajectory in the image to be processed. For example, when a user attempts to mark a crack in an object used for industrial purposes by image processing, this operation input is used to specify the trajectory of the user sliding over the crack in the image to be processed.

FIG. 3 is a flowchart for determining a target range from an image to be processed according to the second embodiment of the present invention. As shown in FIG. 3, in step S201, a trajectory range is determined based on a specific trajectory. For example, this trajectory range may be determined based on the start and end positions of a particular trajectory. It should be understood that the manner in which the trajectory range is determined is not limited to this, and the trajectory range may also be determined based on other aspects.

In step S202, after determining the trajectory range, the trajectory range is scanned using the size of the cutout image as the step size. For example, the trajectory range is scanned in the horizontal and vertical directions within the trajectory range using the size of the cutout image as a step size. Note that the overlap of cutout images when scanning the trajectory range may be random, or may be set in advance as necessary.

In step S203, a target range corresponding to each step size is determined centering on each step size, as in the first embodiment of the present invention. For example, the center of each step size is set as a specific point according to the first embodiment of the present invention, and a target range of size that is in a predetermined proportion to the size of the cut-out image is determined.

FIGS. 4a and 4b are diagrams showing an example of determining a target range from an image to be processed according to the second embodiment of the present invention. As shown in FIG. 4a, the computer specifies a particular trajectory 31 in the image to be processed based on the user's slide of the mouse or stylus over the image to be processed. For example, the trajectory range 30 is determined based on the start position and end position of the specific trajectory 31. As shown in FIG. 4b, after the trajectory range 30 is determined, the trajectory range 30 is simultaneously scanned in the horizontal and vertical directions using the cutout image 32 as a step size. The cutout image 33 and the cutout image 34 show an example in which the trajectory range 30 is scanned in the horizontal direction. Next, with each step size as the center, a target range of size that is in a predetermined proportion to the size of the cut-out image is determined, with the center of each step size as a specific point according to the first embodiment of the present invention. For example, this target area (not shown) may be circular, triangular, or square.

Returning to FIG. 1, in step S103, a predetermined number of cut-out images of a predetermined size are generated within the target range. For example, after determining the target range, a predetermined number of cut-out images of a predetermined size may be randomly generated within the target range. Alternatively, the number of cutout images to be generated at different positions in the target range may be set in advance. For example, the number of cutout images generated at the center of the target range may be set to be larger than at the edges of the target range. Further, the number and size of generated cutout images may be set in advance according to the needs of the user.

Once a cropped image is generated, it can be automatically named and saved by the computer. For example, if a user wants to use this method to mark cracks in industrial applications, after generating the cutout image, the computer will automatically name the generated cutout image "crack" and the , and save it to the corresponding target path. Additionally, different tags can be automatically set depending on different operations. As an example, multiple tags can be set depending on the left and right mouse buttons. For example, when the computer accepts a left mouse button click and slide, the cropped image generated by the above method will be automatically marked as "crack" and automatically saved in the corresponding target path, and the computer will Upon accepting the right mouse button click and slide, the cropped image generated by the above method will be automatically marked as "non-crack" and will be automatically saved in a target path different from the target path of "crack". Ru. Furthermore, as another example, a plurality of tags may be set in response to keyboard input. For example, if you set different tag types for different keyboard inputs in advance and enter them via the keyboard, the final generated cutout image will be automatically named and saved in the target path of the corresponding tag type. It is also possible to do so. Note that the tag setting method described above is not limited to this, and different tags may be set using different methods as necessary.

FIG. 5 is a diagram showing a procedure for generating a cutout image according to the first embodiment of the present invention. As shown in FIG. 5, the computer receives a user's operation input for specifying a specific point in the image to be processed (the image on the left). Based on this operational input, the computer determines a target range of size that is in a predetermined proportion to the size of the desired cropped image (right image). For example, the size of the target range may be twice, three times, etc. the size of the cutout image. Once the target range is determined, a predetermined number of user-desired cutout images of a predetermined size are generated within the target range. Once a cropped image is generated, it can be automatically named and saved by the computer.

FIG. 6 is a diagram showing a procedure for generating a cutout image according to the second embodiment of the present invention. As shown in FIG. 6, the computer receives a user's operation input for specifying a specific trajectory in the image to be processed (the image on the left). A trajectory range is determined based on this specific trajectory; for example, the trajectory range may be determined by a start position and an end position of the specific trajectory. Once the trajectory range is determined, the trajectory range is scanned using the size of the cutout image (right image) desired by the user as the step size. Finally, a target range corresponding to each step size is determined around each step size. The method for determining the target range includes, for example, determining the target range using the method according to the first embodiment described above. For example, the target range is determined by setting the center of each step size as the specific point according to the first embodiment. Once the target range is determined, a predetermined number of user-desired cutout images of a predetermined size are generated within the target range. Once a cropped image is generated, it can be automatically named and saved by the computer.

Next, with reference to FIG. 7, an image processing apparatus according to an embodiment of the present invention will be described. FIG. 7 is a schematic diagram of an image processing device according to an embodiment of the present invention. The functions of the image processing apparatus of this embodiment are similar to the method described with reference to FIG. 1, so detailed description will be omitted for the sake of brevity. As shown in FIG. 7, the image processing apparatus 1000 includes an operation input receiving section 1001, a target range determining section 1002, a cutout image generation section 1003, and a cutout image storage section 1004. Note that although the image processing apparatus 1000 is shown to include only four units in FIG. 7, this is merely an example, and the image processing apparatus 1000 may also include one or more other units. , these parts are omitted here because they are not related to the idea of the present invention.

The operation input reception unit 1001 can receive operation input from a user. There are multiple ways to receive operational input; for example, operational input may be received by tracking mouse clicks and slides at corresponding positions on the image, or operational input may be received by tracking stylus clicks and slides. may be accepted. It should be understood that the method of accepting operation input described above is just an example, and can be flexibly selected depending on the actual situation. The target range determination unit 1002 determines, from the image to be processed, a target range whose size is in a predetermined proportion to the size of the image to be processed, based on the operation input.

In the first embodiment of the present invention, the operation input is used to specify a specific point in the image to be processed. Specifying the target range from the image to be processed includes determining the target range with the specific point as the center. Initially, for example, the operational input may correspond to a user click with a mouse or stylus on the image to be processed, the click specifying a particular point on the image to be processed.

Next, a target range is determined around this specific point. The size of the target range is in a predetermined proportion to the size of the cut-out image. The size of the cut-out image is the size of the image desired by the user. For example, in this embodiment, the target range is defined as a circular area having a radius of a specific value centered on this specific point. Further, the size of the circular area is larger than the size of the cutout image. As another example, a square having a size larger than the size of the cut-out image may be determined as the target range so that the target range of the generated square completely covers the cut-out image with this specific point as the center. Furthermore, as another example, a rectangle whose length and width have a predetermined proportional value to the length and width of the cut-out image may be determined as the target range, centered on this specific point. Here, it is assumed that the generated target range has a predetermined proportional value larger than 1 so that the cutout image can be completely covered. It should be noted that the shape of the target range described above is only an example, and it should be understood that the shape of the desired target range can be flexibly selected as needed.

In the second embodiment of the present invention, the operation input is used to specify a specific trajectory in the image to be processed, and determining the target range from the image to be processed is based on the specific trajectory. scanning the trajectory range using the size of the cropped image as a step size, and determining a target range corresponding to each step size with each step size as a center. First, for example, the operational input may correspond to a user slide with a mouse or stylus in the image to be processed, which slide specifies a particular trajectory in the image to be processed.

Next, a trajectory range is determined based on the specific trajectory. For example, the trajectory range may be determined based on the start and end positions of a particular trajectory. It should be understood that the manner in which the trajectory range is determined is not limited to this, and the trajectory range may also be determined based on other aspects. After determining the trajectory range, the trajectory range is scanned using the size of the cutout image as a step size. For example, the trajectory range is scanned in the horizontal and vertical directions within the trajectory range using the size of the cutout image as a step size.

Finally, centering around each step size, a target range corresponding to each step size is determined as in the first embodiment of the present invention. The cutout image generation unit 1003 generates a predetermined number of cutout images of a predetermined size within the target range. For example, after the target range is determined, the cutout image generation unit 1003 may randomly generate a predetermined number of cutout images of a predetermined size within the target range. Alternatively, the number of cutout images generated at different positions in the target range may be set in advance. For example, the number of cutout images generated at the center of the target range may be set to be larger than at the edges of the target range. Note that the number and size of generated cutout images may be set in advance according to the user's needs.

After generating the cutout image, the automatically named cutout image may be stored by the cutout image storage unit 1004. Next, with reference to FIG. 8, an image processing device according to an embodiment of the present invention will be described. FIG. 8 is a schematic diagram of an image processing device according to an embodiment of the present invention. The functions of the image processing apparatus of this embodiment are similar to the method described with reference to FIG. 1, so detailed description will be omitted for the sake of brevity.

As shown in FIG. 8, the image processing device 1100 includes a memory 1101 and a processor 1102. Note that although FIG. 8 shows that the image processing device 1100 includes only two devices, this is only an example, and the image processing device 1100 may include one or more other devices. , these devices are omitted here because they are not relevant to the idea of the present invention.

Image processing apparatus 1100 of the present invention comprises a memory 1101 configured to store computer readable instructions and a processor 1102 configured to execute the computer readable instructions stored in the memory. Processor 1102 performs the following steps when processing and executing computer readable instructions. That is, receiving an operational input, determining a target range from an image to be processed based on the operational input, and generating a predetermined number of cut-out images of a predetermined size within the target range. Here, the size of the target range is in a predetermined proportion to the size of the image to be processed.

In the first embodiment of the present invention, the operation input is used to specify a specific point in the image to be processed, and specifying the target range from the image to be processed is to specify the specific point in the image to be processed. It involves determining this target range as a focal point.

In the second embodiment of the present invention, the operation input is used to specify a specific trajectory in the image to be processed, and determining the target range from the image to be processed is based on the specific trajectory. to scan this trajectory range using the size of the cropped image as a step size, and to determine a target range corresponding to each step size with each step size as the center. include. Here, determining the trajectory range based on the specific trajectory includes determining the trajectory range based on the start position and end position of the specific trajectory. Once the target range is determined, a predetermined number of cut-out images of a predetermined size are randomly generated from the target range, and the automatically named cut-out images are stored in the memory 1101.

FIG. 9 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure. As shown in FIG. 9, the present disclosure includes a computer-readable storage medium 1200 on which computer-readable instructions 1201 are stored, and when the computer-readable instructions are executed by the computer, the computer performs the following steps. That is, receiving an operational input, determining a target range from an image to be processed based on the operational input, and generating a predetermined number of cut-out images of a predetermined size in this target range. Here, the size of this target range is in a predetermined proportion to the size of the cut-out image.

As described above, in the above aspect of the present disclosure, a user's operation input (for example, click or slide of a mouse or stylus) is accepted, and based on the operation input, a target, such as a circle, a triangle, or a rectangle, is formed from an image to be processed. It can be seen that a range is determined, and within this target range, a predetermined number of cropped images of a predetermined size are generated by the computer. Here, the size of the target range is in a predetermined proportion to the size of the cut-out image. With this method, a cutout image can be automatically and accurately generated even with little operator input. Thereby, it is possible to provide a marked training image to the neural network, and it is possible to reduce theoretical cognitive errors and visual errors, and to reduce the cost of manual marking.

As will be understood by those skilled in the art, aspects of this application can be described or interpreted in terms of a number of patentable types or situations, and include novel and useful processes, machines, products, or combinations of materials; or any new or useful improvements thereto. Accordingly, aspects of the present application may be implemented entirely in hardware, or may be implemented entirely in software (including firmware, resident software, microcode, etc.), or may be implemented entirely in hardware. and software. The hardware or software described above may be referred to as a "data block", "module", "engine", "unit", "component", or "system". Additionally, aspects of the present application may be a computer product residing in one or more computer-readable media, the product including computer-readable program code.

This application uses certain terminology to describe embodiments of the application. "One embodiment," "an embodiment," and/or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of the present application. Accordingly, it is understood that references to "one embodiment" or "an embodiment" or "alternative embodiment" mentioned more than once in different places herein do not necessarily refer to the same embodiment. Should. Moreover, the particular features, structures, or characteristics of one or more embodiments of the present application may be combined in any suitable manner. Unless defined otherwise, all terms (including technical and scientific terms) used herein have the same meaning as understood by one of ordinary skill in the art to which this disclosure belongs. Additionally, terms as defined in a common dictionary, unless expressly defined herein, shall be construed as having meanings consistent with their meaning in the context of the relevant art. , should not be interpreted in an idealized or overly formalized sense. The above content is illustrative of the present disclosure and should not be considered limiting. Although several exemplary embodiments of the present disclosure have been described, those skilled in the art may make many modifications to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. . Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined by the claims. The foregoing is illustrative of the present disclosure and should not be considered limited to the particular embodiments disclosed, and modifications of the disclosed embodiments and other embodiments are within the scope of the appended claims. is intended to be included within the scope of. The disclosure is defined by the claims and their equivalents.

Claims (13)

Accepting operation input;
determining a target range from an image to be processed based on the operation input;
generating a predetermined number of cut-out images of a predetermined size within the target range;
The size of the target range is in a predetermined proportion to the size of the cutout image,
The operation input is used to specify a specific trajectory in the image to be processed,
Determining the target range from the image to be processed includes:
determining a trajectory range based on the specific trajectory;
scanning the trajectory range using the size of the cut-out image as a step size;
An image processing method comprising: determining a target range corresponding to each step size, with each step size as a center . The image according to claim 1 , wherein determining the trajectory range based on the specific trajectory includes determining the trajectory range based on a start position and an end position of the specific trajectory. Processing method. 3. The image processing method according to claim 1 , further comprising randomly generating a predetermined number of cut-out images of a predetermined size from within the target range. 4. The image processing method according to claim 3 , wherein the generated cutout image is automatically named and saved. an operation input reception unit arranged to receive operation input;
a target range determining unit arranged to determine a target range from an image to be processed based on the operation input;
a cutout image generation unit arranged to generate a predetermined number of cutout images of a predetermined size within the target range,
The size of the target range is in a predetermined proportion to the size of the cutout image,
The operation input is used to specify a specific trajectory in the image to be processed,
The target range determining unit determines a trajectory range based on the specific trajectory,
scanning the trajectory range using the size of the cutout image as a step size;
An image processing apparatus, characterized in that a target range corresponding to each step size is determined with each step size as a center . The image processing apparatus according to claim 5 , wherein the target range determining unit determines the trajectory range based on a start position and an end position of the specific trajectory. The image processing device according to claim 5 or 6 , wherein the cutout image generation unit randomly generates a predetermined number of cutout images of a predetermined size from within the target range. The image processing apparatus according to claim 7 , further comprising a cutout image storage unit that saves the automatically named cutout image. a memory arranged to store computer readable instructions;
a processor arranged to execute the computer readable instructions stored in the memory;
When the processor executes the computer readable instructions,
a step of accepting operation input;
determining a target range from the image to be processed based on the operation input;
generating a predetermined number of cut-out images of a predetermined size within the target range;
The size of the target range is in a predetermined proportion to the size of the cutout image,
The operation input is used to specify a specific trajectory in the image to be processed,
Determining the target range from the image to be processed includes:
determining a trajectory range based on the specific trajectory;
scanning the trajectory range using the size of the cut-out image as a step size;
An image processing device comprising: determining a target range corresponding to each step size, with each step size as a center . The image according to claim 9 , wherein determining the trajectory range based on the specific trajectory includes determining the trajectory range based on a start position and an end position of the specific trajectory. Processing equipment. The image processing device according to claim 9 or 10 , further comprising randomly generating a predetermined number of cut-out images of a predetermined size from the target range. The image processing device according to claim 11 , wherein the memory stores the automatically named cutout image. A computer-readable storage medium storing computer-readable instructions, the computer-readable storage medium comprising:
When the computer readable instructions are executed by a computer, the computer:
a step of accepting operation input;
determining a target range from the image to be processed based on the operation input;
generating a predetermined number of cut-out images of a predetermined size within the target range;
The size of the target range is in a predetermined proportion to the size of the cutout image,
The operation input is used to specify a specific trajectory in the image to be processed,
Determining the target range from the image to be processed includes:
determining a trajectory range based on the specific trajectory;
scanning the trajectory range using the size of the cut-out image as a step size;
determining a target range corresponding to each step size around each step size .

Images