JP2019139497A - Image processing system and image processing method - Google Patents

Abstract

To provide an image processing system and an image processing method that enable creation of learning data for object detection AI without human intervention.SOLUTION: An image processing system includes an arithmetic device that executes a predetermined process and a storage device connected to the arithmetic device. The arithmetic device divides an input image by a predetermined grid pattern; estimates an object reflected in each of the divided regions and accuracy thereof and excludes an object whose accuracy of the estimated object is smaller than a predetermined threshold, in which, of objects that were not excluded, a same type of object was estimated; and combines adjacent regions to define an overall grid.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an image processing apparatus and an image processing method for generating learning data for object detection AI.

  With the advancement of object detection technology, AI (Artificial Intelligence) for object detection using deep learning enables identification of multiple object types (dogs, cats, cars, etc.) appearing in an image and information on positions in the image. Now it can be acquired at high speed and with high accuracy.

Real-Time Object Detection, [Search January 6, 2018], Internet <URL: https://pjreddie.com/darknet/yolo/> SSD: Single Shot MultiBox Detector, [Search January 6, 2018], Internet <URL: https://github.com/weiliu89/caffe/tree/ssd>

  In order to improve the accuracy of object detection, it is necessary to learn a record in which a large number of images and types and position information of objects appearing in each image are described. This learning data may require tens of thousands of points, and there is a problem that costs are high if it is created manually.

  As a conventional technique for mechanically extracting a place that looks like an object, there is Selective Search. Selective Search is an algorithm that selects candidate areas by grouping similar areas at the pixel level. In Selective Search, a candidate region is mechanically selected by color information for a similar region, and thus an object may not be appropriately extracted. Further, the candidate area is selected, and it is impossible to identify what the image in the candidate area is. For this reason, learning data for the object detection AI cannot be generated only by the selective search.

  An object of the present invention is to make it possible to create learning data for an object detection AI regardless of human hands.

  A typical example of the invention disclosed in the present application is as follows. That is, the image processing system includes an arithmetic device that executes a predetermined process and a storage device connected to the arithmetic device, and the arithmetic device divides the input image by a predetermined grid pattern, The object reflected in each of the divided areas and the accuracy thereof are estimated, the object whose accuracy of the estimated object is smaller than a predetermined threshold is excluded, and among the objects that are not excluded, the same kind of object is estimated. The entire grid is defined by combining adjacent regions.

  According to one embodiment of the present invention, learning data for object detection AI can be created without human intervention. Problems, configurations, and effects other than those described above will become apparent from the following description of embodiments.

It is a block diagram which shows the structure of the object detection apparatus which concerns on the Example of this invention. It is a figure which shows the structural example of an area | region detection result file. It is a flowchart of the process which a central processing unit performs. It is a detailed flowchart of the object detection process which an area | region detection process part performs. This is the grid pattern file format. It is a flowchart of the detail of a grid search process. It is a figure which shows the example of calculation of a center grid. It is a figure which shows the example of calculation of an average grid. It is a flowchart of the detail of a grid search process. It is a figure explaining merge processing. It is a figure explaining a center grid calculation process.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, in this specification, AI that identifies the type of one object (dog, cat, car, etc.) in one image is called image recognition. Also, an AI in which a plurality of objects appear in one image and the type and position information of each object can be identified is referred to as object detection.

  FIG. 1 is a block diagram showing a configuration of an object detection apparatus according to an embodiment of the present invention.

  The object detection device extracts a type of an object (object) included in an image input to the device and position information in the image. The object detection device includes a central processing unit 010, a data memory 020, a program memory 030, a display device 040, image recognition AI trained data 050, a grid pattern file 060, an image before area detection 070, an area detection result file 080, a keyboard 090, and A computer system having a pointing device 100 is used. The central processing unit 010 includes a data memory 020, a program memory 030, a display device 040, an image recognition AI trained data 050, a grid pattern file 060, a pre-region detection image 070, a region detection result file 080, a keyboard 090, and a pointing device 100. Are connected to each other.

  The central processing unit 010 includes an image recognition AI trained data reading unit 011, a pre-region detection image reading unit 012, a region detection processing unit 013, and a region detection result output unit 014. Each of these units is realized by the central processing unit 010 executing a predetermined program. Note that part of processing performed by the object detection apparatus executing a program may be performed by hardware (for example, FPGA).

  In the central processing unit 010, first, the image recognition AI trained data reading unit 011 reads an image recognition AI file. The image recognition AI is an AI that has been trained so that an object that the user wants to recognize can be identified. Examples include pre-learned files (VGG16, Inception V3, etc.) that are publicly distributed.

  Using this AI function, the region detection processing unit 013 detects an object from the image read by the pre-region detection image reading unit 012. The area detection result output unit 014 outputs the object type specified by the area detection processing unit 013 and position information in the image to a file. The details of the method for specifying the type and position information of the object by the area detection processing unit 013 will be described later.

  The data memory 020 stores data used by each processing unit of the central processing unit 010 for processing. Specifically, the data memory 020 stores prediction image data 021 and image recognition AI trained data 022.

  The image recognition AI trained data 050 is a file for realizing the image recognition AI, and may be created in advance by a user who uses the object detection apparatus of the present embodiment.

  Note that the learning data for the image recognition AI is different from the object detection data. For each directory, the image to be identified, such as only the dog image, only the cat image, and only the human image, is divided into the learning data. Can be created at low cost. In this embodiment, the user can create learning data for object detection only by the image recognition AI.

  The grid pattern file 060 specifies the size when dividing the pre-area detection image 070. The grid pattern file 060 is preferably created in advance by a user who uses the object detection apparatus of the present embodiment, but can be changed by the user.

  The program executed by the central processing unit 010 is provided to the object detection device via a removable medium (CD-ROM, flash memory, etc.) or a network, and is stored in a nonvolatile auxiliary storage device that is a non-temporary storage medium. . For this reason, the object detection device may have an interface for reading data from a removable medium.

  An object detection device is a computer system configured on a single physical computer or a plurality of logically or physically configured computers, and is a virtual system constructed on a plurality of physical computer resources. It may operate on a computer.

  FIG. 2 is a diagram showing a configuration example of the region detection result file 080, and shows a format of the region detection result file 080 output by the object detection processing device.

  The area detection result file 080 stores a record including an image file name 201, an object type 202, an upper left X coordinate 203, an upper left Y coordinate 204, an object width 205, and an object height 206 (for example, CSV File). An image file name 201 is a file name in which an area is detected. In the object type 202, integers from 0 to N are recorded, and each numerical value indicates the object type (0 = dog, 1 = cat, 2 = human, etc.). The upper left X coordinate 203 and upper left Y coordinate 204 of the object are the coordinates of the upper left point of the rectangle in which the object is included in the image. The width 205 and the height 206 of the object are the size of the rectangle in which the object is included in the image (the length in the horizontal and vertical directions from the upper left point to the lower right point). The region detection result file 080 may be used for learning a deep learning model for detecting an object such as SSD or yolov2 for the purpose of improving the speed of object detection.

<About system operation>
FIG. 3 is a flowchart of processing executed by the central processing unit 010.

  First, the image recognition AI trained data reading unit 011 reads the image recognition AI trained data 050 (301).

  Next, the pre-area detection image reading unit 012 reads the pre-area detection image 070 and stores the number of read image files in the ImgNum variable (302).

  Next, the area detection processing unit 013 detects an object for each read image (303), and the area detection result output unit 014 writes the object detection result in the area detection result file 080 (304).

  FIG. 4 is a detailed flowchart of the object detection process 303 executed by the area detection processing unit 013.

  The grid in the present embodiment is a frame for searching for an image. First, in step 401, the grid pattern file 060 is read and the search frame is stored in the memory.

  As the grid pattern file 060, for example, the format shown in FIG. 5 can be used. The grid pattern file 060 is a file in which a grid width (W) 501 and a height (H) 502 are described (for example, in CSV format). Each described grid pattern is at least one of width and height different from the other grid patterns. The unit specified in the grid pattern file 060 is a ratio to an image, a pixel unit, a centimeter, or the like. The user of the object detection apparatus according to the present embodiment can change the grid size according to the size of the pre-area detection image 070 and the ratio of the object to be detected in the image. It should be noted that the object size can be detected from about 2 to 4 times the grid size described in the grid pattern file 060.

  Next, a grid search is performed for each of the plurality of grid patterns read from the grid pattern file 060 to detect the type and area of the object (402). Details of the grid search processing 402 will be described with reference to FIG.

  After the grid search using all the grid patterns is completed, the results obtained by the grid search processing 402 are averaged and merged for each grid pattern to improve the accuracy of the region (403). Details of the merge processing 403 will be described with reference to FIG.

  If the data specifying the type and area obtained in step 403 is used as learning data for the object detection AI, the cost for creating learning data can be reduced.

  FIG. 6 is a detailed flowchart of the grid search process 402. In FIG. 6, the right side is a flowchart of processing, and the left side shows an example of an image to be processed.

  First, the image read in the process 302 for reading the image file before region detection is divided into grids having the width W and the height H of the grid pattern read from the grid pattern file 060 in 401. In the example shown in FIG. 6, the pre-area detection image 070 is divided into nine grids 6011 to 6019 having a width W and a height H (601).

  Each of the divided images is input to the image recognition AI, and the type of object appearing in the grid and the probability of being the object are predicted (602). Through the processing in step 602, the grids 6011, 6012, 6014, and 6015 are predicted to have cars with a probability of 85%, 90%, 90%, and 90%, respectively. Similarly, the grid 6013 shows a signal with a probability of 99%, the grids 6016 and 6019 show a person with a probability of 75% and 85%, respectively, and the grids 6017 and 6018 show a 5% probability. It is predicted that the dog is reflected in the probability.

  As a result of the prediction, a grid having a probability lower than a specific threshold is determined as a grid in which the predicted type of object is not captured (603). For example, if the threshold value is 50%, the probability of dogs in the grids 6017 and 6018 is smaller than the threshold value, so it is determined that the predicted type of object (dog) is not captured and is excluded from the detection target.

  Next, when it is determined that a plurality of adjacent grids are the same type of object, the center position of the grid is obtained (604). For example, since the same type of object (car) is shown in adjacent grids 6011, 6012, 6014 and 6015, one central grid 6041 is obtained from the grids 6011, 6012, 6014 and 6015. Similarly, since the same type of object (person) is shown in the adjacent grids 6016 and 6019, one central grid 6042 is obtained from the grids 6016 and 6019. In the grid 6013, since the signal is detected by only one grid, the center grid 6043 is at the same position as the detected grid. The calculation of the center grid is illustrated in FIG.

  Thereafter, the adjacent grids of the same object are combined as one grid to obtain an entire grid (605). For example, the grids 6011, 6012, 6014, and 6015 are combined to create the entire vehicle grid 6051. Similarly, grids 6016 and 6019 are combined to create an overall grid 6052 for a person. In the grid 6013, since the signal is detected by only one grid, the whole grid 6053 and the center grid 6043 coincide.

  Then, an average of the center grid and the entire grid is obtained (606). In many grids, no object is shown in the corner of the area, so the outer frame is reduced by calculating the average of the center grid and the whole grid. For example, as shown in FIG. 11, when the average of the center grid 1101 and the overall grid 1102 is calculated, an average grid 1103 from which the margins included in the overall grid are removed can be generated. The calculation for obtaining the average grid will be described with reference to FIG.

  In addition, although the process which calculates | requires an average grid using the whole grid and the center grid was demonstrated in FIG. 6, you may obtain | require only a whole grid by integrating | segmenting the area | region divided | segmented into the grid, without calculating | requiring an average grid. In this case, the accuracy of the characteristics of the area in which the object is shown is lowered, but the presence or absence of the object can be reliably detected.

  FIG. 7 is a diagram illustrating a calculation example of the center grid.

  In the grids G1, G2, G3 and G4, the same type of object is detected. Each grid has coordinates of top on the rectangle, left on the left, bottom on the bottom, and right on the right. The rectangular vertex of the grid C that is the center of the grids G1, G2, G3, and G4 is a value obtained by dividing the sum of top, left, right, and bottom coordinates of each grid by the number of grids.

  FIG. 7 shows the calculation formula. G1 (top) to G4 (top) are the Y coordinates of the upper sides of the grids 701 to 704, and the average value of G1 (top) to G4 (top) is the Y coordinate C (top) of the upper side of the center grid. Similarly, G1 (left) to G4 (left) are the X coordinates of the left side of the grids 701 to 704, and the average value of G1 (left) to G4 (left) is the X coordinate C (left) of the left side of the center grid. Become. G1 (right) to G4 (right) are the X coordinates of the right side of the grids 701 to 704, and the average value of G1 (right) to G4 (right) is the X coordinate C (right) of the right side of the center grid. . G1 (bottom) to G4 (bottom) are Y coordinates of the lower sides of the grids 701 to 704, and an average value of G1 (bottom) to G4 (bottom) is a Y coordinate C (bottom) of the lower side of the center grid. .

  FIG. 8 is a diagram illustrating a calculation example of the average grid.

  The rectangular vertex of the grid 802 obtained by averaging the positions of the overall grid 801 and the center grid 803 is a value obtained by dividing the sum of the top, left, right, and bottom coordinates of the overall grid 801 and the center grid 803 by 2.

  FIG. 8 shows a calculation example. G (top) is the Y coordinate of the upper side of the entire grid, C (top) is the Y coordinate of the upper side of the center grid, and the average value of G (top) and C (top) is the Y coordinate of the upper side of the average grid. M (top). Similarly, G (left) is the X coordinate of the left side of the entire grid, C (left) is the X coordinate of the left side of the center grid, and the average value of G (left) and C (left) is the left side of the average grid. Coordinates M (left). G (right) is the X coordinate of the right side of the entire grid, C (right) is the X coordinate of the right side of the center grid, and the average value of G (right) and C (right) is the right side of the average grid. X coordinate. G (bottom) is the Y coordinate of the lower side of the entire grid, C (bottom) is the Y coordinate of the lower side of the center grid, and the average value of G (bottom) and C (bottom) is the lower side of the average grid. Y coordinate.

  FIG. 9 is a flowchart of the same processing as that of FIG. 6, but the size of the grid is reduced. Therefore, it is easy to detect objects (signals, dogs, cats, etc.) smaller than those in FIG. 6, but large objects (cars, etc.) are difficult to detect. For this reason, it is preferable to detect a large object with a large grid and detect a small object with a small grid.

  FIG. 10 is a diagram for explaining the merge process 403.

  As shown in FIG. 10, when an area is searched from an image using a plurality of grid patterns 1 to N, an average grid of objects (cars, signals, people) detected in each image is obtained.

  Next, a plurality of average grids obtained by grid search are integrated. For example, first, an average grid is overlaid for each detected object, and it is determined whether the area where the average grid overlaps exceeds a predetermined threshold. If the overlapping area exceeds a predetermined threshold, it is determined that the same object is detected, and the average value of the four corners (top, left, right, bottom) of each average grid is calculated and the region detection result And The average value can be calculated simply by arithmetic mean,

  Thereafter, the calculated region detection result (coordinate values at the four corners) is output to the region detection result file 080.

  Specifically, since a plurality of areas of the average grid where the car is detected overlap, the four average grids where the car is detected are merged. Merge the average grid where the signal was detected and merge the average grid where the person was detected. By merging, unnecessary areas around the object can be removed, and the area analysis performance can be improved.

  If the overlapping area is smaller than a predetermined threshold value, it is determined that a plurality of objects of the same type are detected, and each average grid may be handled as a different area.

  As described above, according to the embodiment of the present invention, the image processing system divides an input image by a predetermined grid pattern, and estimates an object reflected in each of the divided areas and its accuracy. Since the estimated object accuracy is excluded from objects smaller than a predetermined threshold, among the objects that are not excluded, the same type of object is estimated, and the adjacent grid is combined to define the entire grid. Conventionally, learning data that has been created by manually describing the type and position of an object can be created in the AI, thereby reducing learning data creation costs and improving the accuracy of the learning data. In addition, by using deep learning for selecting object types and object candidates, it is possible to detect an object that has been missed by Selective Search.

  Further, the image processing system defines a center grid arranged at a center position of an adjacent region where the same type of object is estimated, and the center grid and the entire grid for each of the objects for which the center grid is determined. Since the average grid is defined between the two, the margin can be removed and the decrease in learning accuracy due to other objects reflected in the background can be suppressed.

  The grid pattern used to divide the image is prepared with a plurality of rectangles having at least one different width and height, and the image processing system divides the input image by the plurality of grid patterns. Since the processing for determining the entire grid, the center grid, and the average grid is executed for each of the regions that have been formed, it is possible to appropriately detect objects having various shapes (for example, vertically long and horizontally long).

  In addition, the image processing system integrates the average grid determined using the plurality of grid patterns to identify the area where the object exists, and thus can appropriately detect objects of various shapes.

  Further, the image processing system calculates the average of the coordinates of the vertices of the rectangle of the average grid determined using the plurality of grid patterns, and integrates the average grid. Shaped objects can be detected appropriately.

  The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the scope of the appended claims. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and the present invention is not necessarily limited to those having all the configurations described. A part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Moreover, you may add the structure of another Example to the structure of a certain Example. In addition, for a part of the configuration of each embodiment, another configuration may be added, deleted, or replaced.

  In addition, each of the above-described configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them, for example, with an integrated circuit, and the processor realizes each function. It may be realized by software by interpreting and executing the program to be executed.

  Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, and an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, and a DVD.

  Further, the control lines and the information lines are those that are considered necessary for the explanation, and not all the control lines and the information lines that are necessary for the mounting are shown. In practice, it can be considered that almost all the components are connected to each other.

010 Central processing unit 011 Data reading unit 012 Pre-region detection image reading unit 013 Region detection processing unit 014 Region detection result output unit 020 Data memory 021 Image data for prediction 022 Image recognition AI trained data 030 Program memory 040 Display device 050 Image recognition AI trained data 060 Grid pattern file 070 Image before area detection 080 Area detection result file 090 Keyboard 100 Pointing device

Claims (10)

An image processing system,
An arithmetic device that executes a predetermined process, and a storage device connected to the arithmetic device,
The arithmetic unit is:
Divide the input image by a predetermined grid pattern,
Guess the object and its accuracy in each of the divided areas,
Exclude objects whose inferred object accuracy is less than a predetermined threshold;
An image processing system characterized in that, among the objects that are not excluded, an object of the same type is estimated, and an entire grid is defined by combining adjacent regions. The image processing system according to claim 1,
The arithmetic unit is:
Defining a central grid that is placed at the center position of the adjacent region in which the same type of object is estimated;
An image processing system, wherein an average grid is defined between the center grid and the overall grid for each object for which the center grid is defined. The image processing system according to claim 2,
The grid pattern used to divide the image is provided with a plurality of rectangles having different widths and heights,
The said arithmetic unit performs the process which determines the whole grid, a center grid, and an average grid about each area | region which divided | segmented the input image with the some grid pattern, The image processing system characterized by the above-mentioned. The image processing system according to claim 3,
The said arithmetic unit integrates the average grid defined using these grid patterns, and specifies the area | region where the said object exists, The image processing system characterized by the above-mentioned. The image processing system according to claim 4,
The image processing system, wherein the arithmetic unit calculates an average of coordinates of each vertex of a rectangle of an average grid determined using the plurality of grid patterns, and integrates the average grid. An image processing method executed by an image processing system,
The image processing system includes an arithmetic device that executes predetermined processing, and a storage device connected to the arithmetic device,
The method
The arithmetic device divides the input image by a predetermined grid pattern,
The arithmetic device estimates the object and its accuracy in each of the divided areas,
The arithmetic unit excludes objects whose accuracy of the estimated object is smaller than a predetermined threshold;
An image processing method characterized in that the arithmetic device determines a whole grid by combining adjacent regions in which the same kind of objects are estimated among the objects not excluded. The image processing method according to claim 6,
The arithmetic device defines a center grid arranged at a center position of an adjacent region where the same kind of object is estimated;
The image processing method, wherein the arithmetic device determines an average grid between the center grid and the overall grid for each object for which the center grid is determined. The image processing method according to claim 7, comprising:
The grid pattern used to divide the image is provided with a plurality of rectangles having different widths and heights,
The image processing method is characterized in that the calculation device executes a process of determining an overall grid, a center grid, and an average grid for each region obtained by dividing an input image by a plurality of grid patterns. The image processing method according to claim 8, comprising:
The image processing method, wherein the arithmetic device identifies an area where the object exists by integrating average grids determined using the plurality of grid patterns. The image processing method according to claim 9, comprising:
The image processing method, wherein the arithmetic unit calculates an average of coordinates of each vertex of a rectangle of an average grid defined using the plurality of grid patterns, and integrates the average grid.