JP6939065B2 - Image recognition computer program, image recognition device and image recognition method - Google Patents

Description

The present invention relates to, for example, an image recognition computer program, an image recognition device, and an image recognition method for recognizing a subject represented on an image.

Surveillance cameras may be installed inside or outside the building for purposes such as crime prevention, disaster prevention, or information gathering. Based on the image obtained by such a surveillance camera, in order to achieve the desired purpose as described above, it is preferable that the subject to be recognized such as a person or a vehicle can be accurately detected from the image. Therefore, a technique has been proposed in which a region in which a subject is represented in a frame of a target video is extracted by obtaining a background subtraction region between the frame of the background image and the frame of the target video (see, for example, Patent Document 1). ..

Japanese Unexamined Patent Publication No. 06-251151

As one of the techniques for detecting a subject from an image, a classifier using machine learning such as adaBoost, a support vector machine, or a neural network is known. In order to enable highly accurate detection of the subject from the image using such a classifier, various images in which the subject is represented and various images in which the subject is not represented are displayed under the installation environment of the camera. , Collected as a teacher image for learning the classifier.

In order to collect such a teacher image, for each image, it is confirmed whether or not the subject is represented in the image, and if the subject is represented, the area in which the subject is represented is displayed. The work of adding the frame to be shown and the identification information to the image is performed. If the worker performs such work for each image, the man-hours will be enormous. Therefore, it is preferable that such work can be performed automatically as much as possible. However, for example, even when it is automatically determined whether or not a subject is represented in an image by using a technique as described in Patent Document 1, a non-subject may be erroneously detected as a subject or a subject may be detected. May fail to be detected. In such a case, the obtained set of teacher images is not suitable for learning the classifier.

In one aspect, it is an object of the present invention to provide an image recognition computer program capable of identifying an image in which a region in which a subject is represented may be erroneously detected.

According to one embodiment, a computer program for image recognition is provided. This computer program for image recognition detects an area in which the subject is represented from each of a plurality of images in the time series obtained by the imaging unit, and the degree of change in the aspect ratio of the area in which the subject is represented along with the time series. Or, depending on the difference between the foreground value and the background value of each pixel in the area where the subject is represented, the area where the subject is represented may be erroneously detected among the plurality of images. Includes instructions to get the computer to identify the image of interest.

According to one aspect, it is possible to identify an image in which the area in which the subject is represented may be erroneously detected.

It is a hardware block diagram of the image recognition apparatus according to one Embodiment. It is a functional block diagram of a CPU related to an image recognition process. It is explanatory drawing of the outline of the pixel value difference calculation. It is explanatory drawing of the outline of aspect ratio calculation of a subject area. It is a figure which shows an example of the detection information table. It is a figure which shows an example of the image displayed on the user interface. (A) and (b) are diagrams showing an example of an image displayed on the user interface according to a modified example. It is an operation flowchart of an image recognition process.

Hereinafter, the image recognition device, the image recognition method used in the image recognition device, and the computer program for image recognition will be described with reference to the drawings. This image recognition device is used, for example, to create a set of teacher images used for learning a classifier for subject recognition based on machine learning. Therefore, in the installation environment of the camera in which the classifier is used, this image recognition device is a subject area in which the subject to be detected is represented from each of a series of a plurality of images obtained by the camera. Is detected. Then, this image recognition device calculates a statistical representative value of the difference in pixel values between the foreground and the background of each pixel in the subject area, and the aspect ratio of the subject area. Then, this image recognition device needs attention to an image in which the subject area may be erroneously detected according to the statistical representative value of the difference in pixel values or the degree of change in the aspect ratio of the subject area along the time series. Identify as an image. In the present specification, the fact that the subject area is erroneously detected means that not only the subject area represents something other than the subject, but also that the subject area represents only a part of the subject. included.

FIG. 1 is a hardware configuration diagram of an image recognition device according to one embodiment. As shown in FIG. 1, the image recognition device 1 includes a camera 2, a communication interface 3, a user interface 4, a memory 5, a storage medium access device 6, and a CPU (Central Processing Unit) 7.

The camera 2 is an example of an imaging unit, captures a predetermined shooting range, and generates an image in which the shooting range is captured. Therefore, the camera 2 forms an image of a photographing range on a two-dimensional detector composed of an array of solid-state image sensors having sensitivity to visible light such as a CCD or C-MOS, and the two-dimensional detector. It has an imaging optical system. The camera 2 generates an image at regular shooting cycles (for example, 1/30 second). Then, each time the camera 2 generates an image, the camera 2 outputs the image to the communication interface 3 via the communication network.
In the present embodiment, the image generated by the camera 2 is a monochrome image in which the value of each pixel is represented by a luminance value. However, the image generated by the camera 2 may be a color image in which the value of each pixel is represented by the value of the RGB color space or another color space (for example, the HLS color space or the YPbPr color space).

The communication interface 3 includes a communication interface for connecting to a communication network according to a communication standard such as Ethernet (registered trademark) and a control circuit thereof. The communication interface 3 receives an image from the camera 2 via the communication network and passes the received image to the CPU 7. Further, the communication interface 3 may output a set of teacher images for learning of the machine learning system received from the CPU 7 to another device (not shown) via the communication network.

The user interface 4 has, for example, an input device such as a keyboard and a mouse, and a display device such as a liquid crystal display. Alternatively, the user interface 4 may have a device such as a touch panel in which an input device and a display device are integrated. Then, the user interface 4 displays, for example, an image received from the CPU 7 that has undergone image recognition processing and the image recognition result on the display device. Further, the user interface 4 outputs the image displayed on the display device, information representing the circumscribed rectangle of the subject area modified according to the operation of the user who confirmed the image recognition result, subject identification information, and the like to the CPU 7.

The memory 5 is an example of a storage unit, and is, for example, a read / write semiconductor memory and a read-only semiconductor memory. Then, the memory 5 stores, for example, various data and recognition results for executing the image recognition process executed by the CPU 7. Further, the memory 5 may store the image acquired from the camera 2 for a certain period of time.

The storage medium access device 6 is another example of the storage unit, and is a device that accesses a storage medium 8 such as a magnetic disk, a semiconductor memory card, and an optical storage medium. The storage medium access device 6 reads, for example, a computer program for image recognition processing executed on the CPU 7 stored in the storage medium 8 and passes it to the CPU 7. Alternatively, the storage medium access device 6 may receive the created set of teacher images from the CPU 7 and write the set to the storage medium 8.

The CPU 7 is an example of a control unit, and includes, for example, one or a plurality of processors and peripheral circuits thereof. Then, the CPU 7 controls the entire image recognition device 1. Further, each time the CPU 7 receives an image from the camera 2 via the communication interface 3, the CPU 7 associates the image with the image generation date and time and stores the image in the memory 5. Then, the CPU 7 executes an image recognition process for each of the plurality of images to be included in the set of teacher images.

In the present embodiment, the plurality of images to be subjected to the image recognition processing are a series of continuous images in chronological order generated by the continuous shooting by the camera 2. The plurality of images to be subjected to the image recognition processing may include a plurality of sets of such a series of images. Further, the subject to be recognized in the image recognition process is a person. However, the subject to be recognized is not limited to a human being, and may be, for example, a vehicle or an animal other than a human being.

FIG. 2 is a functional block diagram of the CPU 7 regarding image recognition processing. As shown in FIG. 2, the CPU 7 includes a subject detection unit 11, a pixel value difference calculation unit 12, an aspect ratio calculation unit 13, a caution determination unit 14, and a presentation unit 15.

Each of these parts included in the CPU 7 is, for example, a functional module realized by a computer program executed on the processor included in the CPU 7. Alternatively, each of these parts of the CPU 7 may be mounted on the image recognition device 1 as firmware.

The subject detection unit 11 detects a subject area, which is an area in which the subject is represented, from each of a plurality of images to be image recognition processed. Since the subject detection unit 11 may perform the same processing on each of the plurality of images, the processing on one image will be described below.

The subject detection unit 11 detects the subject area by performing background subtraction processing on the image, for example. In this case, the subject detection unit 11 calculates the absolute value of the difference in the brightness values between the corresponding pixels of the image and the background image, and extracts the pixels in which the absolute value of the difference in the brightness values is equal to or more than a predetermined threshold value. Then, the subject detection unit 11 sets the set of extracted pixels as the subject area. The background image is generated, for example, when the camera 2 shoots the shooting range when the subject does not exist in the shooting range of the camera 2, and is stored in the memory 5 in advance.

The subject detection unit 11 may perform labeling processing on the set of extracted pixels, and the pixels connected to each other may be used as one subject area. As a result, the subject detection unit 11 can detect a subject area in which the subject is represented for each subject even when a plurality of subjects are captured in one image. Further, the subject detection unit 11 may use a region obtained by executing the expansion / contraction calculation of the morphology as the subject region before the labeling process. As a result, the subject detection unit 11 can include isolated pixels in the subject area even though a part of the subject is captured.

Further, the subject detection unit 11 may detect the subject area from the image by using a general-purpose classifier for subject detection, which is not optimized for the installation environment of the camera 2. In this case, the subject detection unit 11 sets a window on the image, calculates a feature (for example, a Haar-like feature) to be input to the general-purpose classifier from the value of each pixel in the window, and general-purposes the feature. By inputting to the classifier, it is determined whether or not the subject is shown in the window. Then, the subject detection unit 11 may detect the subject area by performing background subtraction processing again on the window determined to show the subject. By repeating the above process while changing the position of the window, the subject detection unit 11 can detect the subject area of the subject appearing at any position on the image.

As the general-purpose classifier, for example, an adaBoost classifier, a support vector machine, a multi-layer perceptron, or the like can be used. However, the general-purpose classifier has lower subject detection accuracy than the classifier optimized according to the installation environment of the camera 2. Therefore, the general-purpose classifier may fail to detect the subject area even though the subject is captured, even if there is a slight possibility that the area other than the subject is erroneously detected as the subject area. It is preferably adjusted to be low.

When a plurality of subject areas are detected on the image, the subject detection unit 11 leaves the subject area having the largest area among the subject areas, and deletes the other subject areas as erroneously detected. good. Alternatively, the subject detection unit 11 may delete the subject area whose area is less than a predetermined area threshold value as being erroneously detected.

The detected subject area is represented by, for example, a binary image in which the value of each pixel included in the subject area and the value of each pixel outside the subject area are different. The subject detection unit 11 notifies the pixel value difference calculation unit 12 and the aspect ratio calculation unit 13 of the detected subject area for each image.

The pixel value difference calculation unit 12 is a statistical representative value of the difference in pixel values between the foreground and the background in the subject area with respect to the image in which the subject area is detected among the plurality of images to be image recognition processed. Is calculated. The statistical representative value of the difference between the pixel values is one of the feature quantities used for identifying the image requiring attention in which the subject area may be erroneously detected. Since the pixel value difference calculation unit 12 may perform the same processing for each of the plurality of images, the processing for one image will be described below.

For example, when the subject area is confirmed for the immediately preceding image, the pixel value difference calculation unit 12 focuses on the pixel value difference calculation frame having the same size as the circumscribing rectangle of the subject area set for the immediately preceding image. Set according to the position of the subject area of. Then, the pixel value difference calculation unit 12 calculates the difference in pixel values between the foreground and the background for each pixel in the subject area within the pixel value difference calculation frame. For example, in the pixel value difference calculation unit 12, the upper end of the pixel value difference calculation frame coincides with the upper end of the subject area, and the horizontal center of the pixel value difference calculation frame is the horizontal coordinate of the center of gravity of the subject area. Set the pixel value difference calculation frame so that they match. As a result, even if the subject area is not accurately detected in the image of interest, the pixel value difference calculation unit 12 can appropriately set the area in which the pixel value difference is calculated.
If the operator has not confirmed the subject area detected for the immediately preceding image, the pixel value difference calculation unit 12 may use the circumscribed rectangle itself of the subject area of the image of interest as the pixel value difference calculation frame. ..

In the present embodiment, the pixel value difference calculation unit 12 divides the subject area into a plurality of sub-areas. At that time, the pixel value difference calculation unit 12 determines, for example, the value of each pixel in the foreground of the subject area (that is, the subject in the image of interest) so that pixels having pixel values similar to each other are included in the same sub-region. The subject area is divided into sub-areas based on the value of each pixel in the area). In order to divide the subject area into sub-areas, the pixel value difference calculation unit 12 can use an area decomposition method such as a simple area expansion method. When the image generated by the camera 2 is a color image, the pixel value difference calculation unit 12 uses color information such as the Maximumly Stable Color Regions (MSCR) method as the domain decomposition method. May be used.

The pixel value difference calculation unit 12 calculates the absolute value of the difference between the pixel values of the foreground and the background for each pixel in the sub-region for each sub-region. The pixel value of the pixel in the foreground is the pixel value of the corresponding pixel in the image of interest, and the pixel value of the background pixel is the pixel value of the corresponding pixel in the background image. Further, in the present embodiment, the pixel value used for calculating the absolute value of the difference between the pixel values can be a luminance value. However, when the image generated by the camera 2 is a color image, the pixel value used for calculating the absolute value of the difference between the pixel values may be any color component, hue, or saturation value.

The pixel value difference calculation unit 12 calculates, for each sub-region, the average value of the absolute values of the pixel value differences of each pixel in the sub-region as a statistical representative value of the pixel value difference between the foreground and the background. ..

FIG. 3 is an explanatory diagram of an outline of pixel value difference calculation. As shown in FIG. 3, the subject area 310 is specified on the difference image 302 obtained by the background difference between the image 300 of interest and the background image 301. The subject area 310 is divided into a plurality of sub-regions 311 and a statistical representative value mdi of the difference in pixel values between the foreground and the background is calculated for each sub-region 311.

According to the modified example, the pixel value difference calculation unit 12 sets the mode or median of the absolute value of the difference in the brightness values of each pixel in the sub region as the mode between the foreground and the background for each sub region. It may be calculated as a statistical representative value of the difference in pixel values. Further, according to another modification, the pixel value difference calculation unit 12 determines either the average value, the median value, or the mode value of the absolute value of the difference in pixel values between the foreground and the background as the entire subject area. It may be calculated as a statistical representative value of the difference in pixel values between the foreground and the background.

The pixel value difference calculation unit 12 notifies the caution determination unit 14 of the statistical representative value of the difference in pixel values for each sub-region of the subject region of the image for each image. Further, when one statistical representative value is calculated for the entire subject area, the pixel value difference calculation unit 12 notifies the caution determination unit 14 of the statistical representative value calculated for the subject area.

The aspect ratio calculation unit 13 calculates the aspect ratio of the subject area with respect to the image in which the subject area is detected among the plurality of images to be subjected to the image recognition processing. Since the aspect ratio calculation unit 13 may perform the same processing on each of the plurality of images, the processing on one image will be described below.

The aspect ratio calculation unit 13 sets an extrinsic rectangle for the subject area, and calculates the ratio of the aspect width to the width of the extrinsic rectangle as the aspect ratio of the subject area. The range in which the aspect ratio of the subject (for example, a person) represented in the image can be taken is assumed in advance according to the shape of the subject. In addition, the aspect ratio of the subject area may change significantly in time series if a part of the subject is not included in the subject area for a certain image, or conversely, if the subject area includes an object other than the subject. .. Therefore, the time-series change in the aspect ratio of the subject area is one of the other features used to specify the image requiring attention.

FIG. 4 is an explanatory diagram of an outline of the aspect ratio calculation of the subject area. The circumscribed rectangle 402 is set so as to be in contact with the upper and lower ends and the left and right ends of the subject area 401 with respect to the subject area 401 on the image 400. In this case, the aspect ratio R of the subject area 401 is calculated as the ratio of the height H to the width W of the extrinsic rectangle 402. That is, if the coordinates of the pixel SP at the upper left end of the circumscribing rectangle 402 are (x1, y1) and the coordinates of the pixel EP at the lower right end are (x2, y2), the aspect ratio R is expressed by the following equation. ..

The aspect ratio calculation unit 13 notifies the caution determination unit 14 of the aspect ratio of the subject area of the image for each image.

The caution determination unit 14 determines the statistical representative value of the difference between the pixel values between the foreground and the background of the subject area and the subject area for the image in which the subject area is detected among the plurality of images to be subjected to the image recognition processing. Based on the aspect ratio, it is determined whether or not the image requires attention. Since the caution determination unit 14 may perform the same processing for each of the plurality of images, the processing for one image will be described below.

For example, the smaller the difference in pixel values between the subject displayed on the image and the background thereof, the more difficult it becomes to accurately detect the subject. Therefore, the smaller the statistical representative value of the difference between the pixel values between the foreground and the background of the subject area (for example, the average value, the median value, or the mode value of the absolute values of the difference), the more the detection fails for a part of the subject. On the contrary, there is a high possibility that an object other than the subject is erroneously detected as the subject. Further, when a part of the subject area includes an object other than the subject, the statistical representative value of the difference in pixel values between the foreground and the background may be small for the part.

Therefore, when a plurality of sub-regions are set for the subject region of the image of interest, the caution determination unit 14 statistics on the difference in pixel values between the foreground and the background calculated for each sub-region. The target representative value is compared with the pixel value difference threshold Th1. If the statistical representative value of any of the sub-regions is less than the pixel value difference threshold Th1, the subject region including the sub-region may not be detected correctly. Is determined. That is, there is a possibility that only a part of the subject is included in the subject area, or another object different from the subject is included in the subject area. Therefore, the attention-requiring determination unit 14 determines that the image of interest is a caution-required image. In this way, the caution determination unit 14 compares the statistical representative value of the difference in pixel values between the foreground and the background with the pixel value difference threshold Th1 for each sub-region, thereby forming another object in a part of the subject region. Even when is included, the image including the subject area can be specified as an image requiring attention.

When the statistical representative value of the difference between the pixel values between the foreground and the background is calculated for the entire subject area of the image of interest, the caution determination unit 14 determines that the statistical representative value is less than the pixel value difference threshold Th1. In that case, the image of interest may be determined to be an image requiring attention.

The pixel value difference threshold Th1 is, for example, the corresponding pixel of two background images obtained by the camera 2 shooting the shooting range at different timings when the subject does not exist in the shooting range of the camera 2. It can be three times the standard deviation σ of the difference value D (i, j) between them.

In addition, it is assumed that the shape of the subject does not change much between a plurality of images that are continuous in time. Therefore, in each image, when the subject area correctly represents the subject, it is assumed that the aspect ratio of the subject area does not change much between the plurality of images. On the other hand, if the subject area does not accurately represent the subject in any of the images, the aspect ratio of the subject area may change relatively significantly between the image and the images before and after the image.

Therefore careful determination unit 14, and the aspect ratio R _n of the subject region of interest to the image, along a time series of the aspect ratio difference Rv between the aspect ratio R _n-1 in the subject region of the previous interest to the image change It is calculated as a degree, and the degree of change Rv is compared with a predetermined shape threshold Th2. Then, if the degree of change Rv is larger than the predetermined shape threshold value Th2, the attention-requiring determination unit 14 determines the image of interest as the caution-required image. The degree of change Rv is calculated according to, for example, the following equation.

Further, the shape threshold value Th2 is set in advance according to the expected aspect ratio of the subject.

If the subject area is not detected in the image immediately before the image of interest, the degree of change in the aspect ratio of the subject area along the time series is not calculated, so the caution determination unit 14 needs attention to the image of interest. It may be determined as an image. Further, the attention-requiring determination unit 14 may not consider the image in which the subject area is not detected as the caution-required image.

For each image, the caution determination unit 14 determines the position and range of the detected circumscribing rectangle of the subject area, the statistical representative value of the difference between the pixel values for each sub-area in the subject area, the aspect ratio, and the caution. A flag indicating whether or not it is an image is written in the detection information table stored in the memory 5.

FIG. 5 is a diagram showing an example of the detection information table. In the detection information table 500, the detection information for one image is displayed for each row. That is, in the leftmost column of the detection information table 500, a frame number indicating the image generation order is shown. In the second column from the left, the coordinates SPn (x1, y1) of the upper left pixel of the circumscribing rectangle of the detected subject area and the coordinates EPn (x2, y2) of the lower right pixel are shown. In the third column from the left, statistical representative values (md1, md2, ...) of the difference in pixel values between the foreground and the background for each sub-domain in the subject area are shown. The aspect ratio Rn is shown in the second column from the right. And in the rightmost column, a flag indicating whether or not the image needs attention is shown.

The presentation unit 15 causes the user interface 4 to sequentially display each of the plurality of images to be subjected to the image recognition process, for example, in the order of generation time. At that time, the presentation unit 15 refers to the detection information table and displays an image that is not a caution image by superimposing the circumscribed rectangle of the subject area on the image. On the other hand, the presentation unit 15 refers to the detection information table and does not display the circumscribed rectangle of the subject area even if the subject area is detected for the image which is an image requiring attention. Instead, the presentation unit 15 may display a mark indicating that the image requires attention, for example, an arrow, in the vicinity of the subject area.

FIG. 6 is a diagram showing an example of an image displayed on the user interface 4. In this example, of the four images 600 to 603 obtained at times T0 to T3, only the image 603 obtained at time T3 is assumed to be a caution image. The images 600 to 603 are displayed one by one in the order of generation time, for example.

In this case, for the images 600 to 602, the circumscribed rectangles 610 to 612 of the detected subject area are displayed together with the images 600 to 602, respectively. As a result, the operator may confirm that the subject is correctly detected with reference to the extrinsic rectangles 610 to 612 for the images 600 to 602. The presentation unit 15 may receive an operation signal indicating that the operator has confirmed each of the images 600 to 602 from the user interface 4. Then, when the presentation unit 15 receives the operation signal, the presentation unit 15 may attach the identification information of the subject to the circumscribed rectangle. Alternatively, the operator may confirm the subject appearing in each image and add the subject identification information to each image. Further, for each same subject, subject data may be created in which the identification information of the subject, the number of images in which the subject is captured, and the detection information in each image are associated with each other.

On the other hand, for the image 603 which is an image requiring attention, the circumscribed rectangle of the subject area is not displayed. Instead, an arrow 613 indicating that the image needs attention is displayed in the vicinity of the subject area 620. Then, with respect to the image 603, the operator operates the user interface 4 to set the circumscribed rectangle of the subject area, and information representing the position and range of the set circumscribed rectangle is passed to the CPU 7. When the CPU 7 receives information representing the position and range of the circumscribed rectangle set by the user, the presentation unit 15 sets the circumscribed rectangle represented by the information as the true circumscribed rectangle of the subject area. Then, the presentation unit 15 may update the information indicating the position and range of the circumscribed rectangle of the image requiring attention 603 shown in the detection information table with the information indicating the position and range of the circumscribed rectangle set by the operator. good. Further, the operator may operate the user interface 4 to add the subject identification information to the set circumscribing rectangle.

According to the modified example, when the images that are not the images requiring attention are continuous, the presentation unit 15 may display the continuous images in fast forward display. Alternatively, the presentation unit 15 may display each subject area of the continuous image and its circumscribing rectangle at once. For example, the presentation unit 15 may cut out a subject area from each of the continuous images and combine them into one image to represent each subject area and the circumscribed rectangle of the continuous images in one image. Alternatively, the presentation unit 15 may reduce each of the continuous images to generate thumbnail images, and display the thumbnail images on the user interface 4 at the same time. At that time, the presentation unit 15 may display an extrinsic rectangle surrounding the subject area on each thumbnail image.

However, also in this modified example, it is preferable that the presentation unit 15 displays the attention-requiring image on the user interface 4 for each individual attention-requiring image. As a result, the operator can confirm whether or not the subject area detected for a plurality of images at once is correct for an image that is not a caution image, and on the other hand, the subject area can be easily viewed in a state where the attention image is easy to see. You can perform operations to set the circumscribed rectangle.

7 (a) and 7 (b) are diagrams showing an example of an image displayed on the user interface 4 according to this modified example. In this example, it is assumed that none of the three images 700 to 702 obtained at times T0 to T2 are images requiring attention.

In the example shown in FIG. 7A, the images 700 to 702 are fast-forwarded together with the outer frames 710 to 712 of the subject area. At that time, unless the operator performs an operation to stop the fast-forward display via the user interface 4, the presentation unit 15 considers that the subject area of each image is correct, and considers that the subject area of each image is correct for a certain period of time. The next image may be displayed after the display (for example, 1 to 5 seconds).

Further, in the example shown in FIG. 7B, the subject areas 720 to 722 detected from each of the images 700 to 702 are combined on one image 730, and the image 730 is displayed. Therefore, the operator can confirm that the subject areas of the three images 700 to 702 are correct only by checking one image 730.

FIG. 8 is an operation flowchart of the image recognition process executed by the CPU 7. The CPU 7 may execute the image recognition process for each image according to the following operation flowchart.

The subject detection unit 11 detects a subject area in which the subject to be detected is represented on the image (step S101).
The pixel value difference calculation unit 12 divides the subject area into a plurality of sub-areas (step S102). Then, the pixel value difference calculation unit 12 calculates a statistical representative value of the difference in pixel values between the foreground and the background for each sub-region (step S103).

Further, the aspect ratio calculation unit 13 calculates the aspect ratio of the subject area (step S104).

In the caution determination unit 14, the image becomes a caution image based on the statistical representative value of the difference in pixel values between the foreground and the background for each sub-region of the subject area or the degree of change in the aspect ratio along the time series. It is determined whether or not the condition is satisfied (step S105). When neither the statistical representative value of the difference in pixel values nor the degree of change in the aspect ratio along the time series satisfies the condition of the image requiring attention (step S105-No), the presentation unit 15 displays the image on the user interface 4. At the same time, the circumscribed rectangle of the subject area is displayed (step S106). On the other hand, when either the statistical representative value of the difference in pixel values or the degree of change of the aspect ratio along the time series satisfies the condition of the image requiring attention (step S105-Yes), the image of interest is the image requiring attention. be. Therefore, the presentation unit 15 causes the user interface 4 to display a mark indicating that the image needs attention together with the image (step S107). Then, the presentation unit 15 receives the setting information of the circumscribed rectangle of the subject area and the identification information of the subject via the user interface 4, sets the circumscribed rectangle of the subject area according to the setting information, and sets the circumscribed rectangle of the subject in the circumscribed rectangle of the subject. Identification information is attached (step S108).
After step S106 or S108, the CPU 7 ends the image recognition process. The CPU 7 may change the order of the processes of steps S102 and S103 and the processes of step S104, or may execute the processes of steps S102 and S103 and the processes of step S104 in parallel.

As described above, this image recognition device has a statistical representative value of the difference in pixel values between the foreground and the background of the subject area of the image of interest, and changes in the aspect ratio of the subject area along the time series. Calculate the degree. Then, in this image recognition device, whether or not the image of interest is an image requiring attention in which the subject area may be erroneously detected according to the statistical representative value or the degree of change of the aspect ratio along the time series. judge. Therefore, this image recognition device can automatically identify the image requiring attention when generating a set of teacher images for discriminator learning. Further, this image recognition device can reduce the trouble of correcting the circumscribed rectangle by not displaying the circumscribed rectangle of the subject area when the operator confirms the image requiring attention.

According to the modified example, the caution determination unit 14 has a statistical representative value of the difference in pixel values between the foreground and the background of the subject area, and the degree of change of the aspect ratio of the subject area along the time series. It may be determined whether or not the image of interest is an image requiring attention based on only one of the above. In this case, of the pixel value difference calculation unit 12 and the aspect ratio calculation unit 13, the one in which the calculated value is not used in the caution determination unit 14 may be omitted.

According to another modification, the aspect ratio calculation unit 13 may set the circumscribed ellipse of the subject area instead of setting the circumscribed rectangle of the subject area. Then, the aspect ratio calculation unit 13 may calculate the ratio of the diameter in the major axis direction to the diameter in the minor axis direction of the circumscribed ellipse as the aspect ratio.

According to still another modification, when a plurality of subject areas are detected from one image, the pixel value difference calculation unit 12, the aspect ratio calculation unit 13, and the caution determination unit 14 are described above for each subject area. The processing according to the embodiment or the modification may be performed. Then, when the determination condition that the image corresponds to the caution image is satisfied for any one of the plurality of subject areas, the caution determination unit 14 may use the image as the caution image.

According to still another modification, the presentation unit 15 may be omitted. In this case, for example, after the image recognition process for a series of images is completed, the operator refers to the detection information table and operates the user interface 4 to obtain the specified image requiring attention in the user interface 4. It may be displayed in. Then, the operator may refer to the displayed image requiring attention and operate the user interface 4 to add the circumscribed rectangle of the subject area and the identification information of the subject to the image requiring attention.

Further, a computer program for realizing the functions of each part of the CPU of the image recognition device according to the above embodiment or the modification in the computer may be provided in a form stored in a recording medium readable by the computer. The recording medium that can be read by a computer can be, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

All examples and specific terms given herein are intended for teaching purposes to help the reader understand the invention and the concepts contributed by the inventor to the promotion of the art. There is, and should be construed not to be limited to the constitution of any example herein, such specific cited examples and conditions relating to exhibiting superiority and inferiority of the present invention. Although embodiments of the present invention have been described in detail, it should be understood that various modifications, substitutions and modifications can be made thereto without departing from the spirit and scope of the invention.

1 Image recognition device 2 Camera 3 Communication interface 4 User interface 5 Memory 6 Storage medium access device 7 CPU
8 Storage medium 11 Subject detection unit 12 Pixel value difference calculation unit 13 Aspect ratio calculation unit 14 Attention judgment unit 15 Presentation unit

Claims (5)

The area where the subject is represented is detected from each of the plurality of time-series images obtained by the imaging unit.
Of the plurality of images, the region has an absolute value depending on the degree of change in the aspect ratio of the region along the time series or the absolute value of the difference between the foreground value and the background value of each pixel in the region. identify suspicious images that may have been erroneously detected,
An image other than the caution image among the plurality of images is displayed on the display device after superimposing the outer frame of the area in which the subject detected from the image is represented, while the plurality of images The image requiring attention is displayed on the display device without superimposing the outer frame of the area in which the subject is represented, which is detected from the image.
A computer program for image recognition that lets a computer do things. For each of the images in which the region in which the subject is represented among the plurality of images is detected, the region is divided into a plurality of sub-regions, and for each of the plurality of sub-regions, each pixel in the sub-region is obtained. Further let the computer perform the calculation of the statistical representative value of the absolute value of the difference between the foreground value and the background value.
To specify the image requiring attention, the statistical representative value for any sub-region of the plurality of sub-regions among the images in which the region in which the subject is represented is detected is less than the first threshold value. The computer program for image recognition according to claim 1, wherein the image is specified as the image requiring attention. For each of the images in which the region in which the subject is represented is detected among the plurality of images, the computer is further made to calculate the aspect ratio of the region.
To specify the image requiring attention, the absolute value of the difference between the aspect ratio of the image and the aspect ratio of the immediately preceding image in the time series among the images in which the area in which the subject is represented is detected is the first. The computer program for image recognition according to claim 1 or 2, which comprises specifying an image larger than the threshold value of 2 as the image requiring attention. A subject detection unit that detects a region in which a subject is represented from each of a plurality of time-series images obtained by the imaging unit, and a subject detection unit.
Of the plurality of images, the region has an absolute value depending on the degree of change in the aspect ratio of the region along the time series or the absolute value of the difference between the foreground value and the background value of each pixel in the region. A caution determination unit that identifies images requiring attention that may have been erroneously detected,
An image other than the caution image among the plurality of images is displayed on the display device after superimposing the outer frame of the area in which the subject detected from the image is represented, while the plurality of images A presentation unit that displays the image requiring attention on the display device without superimposing the outer frame of the area in which the subject is represented detected from the image.
An image recognition device having. The area where the subject is represented is detected from each of the plurality of time-series images obtained by the imaging unit.
Of the plurality of images, the region has an absolute value depending on the degree of change in the aspect ratio of the region along the time series or the absolute value of the difference between the foreground value and the background value of each pixel in the region. identify suspicious images that may have been erroneously detected,
An image other than the caution image among the plurality of images is displayed on the display device after superimposing the outer frame of the area in which the subject detected from the image is represented, while the plurality of images The image requiring attention is displayed on the display device without superimposing the outer frame of the area in which the subject is represented, which is detected from the image.
Image recognition method including that.

Images