JP2021077091A - Image processing device and image processing method - Google Patents

Abstract

To provide a technique capable of precisely detecting an object even when the object to be detected in an image is small.SOLUTION: The image processing device includes: a setting unit that sets multiple detection target areas where to be tried to detect objects in an acquired captured image; a conversion unit that converts the images at least multiple detection target areas out of the captured images from a color image into a gray scale image; and an object detection unit that has multiple channels for independently inputting the gray scale images in each of the multiple detection target areas, and outputs the detection result of the object based on the detection processing of the object which is made on each of the channels.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus and an image processing method.

Conventionally, an object such as a face in an image has been detected by using a neural network (see, for example, Patent Document 1). In recent years, the development of an object detection method using a multi-layer neural network (DNN: Deep Neural Network) has been active.

Japanese Unexamined Patent Publication No. 2012-48476

By the way, in the object detection using the DNN, in order to match the size of the input layer of the DNN, an image size reduction process (a process of lowering the resolution) may be performed on the captured image taken by the camera. For example, in a situation where the calculation resources for automobiles are limited, it is common to input the reduced image to the DNN and perform the object detection process.

For example, when the size of the object to be detected in the captured image is small, or when the object to be detected is apparently small because it exists in a distant place, the feature amount is lost by the image size reduction process. There is. For this reason, if the image captured by the camera is simply reduced and the object detection process using the DNN is performed, the accuracy of the object detection may decrease.

In view of the above problems, it is an object of the present invention to provide a technique capable of accurately detecting an object even when the object to be detected in the image is small.

In order to achieve the above object, the image processing apparatus of the present invention includes a setting unit for setting a plurality of detection target areas for attempting to detect an object in the acquired captured image, and at least the plurality of detection target regions of the captured image. It has a conversion unit that converts an image from a color image to a grayscale image, and a plurality of channels for separately inputting the grayscale image of each of the plurality of detection target areas, and the object detection process performed for each channel. The configuration (first configuration) includes an object detection unit that outputs the detection result of the object based on the above.

Further, in the image processing device having the first configuration, it is preferable that the setting unit has a configuration (second configuration) in which the plurality of detection target regions are set for a part of the captured image. ..

Further, in the image processing apparatus having the first or second configuration, the object detection unit integrates the processing results for each channel obtained in the coordinate system of each detection target region into the coordinate system of the entire captured image. It is preferable that the configuration is such that the detection result of the object is output (third configuration).

Further, in the image processing apparatus having any of the first to third configurations, the resolution of the grayscale image input to each of the channels is changed according to the position where the detection target area is set. (Fourth configuration).

Further, in the image processing apparatus having any of the first to fourth configurations, the plurality of detection target regions are three and the plurality of channels are three (fifth configuration). Is preferable.

Further, the image processing device having any of the first to fifth configurations is currently acquired the captured image based on the detection result of the object by the object detection unit with respect to the captured image previously acquired. A search range may be set for the object, and a tracking unit for tracking the object within the search range may be further provided (sixth configuration).

Further, in the image processing apparatus having the sixth configuration, the search range may be changed based on the locus information indicating the past movement of the object (seventh configuration).

Further, in the image processing apparatus having any of the first to seventh configurations, the setting unit sets a region having a large amount of change as the detection target region by comparing the captured image with the background image prepared in advance. It may be a configuration to be set (eighth configuration).

Further, in order to achieve the above object, the image processing method of the present invention includes a setting step of setting a plurality of detection target areas for attempting to detect an object in the acquired captured image, and at least the plurality of detection targets of the captured image. In the conversion step of converting the image of the region from the color image to the grayscale image, and the grayscale image of each of the plurality of detection target regions is input to different channels, and the object detection process performed for each channel is performed. Based on this, the configuration (9th configuration) includes an object detection step for outputting the detection result of the object.

According to the present invention, even if the object to be detected in the image is small, the object can be detected with high accuracy.

The figure which shows the structure of the image processing apparatus which concerns on 1st Embodiment The figure which shows the setting example of a plurality of detection target areas by a setting part The figure which shows another setting example of a plurality of detection target areas by a setting part Schematic diagram for explaining the function of the object detection unit A flowchart showing an operation example of the image processing apparatus according to the first embodiment. The figure which shows an example of the photographed image acquired by the acquisition part The figure which shows the setting example of the detection target area Diagram exemplifying the detection result of an object The figure which shows the structure of the image processing apparatus which concerns on 2nd Embodiment Diagram for explaining the function of the tracking unit Diagram for explaining the change of the search range based on the trajectory information The figure for demonstrating the operation example of the image processing apparatus which concerns on 2nd Embodiment

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

<1. First Embodiment>
(1-1. Configuration of image processing device)
FIG. 1 is a diagram showing a configuration of an image processing device 1 according to a first embodiment of the present invention. Note that FIG. 1 shows only the components necessary for explaining the features of the image processing device 1 of the first embodiment, and the description of general components is omitted. Further, FIG. 1 also shows a camera 2 which is a component different from the image processing device 1 in order to facilitate understanding.

The image processing device 1 may be mounted on a moving body such as a vehicle. Vehicles include a wide range of vehicles with wheels, such as automobiles, trains, and automatic guided vehicles. The image processing device 1 may be included in an in-vehicle device such as a navigation device or a drive recorder mounted on a vehicle, for example. The image processing device 1 does not have to be mounted on a moving body, and may be arranged in a building such as a monitoring facility provided in a commercial facility or a parking lot, or a tollhouse on an expressway, for example. Further, the image processing device 1 may be included in, for example, a server device such as a cloud server provided so as to be able to communicate with a terminal device such as an in-vehicle device via a network or the like. Further, the image processing device 1 may be included in a mobile terminal such as a smartphone or a tablet, for example.

The camera 2 may be mounted on a moving body such as a vehicle, or may be fixedly arranged in a building such as a commercial facility or outdoors such as a parking lot. The camera 2 outputs a captured image (captured image) to the image processing device 1, for example, by wire, wirelessly, or by using a network.

As shown in FIG. 1, the image processing device 1 includes an acquisition unit 11, a control unit 12, and a storage unit 13.

The acquisition unit 11 acquires a captured image. The acquisition unit 11 continuously acquires analog or digital captured images from, for example, a camera 2 mounted on a vehicle at a predetermined cycle (for example, a 1/30 second cycle). An aggregate of captured images (one frame image) acquired by the acquisition unit 11 is a moving image captured by the camera 2. In the present embodiment, the captured image acquired by the acquisition unit 11 is a color image. When the acquired captured image is analog, the acquisition unit 11 converts the analog captured image into a digital captured image (A / D conversion). The acquisition unit 11 outputs the acquired captured image (or the converted image if A / D conversion is performed) to the control unit 12.

The control unit 12 is a controller that comprehensively controls the entire image processing device 1. The control unit 12 is configured as a computer including, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, which are hardware processors.

The storage unit 13 is composed of, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, or a storage device using a portable recording medium such as an optical disk. The storage unit 13 stores a program as firmware and various data. In the present embodiment, the storage unit 13 stores the learned model used by the object detection unit 123, which will be described later. The trained model can be obtained by known deep learning (deep learning) using, for example, a CNN (Convolutional Neural Network) or the like.

The setting unit 121, the conversion unit 122, and the object detection unit 123 shown in FIG. 1 are realized by the CPU of the control unit 12 executing arithmetic processing according to a program stored in the storage unit 13. It is a function. In other words, the image processing device 1 includes a setting unit 121, a conversion unit 122, and an object detection unit 123.

In addition, at least one of the setting unit 121, the conversion unit 122, and the object detection unit 123 in the control unit 12 is an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a GPU (Graphics Processing Unit). ) Etc. may be configured.

Further, the setting unit 121, the conversion unit 122, and the object detection unit 123 are conceptual components. The functions executed by one component may be distributed to a plurality of components, or the functions of the plurality of components may be integrated into one component. Further, the acquisition unit 11 may be configured to be realized by the CPU of the control unit 12 performing arithmetic processing according to a program. Further, with respect to the specific hardware configuration of the image processing device 1, the components may be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 12 may include a plurality of hardware processors.

The setting unit 121 sets a plurality of detection target areas for attempting to detect an object in the acquired captured image. The captured image in which a plurality of detection target areas are set may be the image itself obtained from the acquisition unit 11, or may be an image obtained from the acquisition unit 11 and then processed. Examples of the processing to be performed include grayscale conversion. Further, the object may be a moving body or a stationary object. Specifically, the object is, for example, a person, a vehicle, or the like. The object may be a part that is understood as a part of something, such as a human face or a vehicle license plate.

FIG. 2 is a diagram showing a setting example of a plurality of detection target areas 31 by the setting unit 121. In the example shown in FIG. 2, the setting unit 121 sets a plurality of detection target areas 31 for a part of the range 30 of the captured image 3. According to this, the object is detected by targeting a part of the captured image 3, and the burden of the object detection process is reduced as compared with the case where the object is detected by targeting the entire range of the captured image 3. Can be done.

A part of the range 30 is determined in consideration of, for example, the purpose of detecting an object, the type of an object to be detected, and the like. For example, when it is decided how many meters away from the own vehicle to detect an object, or when it is known that the range of the object appearing in the captured image is a limited range, a part of the range is used. 30 is a specific distance range from the camera 2. In such a case, a part of the range 30 can be obtained by using a known parameter regarding the set position of the camera 2 and triangulation.

In the example shown in FIG. 2, the plurality of detection target areas 31 are obtained by dividing a part of the range 30. Specifically, the plurality of detection target areas 31 are obtained by evenly dividing a part of the rectangular range 30 into three. That is, in the example shown in FIG. 2, the plurality of detection target areas 31 are three. Each of the three detection target areas 31 has a rectangular shape and is arranged in the left-right direction of the captured image 3. The three detection target areas 31 are composed of a first detection target area 31a located at the left end, a second detection target area 31b located in the middle, and a third detection target area 31c. The sizes of the three detection target regions 31a, 31b, and 31c are preferably the same as each other, but may be different from each other in some cases.

The shape of the detection target area 31 set by the setting unit 121 is not limited to the rectangular shape, and may be changed as appropriate. The setting unit 121 may set the plurality of detection target areas 31 not only for a part of the range 30 of the captured image 3 but for the entire range. That is, the entire range of the captured image 3 may be divided into a plurality of regions, and each divided region may be used as the detection target region 31. The plurality of detection target areas 31 set by the setting unit 121 may be fixed areas, but may be, for example, areas that are changed according to the steering angle of the vehicle when the camera 2 is an in-vehicle camera.

FIG. 3 is a diagram showing another setting example of the plurality of detection target areas 31 by the setting unit 121. As shown in FIG. 3, the plurality of detection target areas 31 (specifically, the three detection target areas 31) may occupy only a part of a part of the range 30 of the captured image 3. The plurality of detection target areas 31 do not have to be adjacent to each other.

Further, the plurality of detection target areas 31 may be obtained without setting a part of the range 30. For example, the setting unit 121 may set a region having a large amount of change as the detection target region 31 by comparing the captured image 3 with the background image prepared in advance. According to this, it is possible to efficiently perform the object detection process by focusing on the area where the object is likely to exist.

In the case of this configuration, object detection (for example, classification) is performed by a learned model (for example, CNN or the like) in which machine learning is performed on a region where the amount of change is large. The background image is stored in the storage unit 13. For the purpose of appropriately obtaining the type of the object by using the comparison with the background image, the processing of the object detection may be the same as the processing of the object detection unit 123 described below, but it is the same as the processing of the object detection unit 123. It may be a different process.

The background image may be an image obtained by a known method (for example, a difference method). However, the background image may be an image obtained by a method other than the following known methods.

The background image generator (not shown) that generates a background image detects an object using a trained model for each of a plurality of images obtained at different timings. The plurality of images obtained at different timings may be images obtained from one camera, or may be, for example, images obtained from a plurality of connected cars equipped with cameras. The plurality of images are photographed images showing the same place.

The trained model used by the background image generator may be a neural network capable of detecting an object such as a CNN. The trained model may be configured to perform image segmentation that labels the meaning for each pixel, or may be configured to classify the types of objects.

The background image generator generates a deducted image obtained by removing the region where the object is detected for each of the plurality of images obtained at different timings. The background image generator completes a background image in which no object exists by combining the obtained plurality of subtracted images. The background image generation device preferably generates a background image at regular intervals for the background image at the same location.

According to such a background image generation device, it is possible to generate an accurate background image in order to generate a background image using a trained model that has undergone machine learning. If the background image generated by the background image generation device is used, for example, a shadow whose shape changes with time can be easily distinguished from an object, and the object can be appropriately detected.

The conversion unit 122 converts at least a plurality of images in the detection target area 31 among the captured images 3 obtained from the acquisition unit 11 from a color image to a grayscale image. The conversion unit 122 may convert the entire captured image into a grayscale image, but may convert only a plurality of detection target areas 31 into a grayscale image. For example, in the example shown in FIG. 2, the images of the three detection target areas 31a to 31c are converted into grayscale images. The conversion method to a grayscale image may be, for example, an NTSC weighted average method, a method of extracting one element value of RGB and adopting it as a grayscale value, or the like. By grayscale conversion by the conversion unit 122, a grayscale image of 256 gradations (8 bits) can be obtained in each detection target area 31.

The object detection unit 123 performs object detection processing using a learned model (CNN as an example) after machine learning stored in the storage unit 13. The object detection process includes a part for extracting features from an image by convolution and pooling, and a part for classifying based on the features extracted by repeating a fully connected layer.

The trained model used in the object detection unit 123 is, as a preferred form, a trained model in which training is performed using a grayscale image as teacher data. Therefore, the grayscale image can accurately detect the object.

FIG. 4 is a schematic diagram for explaining the function of the object detection unit 123. The object detection unit 123 has a plurality of channels for separately inputting grayscale images of each of the plurality of detection target areas 31. Channels are paraphrased as input layers. In the present embodiment, the grayscale image 31aG of the first detection target area 31a is input to 1ch. The grayscale image 31bG of the second detection target area 31b is input to 2ch. The grayscale image 31cG of the third detection target area 31c is input to 3ch. That is, in the present embodiment, the plurality of channels is three.

In the present embodiment, the upper limit of the image size (resolution) that can be input in each channel (1ch, 2ch, 3ch) is the same as each other and is a fixed value. The grayscale images 31aG, 31bG, and 31cG of the detection target areas 31a, 31b, and 31c are input to each channel with the same resolution or reduced resolution depending on the input allowable size of each channel. In this embodiment, the size of the image input to each channel is the same.

The resolutions of the grayscale images 31aG, 31bG, and 31cG input to each channel (1ch, 2ch, 3ch) are preferably changed according to the position where the detection target area 31 is set. When the detection target area 31 is set at a position far from the camera 2, it is preferable not to reduce the resolution, or when it is necessary to reduce the resolution, it is preferable to reduce the degree of reduction as much as possible. When the detection target area 31 is set closer to the camera 2, it is preferable to reduce the resolution as much as possible within a range in which the detection accuracy of the object does not decrease. With such a configuration, when it is necessary to detect an object existing in a distant place, it is possible to suppress a decrease in the detection accuracy of the object. On the other hand, when it is necessary to detect an object existing in the vicinity, the processing load of the object detection process can be reduced.

The object detection unit 123 outputs an object detection result based on the result of the object detection process performed for each channel. As shown in FIG. 4, object detection processing using the trained model (DNN) is performed on each of the grayscale images 31aG, 31bG, and 31cG input to each channel (1ch, 2ch, 3ch). .. Object detection processing for each channel proceeds in parallel. The object detection unit 123 once obtains the object detection processing result for each of the detection target areas 31a, 31b, and 31c.

As shown in FIG. 4, the object detection unit 123 performs an integrated process for integrating the results of the object detection process for each channel. Specifically, the object detection unit 123 integrates the processing results for each channel obtained in the coordinate systems of the detection target areas 31a, 31b, and 31c into the coordinate system of the entire captured image 3 and outputs the object detection result. .. Each of the detection target areas 31a, 31b, and 31c is an image cut out from the captured image 3, and the coordinate region occupied in the captured image 3 is known. Therefore, the detection area of the object obtained in each channel can be converted into the coordinates of the entire captured image 3. When an object exists over a plurality of detection target areas 31, the objects may be detected in duplicate in each detection target area 31, and the detected areas of the duplicated detected objects are combined. There is a need to.

The object detection result that integrates the processing results of each channel is, for example, an image in which a bounding box surrounding the detected object area is added to the captured image (color image). For example, the image is output to a display device (not shown), and an image with a bounding box indicating the detection of an object is displayed on the screen of the captured image 3.

In some cases, the object detection unit 123 may output the results of the object detection processing for each channel separately without integrating them. However, if the results of the object detection processing for each channel are integrated as in the present embodiment, it is possible to make it easier to recognize the object detection result in the entire captured image.

In the conventional configuration in which a color image is input to a trained model (CNN) that detects an object, for example, a color photographed image is decomposed into RGB components, and the three decomposed components are separated into separate luminance channels (Rch,). It is input to Gch, Bch) and the object detection process is performed. In this respect, in the present embodiment, the image input to the trained model is a grayscale image. In the grayscale image, since only one channel is used, two channels are left over when a trained model having the same configuration as the above-mentioned conventional configuration is assumed.

Therefore, in the present embodiment, the three input channels are effective as a configuration in which the grayscale images 31aG, 31bG, and 31cG of the three detection target areas 31a, 31b, and 31c obtained from the captured image 3 are input to different channels, respectively. I am using it. That is, the present invention can be realized by applying a conventional trained model. From this point of view, for example, the grayscale image 31aG of the first detection target area 31a is input to R (Red) ch, which is one of the luminance channels, and the grayscale image 31bG of the second detection target area 31b is It is input to G (Green) ch, which is one of the luminance channels, and the grayscale image 31cG of the third detection target region 31c is input to B (Blue) ch, which is one of the luminance channels. In other words, in the present embodiment, the plurality of channels are RGB three channels.

According to the present embodiment, when an object is detected from one captured image 3, the image can be grayscaled so that the image can be divided into a plurality of channels and input. The image input to each channel is an image obtained by dividing a part of the captured image, and the input size (number of pixels) of the image to each channel can be reduced. As a result, the degree to which the resolution of the image input to each channel (the upper limit of the input size is fixed) is lowered can be reduced, and the loss of the feature amount of the small object displayed in the image can be suppressed. That is, according to the present embodiment, for example, even if the object to be detected in the captured image 3 is a small object such as a face or a license plate, the object can be detected with high accuracy.

Further, since the trained model of the present embodiment can have the same configuration as the trained model in which the conventional color image is input and the object is detected, the processing load becomes extremely large as compared with the conventional configuration. It is possible to avoid the demand for high-performance processing equipment.

(1-2. Operation example of image processing device)
FIG. 5 is a flowchart showing an operation example of the image processing device 1 according to the first embodiment of the present invention. The image processing device 1 operates the flowchart shown in FIG. 5 every time a captured image is acquired by, for example, the acquisition unit 11.

In step S1, the acquisition unit 11 acquires the captured image 3 from the camera 2. The acquisition unit 11 acquires, for example, a captured image 3 as shown in FIG. The photographed image 3 shown in FIG. 6 shows two people H walking along the wall W arranged on the side of the road R. The two persons H are an adult man and a girl, and hereinafter, an adult man may be referred to as a person H1 and a girl may be referred to as a person H2. When the acquisition unit 11 acquires the captured image, the process proceeds to the next step S2.

In step S2, the setting unit 121 sets three detection target areas 31a, 31b, and 31c in the captured image 3. For example, when the object to be detected is a face and the captured image 3 shown in FIG. 6 is acquired, the three detection target areas 31a, 31b, and 31c shown by the broken lines in FIG. 7 are set in the captured image 3. The detection target areas 31a, 31b, and 31b are obtained, for example, by dividing the range 30 in which the face F of the person H is desired to be detected in the captured image 3 into three evenly in the left-right direction. When the setting of the detection target areas 31a, 31b, and 31c by the setting unit 121 is completed, the process proceeds to the next step S3.

In step S3, the conversion unit 122 converts each of the three detection target areas 31a, 31b, and 31c images (color images) into grayscale images. The process of step S3 may be performed before the process of step S2. In this case, after the entire captured image 3 is converted into a grayscale image, three detection target regions 31a, 31b, and 31c may be set in the grayscale captured image. When the grayscale conversion by the conversion unit 122 is completed, the process proceeds to the next step S4.

In step S4, the grayscale images of the detection target areas 31a, 31b, and 31c are input to the separate channels (1ch, 2ch, 3ch) of the trained model (CNN). The grayscale image input to each channel is reduced in resolution as needed. In the example shown in FIG. 7, a grayscale image showing only the wall W is input to 1ch. A grayscale image showing the face F of the person H1 and a part of the face F of the person H2 is input to 2ch. A grayscale image showing a part of the face F of the person H2 is input to 3ch. When the input of the grayscale image to each channel is completed, the process proceeds to the next step S5.

In step S5, object detection processing using the trained model (CNN) is performed for each channel (1ch, 2ch, 3ch). In the example shown in FIG. 7, an object (here, face F) is not detected in 1ch, a face F between a person H1 and a person H2 is detected in 2ch, and a face F of a person H2 is detected in 3ch. When all the object detection processes for each channel are completed, the process proceeds to the next step S6.

In step S6, the results of the detection processing for each channel (1ch, 2ch, 3ch) are integrated, and the detection result of the object in the captured image 3 is output. The detection results for each channel obtained in the coordinate systems of the detection target areas 31a, 31b, and 31c are converted into the results of the coordinate system of the entire captured image 3, and the processing results are integrated.

In the example shown in FIG. 7, the face F of the person H2 is detected in duplicate on 2ch and 3ch. When the coordinate system is transformed and the processing results of each channel are integrated, the objects judged to be detected in duplicate are combined. Whether or not the objects are detected in duplicate can be determined based on, for example, the distance between the detected objects when converted to the coordinate system of the entire captured image 3. For example, when it is determined that the detected objects overlap, or when it is determined that the distance between the detected objects is extremely short, it is determined that the detected objects overlap. In addition, additional information (for example, age, gender, etc.) that can be obtained at the time of object detection may also be referred to in the duplication determination.

FIG. 8 is a diagram illustrating an object detection result. FIG. 8 shows the result of proceeding with the processing of the example shown in FIG. A bounding box B indicating object detection is attached to the face F of the person H1 detected only on 2ch. For the face F of the person H2 detected in both 2ch and 3ch, the results of both channels are combined, and one bounding box B is given to the face F of the person H2. When the object detection result is output, the processing of the flowchart shown in FIG. 5 is temporarily terminated. By acquiring the next frame image, the processing of the flowchart shown in FIG. 5 is restarted.

<2. Second Embodiment>
Next, the image processing apparatus according to the second embodiment will be described. In the description of the image processing apparatus of the second embodiment, the description of the portion overlapping with the first embodiment will be omitted unless it is particularly necessary to explain.

(2-1. Configuration of image processing device)
FIG. 9 is a diagram showing a configuration of an image processing device 1A according to a second embodiment of the present invention. Note that FIG. 9 shows only the components necessary for explaining the features of the image processing apparatus 1A of the second embodiment, and the description of general components is omitted. Further, FIG. 9 also shows a camera 2 which is a component different from the image processing device 1A for easy understanding.

As shown in FIG. 9, the image processing device 1A includes an acquisition unit 11, a control unit 12A, and a storage unit 13. Since the acquisition unit 11 and the storage unit 13 are the same as those in the first embodiment, the description thereof will be omitted.

The control unit 12A is a controller that comprehensively controls the entire image processing device 1A as in the first embodiment. The control unit 12A is configured as a computer including, for example, a CPU, a RAM, a ROM, and the like. However, the control unit 12A has a function different from that of the first embodiment. The setting unit 121, the conversion unit 122, the object detection unit 123, and the tracking unit 124 shown in FIG. 9 are realized by the CPU of the control unit 12A executing arithmetic processing according to a program stored in the storage unit 13. This is a function of the control unit 12A. In other words, the image processing device 1A includes a setting unit 121, a conversion unit 122, an object detection unit 123, and a tracking unit 124.

In addition, at least one of each unit 121 to 124 of the control unit 12A may be configured by hardware such as ASIC, FPGA, and GPU. Further, each part 121 to 124 is a conceptual component. The functions executed by one component may be distributed to a plurality of components, or the functions of the plurality of components may be integrated into one component.

Since the configurations of the setting unit 121, the conversion unit 122, and the object detection unit 123 are the same as those in the first embodiment, the description thereof will be omitted.

For the purpose of improving the real-time property of object detection, the object detection unit 123 of the second embodiment does not necessarily have to perform the object detection process of the first embodiment. The object detection unit 123 may be provided with an algorithm that can detect an object from the captured image 3 in a global manner. The object detection unit 123 may have a known configuration for detecting an object using a learned model (CNN or the like) obtained by deep learning. In this case, the setting unit 121 and the conversion unit 122 may not be provided.

The tracking unit 124 includes an algorithm that can detect an object locally and detect the object at a higher speed than the object detection unit 123. The tracking unit 124 replaces the object detection unit 123 to detect an object. Specifically, the tracking unit 124 sets a search range for the currently acquired captured image 3 based on the detection result of the object of the object detection unit 123 with respect to the previously acquired captured image 3. Then, the tracking unit 124 tracks the object within the search range. The tracking unit 124 can track the object by reducing the processing load in order to track the object in the range in which the object is likely to be detected in the captured image 3.

FIG. 10 is a diagram for explaining the function of the tracking unit 124. In FIG. 10, the frame B shown by the thick broken line is a bounding box showing the position of the object detected by the object detection unit 123 in the frame image one frame before the present, which is conveniently superimposed on the current frame image 3. Is.

The tracking unit 124 sets the search range 40 for the current frame image 3 based on the position of the bounding box B obtained in the frame image one frame before. The search range 40 is set to surround the bounding box B in the previous frame image, for example, in consideration of the possibility that the object may move. That is, the search range 40 is set to be larger than the bounding box B in the previous frame image. Depending on the type of the object to be detected, for example, the range in which the object can move from the shooting timing one frame before to the shooting timing of the current frame differs. Therefore, it is preferable that the search range 40 is changed depending on the type of the object to be detected. For example, the search range may be set wider when the detection target is a license plate than when the detection target is a human face.

When a plurality of objects are detected in the frame image one frame before the present, the tracking unit 124 sets the search range 40 for each of the plurality of objects. In the example shown in FIG. 10, since the face F of the person H1 and the face F of the person H2 are detected by the object detection unit 123, the search range 40 is set for each of the two faces F. ing.

Further, the search range 40 may be changed based on the locus information indicating the movement of the past object. The locus information can be obtained, for example, by connecting the center positions of the bounding boxes B of the same object in a plurality of past frames. FIG. 11 is a diagram for explaining the change of the search range 40 based on the locus information. In the example shown in FIG. 11, it is expected that the moving direction of the object (face) is the direction of the thick arrow X from the locus information. In consideration of the prediction based on this locus information, the search range 40 is set wide and biased in the direction of the thick arrow X with reference to the bounding box B. This is because it is considered unlikely that the object will move in the direction opposite to the thick arrow X. When the direction of the arrow X changes, the search range 40 is changed.

In this way, if the search range 40 is not set uniformly but is changed according to the trajectory information, the range for searching the object for tracking the object is narrowed down to the range where the object is likely to exist. It is possible to increase the processing speed of the tracking process. In the example shown in FIG. 11, only the tendency in the moving direction is extracted from the locus information and the search range 40 is changed, but this is only an example. For example, the search range 40 may be set by extracting the tendency of the moving speed in addition to the moving direction from the locus information and taking the moving speed into consideration in the moving direction.

The tracking unit 124 tracks an object by, for example, template matching. The tracking unit 124 obtains a template image of an object from, for example, the result of object detection by the object detection unit 123 one frame before. Then, the tracking unit 124 searches for an image having the same pattern as the template image within the search range 40. When the tracking unit 124 finds a pattern having a similarity equal to or higher than a threshold value, the tracking unit 124 detects the region as an object to be tracked. For example, when the tracking unit 124 succeeds in tracking an object, the tracking unit 124 applies a bounding box B to the position of the detected object in the same manner as the object detection unit 123.

The tracking unit 124 may perform template matching by scaling the template image according to the trajectory information. For example, when it is determined from the trajectory information that the object is approaching the camera 2, the template image is enlarged. On the other hand, when it is determined from the trajectory information that the object is away from the camera 2, the template image is reduced. Further, the tracking unit 124 may track an object within the search range 40 by another tracking method such as KCF (Kernelized Correlation Filter), instead of the method using the above-mentioned template matching.

(2-2. Operation example of image processing device)
FIG. 12 is a diagram for explaining an operation example of the image processing device 1A according to the second embodiment. In FIG. 12, “In” indicated by the broken line arrow indicates the timing at which the image processing device 1A acquires the captured image 3 from the camera 2. As shown in FIG. 12, the timing of acquiring the captured image 3 occurs in a predetermined cycle (for example, 1/30 second).

In FIG. 12, the thick arrow indicates the state in which the process is being executed. In the example shown in FIG. 12, when the first captured image 3 is acquired, the object detection unit 123 performs the object detection process. Specifically, as in the case of the first embodiment (see FIG. 5), the processing by the setting unit 121 and the conversion unit 122 is executed before the processing of the object detection unit 123 is performed. Therefore, in FIG. 12, the thick arrow indicating the state in which the processing of the object detection unit 123 is being executed includes the processing by the setting unit 121 and the conversion unit 122.

While the process by the object detection unit 123 is being executed, the process by the tracking unit 124 is not executed. When the object detection by the object detection unit 123 is completed, the processing by the tracking unit 124 is executed. While the processing by the tracking unit 124 is being executed, the processing by the object detection unit 123 is not executed. That is, the object detection unit 123 and the tracking unit 124 operate alternately.

In the present embodiment, the object detection unit 123 and the tracking unit 124 alternately perform processing for each frame. However, this is an example. For example, after the processing of the frame image by the object detection unit 123 is completed, the processing by the tracking unit 124 may be performed on the subsequent two or more frame images. In this case, the tracking unit 124 sets a search range for the current frame image 3 based on the tracking result of the tracking unit 124 for the frame image 3 one frame before, and tracks the object within the search range. Sometimes.

Detection of an object using a trained model trained by deep learning has a large processing load and may take a long time to process. When trying to improve the detection accuracy of an object, the processing time tends to be long. In the example shown in FIG. 12, the processing time of the object detection unit 123 is long, and the object detection process by the object detection unit 123 in the current frame image 3 is not completed by the time the next frame image 3 is acquired.

However, when the object detection process by the object detection unit 123 is completed, the object tracking process by the tracking unit 124, which has a high processing speed, is performed on the next frame image 3. The process by the tracking unit 124 is completed by the time the next frame image 3 is acquired. That is, when viewed in units of two frames, the detection of the object for each frame is completed by the time the next frame is acquired. When the processing by the tracking unit 124 is completed, the processing by the object detection unit 123 is performed again, and the processing by the object detection unit 123 and the processing by the tracking unit 124 are alternately repeated.

According to the present embodiment, after the processing by the object detection unit 123 aiming at the improvement of the detection accuracy is performed, the processing by the tracking unit 124 aiming at the improvement of the processing speed is performed, and the alternating processing is repeated. Therefore, according to the present embodiment, it is possible to suppress the deterioration of the real-time property of the object detection while improving the detection accuracy of the object.

<3. Points to note>
The various technical features disclosed herein can be modified in addition to the above embodiments without departing from the gist of the technical creation. That is, it should be considered that the above embodiments are exemplary in all respects and are not restrictive, and the technical scope of the present invention is not the description of the above embodiments but the claims. It is shown and should be understood to include all modifications that fall within the meaning and scope of the claims. In addition, a plurality of embodiments and modifications shown in the present specification may be appropriately combined and implemented to the extent possible.

1, 1A ... Image processing device 31 ... Detection target area 40 ... Search range 121 ... Setting unit 122 ... Conversion unit 123 ... Object detection unit 124 ... Tracking unit

Claims (9)

A setting unit that sets multiple detection target areas that try to detect an object in the acquired captured image,
A conversion unit that converts at least the images of the plurality of detection target areas among the captured images from a color image to a grayscale image, and
An object detection unit that has a plurality of channels for separately inputting the grayscale image of each of the plurality of detection target areas and outputs the detection result of the object based on the detection processing of the object performed for each channel.
An image processing device. The image processing device according to claim 1, wherein the setting unit sets a plurality of detection target areas for a part of a range of the captured image. Claim 1 or 2 that the object detection unit integrates the processing result for each channel obtained in the coordinate system of each detection target area into the coordinate system of the entire captured image and outputs the detection result of the object. The image processing apparatus according to. The image processing apparatus according to any one of claims 1 to 3, wherein the resolution of the grayscale image input to each of the channels is changed according to a position where the detection target area is set. The plurality of detection target areas are three.
The image processing apparatus according to any one of claims 1 to 4, wherein the plurality of channels are three. Based on the detection result of the object of the object detection unit for the captured image acquired earlier, a search range is set for the captured image currently acquired, and the tracking of the object is performed within the search range. The image processing apparatus according to any one of claims 1 to 5, further comprising a tracking unit for performing the image processing. The image processing apparatus according to claim 6, wherein the search range is changed based on trajectory information indicating the movement of the object in the past. The image processing apparatus according to any one of claims 1 to 7, wherein the setting unit sets a region having a large amount of change as the detection target region by comparing the captured image with a background image prepared in advance. .. A setting process that sets multiple detection target areas that try to detect an object in the acquired captured image, and
A conversion step of converting at least the images of the plurality of detection target areas among the captured images from a color image to a grayscale image, and
An object detection step of inputting the grayscale images of each of the plurality of detection target areas to different channels and outputting the detection result of the object based on the detection process of the object performed for each channel.
An image processing method.

Images