JP2016153984A - Neural network processor, neural network processing method, detection device, detection method, and vehicle - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a neural network processor, a neural network processing method, a detection device and a detection method which detect the position of a detection object with a small processing load and with high accuracy, and to provide a vehicle having such a detection device.SOLUTION: A neural network processor (1) is provided, which includes a horizontal direction processing part (120h) for performing non-linearization processing and pooling processing of a map based on an input image, and outputting a vector (51) indicating a horizontal direction position of a detection object in the input image while maintaining the horizontal direction resolution of the input image; and a vertical direction processing part (120v) for performing non-linearization processing and pooling processing of the map based on the input image, and outputting a column vector (52) indicating a vertical direction position of the detection object in the input image while maintaining the vertical direction resolution of the input image.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a neural network processing device, a neural network processing method, a detection device and a detection method for detecting a detection target, and a vehicle having such a detection device.

  By detecting an object (such as a preceding vehicle) on the road using an in-vehicle camera, it is possible to support safe driving of the automobile. Therefore, detecting the presence and position of an object and its position is one technical problem. As one method of object detection, an image recognition method using a convolutional neural network is known (for example, Non-Patent Document 1).

  In the convolutional neural network, the convolution operation on the input image and the reduction process called pooling are repeated, and when the resolution is sufficiently reduced, the signal is input to the fully connected network (multilayer perceptron) and the final output Is obtained.

Y. LeCun, B. Boser, JS Denker, D. Henderson, RE Howard, W. Hubbard, and LD Jackel, "Handwritten Digit Recognition with a Back-Paopagation Network", Advances in Neural Information Processing Systems (NIPS), pp. 396-404, 1990.

  However, such a convolutional neural network has the following problems.

  The first is the problem of detection accuracy. In a normal convolutional neural network, the resolution is lowered because the input image is pooled in the horizontal and vertical directions. The accuracy with which the object is specified depends on the resolution, but since the resolution is lowered, the position of the object cannot always be detected with high accuracy. However, if pooling is not performed at all, the dimension of the vector in the signal input to the fully coupled network becomes significantly large, which not only imposes memory, but also tends to cause overlearning. Therefore, it is impractical not to perform pooling.

  There is also a problem of processing load. In a normal convolutional neural network, the size of the input image needs to be determined in advance. However, it is not known what size image is optimal for object position detection. Therefore, it is necessary to resize the image from the in-vehicle camera into several sizes to generate a plurality of pyramid images, and to apply a sliding window to each of them. Therefore, the processing load must be increased.

  The present invention has been made in view of such problems, and an object of the present invention is to provide a neural network processing apparatus and a neural network for detecting the position of a detection target with a low processing load and high accuracy. It is to provide a processing method, a detection device and a detection method, and to provide a vehicle having such a detection device.

  According to one aspect of the present invention, a non-linearization process and a pooling process are performed on a map based on an input image, and a horizontal direction of a detection target in the input image is maintained while maintaining a horizontal resolution of the input image. A horizontal direction processing unit that outputs a row vector indicating a position, and a map based on the input image are subjected to a non-linearization process and a pooling process, while maintaining a vertical resolution of the input image. There is provided a neural network processing device comprising: a vertical processing unit that outputs a column vector indicating a vertical position of the detection object.

  According to this configuration, since the row vector and the column vector are output while maintaining the horizontal and vertical resolutions of the input image, the position of the detection target can be detected with high accuracy.

  The horizontal direction processing unit has a first pooling unit that performs a pooling process only in the vertical direction on a map based on the input image, and the vertical direction processing unit applies to a map based on the input image, It is desirable to have a second pooling part that performs the pooling process only in the horizontal direction.

  According to this configuration, since the first pooling unit performs the pooling process only in the vertical direction, the row vector can be output while maintaining the horizontal resolution, and the second pooling unit performs the pooling process only in the horizontal direction. As a result, column vectors can be output while maintaining the vertical resolution.

  Preferably, the horizontal direction processing unit includes a first horizontal direction map generation unit and a second horizontal direction map generation unit, and the vertical direction processing unit includes a first vertical direction map generation unit and a second vertical direction map generation. The first horizontal map generation unit generates a first output map by performing non-linearization processing and pooling processing only in the vertical direction on the map based on the input image, and generates the first output map. The vertical direction map generation unit performs a non-linearization process and a pooling process only in the horizontal direction on the map based on the input image to generate a second output map, and the second horizontal direction map generation unit A non-linearization process and a pooling process are performed on the first output map and the second output map to generate a third output map having the same horizontal resolution as the first output map. The second vertical map generation unit performs a non-linearization process and a pooling process on the first output map and the second output map, and has a fourth vertical resolution equal to that of the second output map. An output map is generated, the row vector is generated based on the third output map, and the column vector is generated based on the fourth output map.

  According to this configuration, a column vector is generated using the first output map generated by the first horizontal map generation unit, and a row vector is generated using the second output map generated by the first vertical map generation unit. Is done. Therefore, detection accuracy is improved.

  Specifically, the second horizontal map generation unit performs a convolution operation on the first output map and the second output map to generate a first intermediate map and a second resolution that are equal in resolution to the first output map. A first non-linearization processing unit that generates an intermediate map and adds the first intermediate map and the second intermediate map to generate a third intermediate map; A third pooling unit that performs a pooling process to generate the third output map, and the second vertical map generation unit performs a convolution operation on the first output map and the second output map. To generate a fourth intermediate map and a fifth intermediate map having the same resolution as the second output map, respectively, and add the fourth intermediate map and the fifth intermediate map to obtain a sixth intermediate map. A second non-linearization processing unit that generates a loop, and a fourth pooling unit that performs a pooling process only in the horizontal direction on the sixth intermediate map to generate the fourth output map. .

  More specifically, the horizontal and vertical resolutions of the second intermediate map are p1 times and 1 / q1 times the horizontal and vertical resolutions of the second output map, respectively (p1 and q1 are integers). The first non-linearization processing unit corresponds to an inner product of a certain pixel value of the second output map and a filter coefficient of the first filter corresponding to a certain part of the second output map. The process of setting the values of the pixels in the second intermediate map and the (p1-1) pixels to the right of the second intermediate map is performed while shifting the first filter by q1 pixels in the vertical direction of the second output map. The second intermediate map is generated, and the horizontal and vertical resolutions of the fourth intermediate map are 1 / p2 times q2 and the horizontal and vertical resolutions of the first output map, respectively. (P2 and q2 are integers), and the second non-linearization processing unit calculates an inner product of a certain pixel value of the first output map and a filter coefficient of the second filter, as a result of the first output map. The processing for setting the value of the pixel in the fourth intermediate map corresponding to a certain part and the value of (q2-1) pixels below it is performed by the second filter by p2 pixels in the horizontal direction of the first output map. The fourth intermediate map may be generated by shifting while shifting.

In addition, an image in which pixels corresponding to the upper, lower, left, and right edges of the detection target in the sample image have the first value and the other pixels have the second value is the input image, and the horizontal direction processing unit and the vertical direction processing unit are The row vector and the column vector as correct data may be output by performing the non-linearization process including a convolution operation using a filter whose center is the first value and the other is the second value. .
With this configuration, correct data can be generated from the sample image and used for learning processing.

In addition, the pixel in which the detection target in the sample image has the first value and the other pixel has the second value is the input image, and the horizontal processing unit and the vertical processing unit are The row vector and the column vector as correct data may be output by performing the non-linearization process including a convolution operation using a filter in which is the first value and the other is the second value.
With this configuration, correct data can be generated from the sample image and used for learning processing. This is particularly effective when there are a plurality of detection objects.

  A neural network parameter used in the non-linearization process may be generated based on the sample image and the correct answer data.

The row vector in which the column corresponding to the left end and the right end of the detection target object is the first value, and the column corresponding to the left side from the left end and the column corresponding to the right end from the right end is the second value. And the vertical processing section has the first value corresponding to the upper end and the lower end of the detection object, and the row corresponding to the upper side and the lower end of the detection target. The neural network parameters are preferably generated so as to output the binary column vector.
Accordingly, the position of the detection target can be indicated by the value of each column in the row vector and the value of each row in the column vector.

The sample image preferably includes a sample image including a first detection target and a sample image including a second detection target different from the first detection target.
According to this configuration, a plurality of types of detection objects can be detected.

The sample image preferably includes a sample image including only a part of the detection target.
According to this configuration, even when only a part of the detection target is included in the input image, the detection target can be detected.

An area specifying unit that specifies an area including the detection target object from the input image based on the row vector, and the vertical direction processing unit applies a map based on the input image to the specified area. Non-linear processing and pooling processing may be performed.
According to this configuration, the detection accuracy can be improved when the detection target is vertically long.

When it is determined that the detection target is not included in the input image based on the row vector, it is preferable that the vertical direction processing unit does not perform nonlinear processing and pooling processing.
According to this configuration, when there is no detection target, the vertical processing unit does not perform processing, and the amount of calculation can be reduced.

  According to another aspect of the present invention, the position of the detection object in the input image is specified based on the neural network device and the row vector and the column vector output by the neural network device. There is provided a detection device including a specific unit.

The specifying unit is a column in the row vector, and a column whose value satisfies the first threshold condition is set as a left end candidate or a right end candidate of the detection target, and is a row in the column vector, and the value is the first It is desirable that a row satisfying the two threshold conditions be an upper end candidate or a lower end candidate of the detection target.
According to this configuration, it is possible to remove noise in the output row vector and column vector, and the detection accuracy is improved.

Further, the left end candidate and the right end candidate of the detection target specified by the specifying unit are separated by a number of first pixels or more, and the upper end candidate and the lower end candidate of the detection target specified by the specifying unit are It is desirable that the number of pixels is two or more.
According to this configuration, the noise of the output row vector and column vector can be removed, and the detection accuracy is improved.

  According to another aspect of the present invention, there is provided a vehicle including a vehicle main body, a camera attached to the vehicle main body, and the detection device that processes an image from the camera as the input image. .

  According to another aspect of the present invention, a non-linearization process and a pooling process are performed on a map based on an input image, and a detection target in the input image is maintained while maintaining a horizontal resolution of the input image. Outputting a row vector indicating a horizontal position of the input image, and performing a non-linearization process and a pooling process on the map based on the input image, and maintaining the vertical resolution of the input image in the input image Outputting a column vector indicating a vertical position of the detection object, and providing a neural network processing method.

  According to another aspect of the present invention, a non-linearization process and a pooling process are performed on a map based on an input image, and a detection target in the input image is maintained while maintaining a horizontal resolution of the input image. Outputting a row vector indicating a horizontal position of the input image, and performing a non-linearization process and a pooling process on the map based on the input image, and maintaining the vertical resolution of the input image in the input image A detection method comprising: outputting a column vector indicating a vertical position of the detection object; and specifying a position of the detection object in the input image based on the row vector and the column vector. Provided.

  A row vector indicating the horizontal position of the detection target in the input image is output while maintaining the horizontal resolution of the input image, and the detection target of the input image is maintained while maintaining the vertical resolution of the input image. Since the row vector indicating the vertical position is output, the position of the detection target can be detected with high accuracy. Furthermore, since there are no restrictions on the size and aspect ratio of the detection target, the processing load can be reduced.

The figure explaining the outline | summary of the processing operation of the neural network process part 1 which concerns on 1st Embodiment. 1 is a block diagram showing a schematic configuration of a detection apparatus 100 according to a first embodiment. 1 is a block diagram showing a schematic configuration of a neural network processing unit 1 according to a first embodiment. The figure explaining the outline of the processing operation of the neural network process part. The figure explaining the process of the non-linearization process part 121h. The figure explaining the convolution calculation in the non-linearization process part 121h. The figure explaining the pooling process of the pooling part 122h in the horizontal direction map production | generation part 12h. The figure explaining the input image for producing | generating correct data from a sample image. The figure which shows typically the process of the neural network process part 1 at the time of correct data generation. The block diagram which shows schematic structure of the neural network process part 1 'which concerns on 2nd Embodiment. The block diagram which shows schematic structure of the horizontal direction map production | generation part 12h 'and the vertical direction map production | generation part 12v'. The figure explaining the outline of the processing operation of the horizontal direction map production | generation part 12h 'and the vertical direction map production | generation part 12v'. The figure explaining the production | generation method of the map 93. FIG. The figure explaining the production | generation method of map 64 '. The figure which shows typically the processing operation of the neural network process part 1 'which concerns on 2nd Embodiment. The figure explaining specification of the right-and-left end position of a detection target by specific part 3. The figure explaining the identification of the upper and lower ends of a detection target by the identification unit. The figure which shows the 1st example of the processing operation of the specific | specification part 3. FIG. The figure explaining the noise removal by the specific | specification part 3. FIG. The flowchart which shows the procedure of noise removal. The figure which shows the 2nd example of the processing operation of the specific | specification part 3. FIG. The figure which shows the 3rd example of the processing operation of the specific | specification part 3. FIG. The figure which shows the 4th example of the processing operation of the specific | specification part 3. FIG. The figure which shows the 5th example of the processing operation of the specific | specification part 3. FIG. The figure which shows the 6th example of the processing operation of the specific | specification part 3. FIG. The figure explaining the method of calculating the distance Z of the advancing direction of the vehicle-mounted camera 101 and the detection target object 102. FIG. Lateral distance X _L between the vehicle-mounted camera 101 and the detection object 102, diagram for explaining a method of calculating the X _R. The block diagram which shows schematic structure of the neural network process part 1 '' which concerns on 4th Embodiment. The figure explaining the outline | summary of the processing operation of the neural network process part 1 which concerns on 5th Embodiment. The figure explaining the input image for producing | generating correct data from a sample image. The figure explaining the input image for producing | generating correct data from another sample image. The figure explaining the processing operation of the area | region specific part 13. FIG.

  Embodiments according to the present invention will be specifically described below with reference to the drawings.

(First embodiment)
In the first embodiment, the positions of the upper, lower, left and right ends of the detection object in the input image, more specifically, the left end position xl, the right end position xr, the upper end position yt, and the lower end position yb are specified without impairing the resolution of the input image. To do.

  Therefore, a virtual binary image having the same resolution as the input image is considered. In this binary image, only the values of the pixels (xl, yt), (xr, yt), (xl, yb), (xr, yb) are 1, and the values of the other pixels are 0. When the binary image as described above is obtained by processing the input image, the upper, lower, left and right ends of the detection target are clarified.

  Further, if this binary image is regarded as a matrix, this binary image has only a column vector in which only the values of the yt row and the yb row are 1 and the values of the other rows are all 0, and the values of the xl and xr columns. 1 and the values of the other columns are decomposed into products with row vectors which are all 0. If the number of columns in the column vector and the number of columns in the row vector are equal to the vertical and horizontal resolutions of the input image, the positions of the upper, lower, left and right ends of the detection target can be specified without impairing the resolution of the input image.

  Therefore, in the present embodiment, a neural network processing apparatus (neural network processing unit) that performs neural network processing on an input image and outputs the above column vector and row vector without losing the resolution is disclosed.

  FIG. 1 is a diagram for explaining the outline of the processing operation of the neural network processing unit 1 according to the first embodiment. As shown in FIG. 1A, the number of vertical pixels in the input image is a, and the number of horizontal pixels is b (hereinafter, this is also simply referred to as “the number of pixels in the input image is b × a”). To do. Although specific processing contents will be described later, in this case, the neural network processing unit 1 outputs a row vector 51 having the number of elements (number of columns) b and a column vector 52 having the number of elements (number of rows) a. .

  As shown in FIG. 1B, when the boundary of the detection object in the input image is defined by a rectangle, the upper left, upper right, lower left and lower right pixel positions are (xl, yt) and (xr, yt), respectively. , (Xl, yb), (xr, yb). At this time, the neural network processing unit 1 has a row vector 51 in which only the values in the xl and xr columns are 1 and the values in the other columns are 0, and the values in only the yt and yb rows are 1 and the other rows. Neural network processing is performed on the input image so as to output a column vector 52 whose value is 0.

  A column having a value of 1 in the row vector 51 indicates a left end position and / or a right end position of the detection target. A row having a value of 1 in the column vector 52 indicates the upper end position and / or the lower end position of the detection target. Therefore, the position of the detection target can be specified based on such row vector 51 and column vector 52.

  Hereinafter, as a specific example, a detection device that captures the front of a vehicle with an in-vehicle camera and detects the positions of the upper, lower, left and right ends of a detection object (for example, a person, another vehicle, a sign) using the obtained image as an input image explain.

  FIG. 2 is a block diagram illustrating a schematic configuration of the detection apparatus 100 according to the first embodiment. The detection apparatus 100 includes a neural network processing unit 1, a learning unit 2, and a specifying unit 3. In this detection apparatus 100, the neural network processing unit 1 and the learning unit 2 generate neural network parameters in advance using sample images (this is also referred to as “learning processing”). Then, the neural network processing unit 1 and the specifying unit 3 detect a detection target from the input image using the generated neural network parameters (this is also referred to as “detection processing”).

  In the learning process, first, the neural network processing unit 1 performs neural network processing on an input image obtained by processing a sample image prepared in advance using neural network parameters for learning processing, and obtains correct data. Generate. Subsequently, the neural network processing unit 1 performs a neural network process using the sample image as an input image, and outputs a row vector 51 and a column vector 52. The learning unit 2 generates a neural network parameter for detection processing so that the row vector 51 and the column vector 52 are as close as possible to the correct data.

  In the detection process, an image from the in-vehicle camera is input to the neural network processing unit 1 as an input image. Then, the neural network processing unit 1 performs neural network processing on the input image using the neural network parameters for detection processing generated by the learning unit 2, and a row vector 51 and a column vector indicating the position of the detection target object 52 is output. The specifying unit 3 interprets the output row vector 51 and column vector 52 and specifies the position of the detection target.

  In the learning process and the detection process, the neural network structure in the neural network processing unit 1 is common. Therefore, first, the neural network structure will be described, and then the learning process and the detection process will be described in detail.

  FIG. 3 is a block diagram illustrating a schematic configuration of the neural network processing unit 1 according to the first embodiment. As shown in FIG. 3, the neural network processing unit 1 includes a non-linearization processing unit 11, a horizontal direction processing unit 120h, and a vertical direction processing unit 120v. The horizontal direction processing unit 120h includes one or more horizontal direction map generation units 12h, and the vertical direction processing unit 120v includes one or more vertical direction map generation units 12v.

  Note that the non-linearization processing unit 11 may not be provided, and two or more non-linearization processing units 11 may be connected in cascade. The number of horizontal direction map generation units 12h and vertical direction map generation units 12v is not limited, but the number of both is the same.

  The first-stage non-linearization processing unit 11 performs convolution calculation and activation processing using the input image as a map, and outputs a new map. When the input image is grayscale, the number of maps is one. When the input image is composed of an R image, a G image, and a B image, the number of maps may be three, an R image, a G image, and a B image. When there are a plurality of non-linearization processing units 11, the next-stage non-linearization processing unit 11 performs the same processing on the map output from the preceding non-linearization processing unit 11. The specific processing content is the same as the processing in the non-linearization processing unit 121h of the horizontal direction map generation unit 12h described below. The processing by the non-linearization processing unit 11 has a role of extracting an edge of an object.

  In the horizontal direction processing unit 120h and the vertical direction processing unit 120v, a map based on an input image, that is, a map output from the non-linearization processing unit 11 at the final stage, or an input image when the non-linearization processing unit 11 is not provided. Entered. When there is one type of detection object, the horizontal processing unit 120h outputs one map (that is, the row vector 51), and the vertical processing unit 120v outputs one map (that is, the column vector 52).

  Each horizontal direction map generation unit 12h in the horizontal direction processing unit 120h includes a non-linearization processing unit 121h and a pooling unit 122h. The non-linearization processing unit 121h performs non-linearization processing on the map. Although the number of maps can be changed by the non-linearization process, the resolution of the map is not changed. The pooling unit 122h performs pooling only in the direction perpendicular to the map. This process reduces the vertical resolution of the map, but does not change the horizontal resolution.

  Each vertical direction map generation unit 12v in the vertical direction processing unit 120v also includes a non-linearization processing unit 121v and a pooling unit 122v. The nonlinear processing unit 121v performs nonlinear processing on the map. Although the number of maps can be changed by the non-linearization process, the resolution of the map is not changed. The pooling unit 122v performs pooling only in the horizontal direction with respect to the map. This process reduces the horizontal resolution of the map, but does not change the vertical resolution.

  FIG. 4 is a diagram for explaining the outline of the processing operation of the neural network processing unit 1, and pays particular attention to the change in the resolution of the map. In the figure, solid line arrows mean non-linearization processing by the non-linearization processing units 11, 121h, 121v, and broken line arrows mean pooling processing by the pooling units 122h, 122v. According to the expression of FIG. 4, the number of maps of the input image is m0, the number of maps after the k-th stage (excluding the last stage) nonlinear processing is mk, and the number of maps after the last stage nonlinear processing. Is clearly 1. Also, it is clearly shown that the map resolution changes only in the vertical direction or in the horizontal direction at each pooling process.

  FIG. 5 is a diagram illustrating the processing of the non-linearization processing unit 121h. Since the processing of the nonlinear processing units 11, 121h, and 121v is common, the nonlinear processing unit 121h will be described unless otherwise specified.

  The non-linearization processing unit 121h performs convolution operations, activation processing, and bias addition processing on the maps 61 to 63, and generates intermediate maps 71 to 73. In the figure, an example is shown in which three intermediate maps 71 to 73 are generated from three maps 61 to 63, but the number of input maps and intermediate maps is not limited. However, the number of intermediate maps generated by the first-stage nonlinearization processing unit 121h is equal to the number of intermediate maps generated by the first-stage nonlinearization processing unit 121v. The same applies to the second and subsequent stages. Then, the nonlinear processing units 121h and 121v in the final stage generate one intermediate map.

  In FIG. 5, the solid line arrows indicate the convolution operation and the activation process. Details of the convolution operation will be described later with reference to FIG. 6, but the coefficient of the filter used for the convolution operation may be different for each solid line arrow. The activation process is a process for activating the result of the convolution operation using an activation function such as a sigmoid function. A one-dot chain arrow indicates a bias addition process. A scalar bias is multiplied by 1, and is added to all elements of the result of the activation process. The bias can be different for each dashed line arrow.

  In the example of FIG. 5, the maps 61 to 63 are input to the non-linearization processing unit 121h, the result of the convolution calculation and activation processing for the map 61, the result of the convolution calculation and activation processing for the map 62, and the map 63 It is shown that the result of the convolution operation and the activation process is added, and the bias is added to generate an intermediate map 71 that is subsequently input to the pooling unit 122h.

  The filter coefficients and bias are the neural network parameters. The learning processing neural network parameter is a predetermined fixed value, and the detection processing neural network parameter is generated by the learning unit 2.

  FIG. 6 is a diagram for explaining the convolution operation in the non-linearization processing unit 121h. FIG. 6A shows a convolution operation in which the map size decreases, and FIG. 6B shows a convolution operation in which the map size is unchanged.

  In the convolution operation of FIG. 6A, the non-linearization processing unit 121h first sets the filter 80 in, for example, a 3 × 3 pixel region in which the pixel (1, 1) of the map 64 before the convolution operation is at the upper left. Then, the non-linearization processing unit 121h sets the inner product of the filter coefficient and the pixel value of the map 64 as the value of the pixel (1, 1) of the map 81 after the convolution calculation. Thereafter, the non-linearization processing unit 121h calculates each pixel value of the map 81 after the convolution operation while shifting the position of the filter 80 to the right by one pixel. In this case, the values of the rightmost two columns and the lowermost two rows in the map 81 after the convolution operation are not calculated. Therefore, the size of the map 81 after the convolution operation is reduced by two pixels in both the horizontal direction and the vertical direction, compared to the map 64 before the convolution operation.

  On the other hand, in the convolution operation of FIG. 6B, the non-linearization processing unit 121h sets the filter 80 in, for example, a 3 × 3 pixel region centered on the pixel (1, 1) of the map 64 before the convolution operation. Then, the non-linearization processing unit 121h sets the inner product of the filter coefficient and the pixel value of the map 64 as the value of the pixel (1, 1) of the map 82 after the convolution calculation. However, the value of the pixel in the region that protrudes from the map 64 before the convolution operation is set to 0 (referred to as zero padding). Thereafter, the non-linearization processing unit 121h calculates each pixel value of the map 82 after the convolution calculation while shifting the position of the filter 80 to the right by one pixel. In this case, the size of the map 82 after the convolution operation is equal to the size of the map 64 before the convolution operation.

  In any case, since the map 64 is not reduced, it can be considered that the resolution of the map is the same before and after the convolution operation regardless of whether the size (number of pixels) of the map changes or not.

  The size of the filter 80 is not particularly limited, but in the following specific example, the number of horizontal pixels and the number of vertical pixels of the filter 80 are equal to each other and are common to all the non-linearization processing units 121h and 121v. The size of The coefficient of each filter 80 is generated by learning processing, and may be different for each of the non-linearization processing units 121h and 121v. Further, it is assumed that a convolution operation in which the map size in FIG. 6B is unchanged is performed.

  FIG. 7 is a diagram illustrating the pooling process of the pooling unit 122h in the horizontal direction map generation unit 12h. The pooling unit 122h pools (reduces) the map only in the vertical direction without changing the horizontal resolution, in other words, while maintaining the number of pixels in the horizontal direction. Hereinafter, an example in which the number of pixels in the vertical direction is pooled to ½ will be described in detail.

  First, the pooling unit 122h divides the intermediate map 71 from the non-linearization processing unit 121h into a grid 84 of 1 pixel in the horizontal direction × 2 pixels in the vertical direction (FIG. 7A). The total number of grids 84 is ½ of the intermediate map 71. Next, the pooling unit 122h selects a pixel having the maximum value in each grid 84 (shows a pixel in which the black pixel in FIG. 7B is selected). Then, the pooling unit 122h generates a map 85 after the pooling process by filling other pixels with the selected pixel in each grid 84 (FIG. 7C). As a result, the intermediate map 71 generated by the non-linearization processing unit 121h is pooled only in the vertical direction.

In FIG. 7B, the pooling unit 122h may use the average value of the pixels in the grid 84 instead of selecting the maximum value in each grid 84. The pooling unit 122h preferably pools the number of pixels in the vertical direction to (1/2) ⁿ (n is an integer of 1 or more), but may be pooled to 1/3 or 1/5.

  On the other hand, the pooling unit 122v in the vertical direction map generation unit 12v pools the map only in the horizontal direction while maintaining the number of pixels in the vertical direction without changing the vertical resolution. Others are the same as the pooling part 122h of the horizontal direction map production | generation part 12h.

As described above, the vertical pixel number and the horizontal pixel number of the map become ½ each time one of the pooling parts 122h and 122v passes. For example, when eight stages of pooling units 122h and 122v are provided and the number of pixels of the input image is 512 × 256, one row vector 51 and two column vectors 52 are finally output.
Subsequently, the learning process will be described.

  In order to perform the learning process, it is necessary to prepare ideal row vectors 51 and column vectors 52 for a certain sample image as correct data in advance. A method of generating correct data by the neural network processing unit 1 will be described.

  FIG. 8 is a diagram for explaining an input image for generating correct answer data from a sample image. (A) in the figure is an example of a sample image. The positions of the upper, lower, left and right ends of the detection target are specified by hand, and the input image shown in (b) in FIG. The input image has the same number of pixels as the sample image. Only the pixel values at the upper, lower, left and right ends of the detection target are 1, and the values of the other pixels are 0.

  An input image shown in FIG. 8B is input to the neural network processing unit 1. In the learning process, the number of maps is 1. Of the learning neural network parameters, all biases are set to zero. The filter coefficient is set to 1 only at the center value of the filter, and 0 is set to other values. Note that the size of the filter is made common during the learning process and the detection process.

  FIG. 9 is a diagram schematically illustrating processing of the neural network processing unit 1 when generating correct answer data. As illustrated, the row vector 51 and the column vector 52 are output through the processing of the non-linearization processing units 11, 121h, 121v and the pooling units 122h, 122v. These row vector 51 and column vector 52 are correct answer data, and a set of sample images and correct answer data is obtained.

  In order to increase the detection accuracy, as many sample images as possible are used. For example, it is desirable to use a sample image that includes only a part of the detection object. As a result, even when only a part of the detection object is shown in the input image, the detection object can be accurately detected. In addition, it is desirable to use sample images having various aspect ratios of rectangles surrounding the detection target. Thereby, even when the aspect ratio differs depending on the direction in which the detection target is photographed (for example, when the detection target is a vehicle), the detection target can be detected with high accuracy.

  In addition, sample images with a large number of pixels, small sample images, sample images with a detection target in the center, sample images at the edges, sample images with a large target detection object, small sample images, and the detection target are facing the front. It is desirable to use a sample image or a sample image facing diagonally.

The learning unit 2 receives the correct data (that is, the row vector 51 and the column vector 52) t ⁱ _m (i is an index of the sample image and m is a row vector or a column vector) obtained by the above processing. And an output when the neural network processing unit 1 processes the sample image x ⁱ with the neural network parameter W (that is, the row vector 51 and the column vector 52) f _m (x ⁱ ; W) is input. The
Then, the learning unit 2 defines an objective function E (W) as in the following equation (1).

Here, n is the number of sample images. The objective function E (W) is a function of the neural network parameter W (that is, the filter coefficient and the bias), and becomes 0 only when the output f _m (x ⁱ ; W) and the correct answer data t ⁱ _m match, and does not match. If it is greater than zero.

Therefore, the learning unit 2 generates the neural network parameter W so that the objective function E (W) is as small as possible. In order to minimize the objective function E (W), for example, a publicly known error back propagation method can be applied. Specifically, until the objective function E (W) converges, the update shown in the following formula (2) is performed. The rules may be applied in order from the horizontal direction map generation unit 12h side and the vertical direction map generation unit 12v side of the final stage.

The filter coefficient and the bias generated in this way are used for detection processing as a detection processing neural network parameter.
Subsequently, the detection process will be described.

  At the time of detection processing, neural network parameters for detection processing are set in the non-linearization processing units 11, 121h, 121v of the neural network processing unit 1. An image from the in-vehicle camera is input to the neural network processing unit 1 as an input image. Then, as shown in FIG. 1, the neural network processing unit 1 generates a row vector 51 and a column vector 52.

  If the neural network parameters for detection processing are appropriate, the column and row values corresponding to the upper, lower, left and right edges of the detection target in the row vector 51 and the column vector 52 are 1 (or values close to 1), and other values Becomes 0 (or a value close to 0). Therefore, the specifying unit 3 can specify the positions of the upper, lower, left and right ends of the detection target in the input image based on the row vector 51 and the column vector 52. The specifying unit 3 will be described in detail in the third embodiment.

  As described above, in the first embodiment, the horizontal direction processing unit 120 h that generates the row vector 51 and the vertical direction processing unit 120 v that generates the column vector 52 are provided separately. And the pooling part 122h in the horizontal direction processing part 120h does not pool the map in the horizontal direction but only in the vertical direction. The pooling unit 122v in the vertical processing unit 120v does not pool the map in the vertical direction, but pools the map only in the horizontal direction.

  Therefore, the row vector 51 indicating the horizontal position of the detection object can be generated while maintaining the horizontal resolution of the input image, and the vertical position of the detection object can be maintained while maintaining the vertical resolution of the input image. Can be generated. Therefore, the position of the detection target can be detected with high accuracy.

  In addition, according to the present embodiment, since there is no limitation on the size and aspect ratio of the detection target object, generation of a pyramid image and application of a sliding window are unnecessary, and the processing load on the neural network processing unit 1 can be reduced. .

(Second Embodiment)
In the second embodiment to be described next, the horizontal direction processing unit 120h generates the row vector 51 using the map generated by the vertical direction processing unit 120v, and also the map generated by the horizontal direction processing unit 120h. The vertical processing unit 120v is used to generate the column vector 52.

  FIG. 10 is a block diagram illustrating a schematic configuration of the neural network processing unit 1 ′ according to the second embodiment. As a difference from FIG. 3, at least two horizontal direction map generators 12h and 12h 'and vertical direction map generators 12v and 12v' are provided. The first-stage horizontal direction map generation unit 12h and the vertical direction map generation unit 12v are the same as those described in the first embodiment. On the other hand, the second and subsequent horizontal direction map generation units 12h ′ not only include the map output from the previous horizontal direction map generation unit 12h (12h ′) but also the previous vertical direction map generation unit 12v (12v ′). ) Is also used to output a new map. The vertical direction map generation unit 12v ′ in the second and subsequent stages is not only the map output from the vertical direction map generation unit 12v (12v ′) in the previous stage, but also from the horizontal direction map generation unit 12h (12h ′) in the previous stage. A new map is output using the output map.

  FIG. 11 is a block diagram showing a schematic configuration of the horizontal direction map generator 12h ′ and the vertical direction map generator 12v ′. As shown in the figure, the second-stage horizontal map generation unit 12h ′ has a non-linearization processing unit 121h ′, and outputs an output map from the first-stage horizontal direction map generation unit 12h and a first-stage vertical direction map generation unit 12v. An intermediate map is generated by performing a non-linearization process on the output map. Further, the vertical direction map generation unit 12v ′ includes a similar non-linearization processing unit 121v ′. The pooling portions 122h and 122v are the same as those in FIG.

  FIG. 12 is a diagram for explaining the outline of the processing operation of the horizontal direction map generation unit 12h ′ and the vertical direction map generation unit 12v ′. In the figure, the two maps 64 and 65 output from the first horizontal map generation unit 12h are input to the horizontal map generation unit 12h ′, and the horizontal map generation unit 12h ′ outputs the two maps. Are two maps 91 and 92. Also, the two maps 66 and 67 output from the initial vertical map generation unit 12v are input to the vertical direction map generation unit 12v ′, and the vertical direction map generation unit 12v ′ outputs two. Two maps 93 and 94 are assumed.

  As illustrated, the non-linearization processing unit 121 h ′ generates an intermediate map 74 from the maps 64 to 67. The resolution of the intermediate map 74 is equal to the resolution of the maps 64 and 65. Then, the pooling unit 122h pools the intermediate map 74 only in the vertical direction to generate a map 91. That is, the horizontal resolution of the map 91 is equal to the horizontal resolution of the map 64. Similarly, the non-linearization processing unit 121 h ′ generates an intermediate map 75 from the maps 64 to 67. Then, the pooling unit 122h generates a map 92 from the intermediate map 75.

On the other hand, the non-linearization processing unit 121v ′ generates an intermediate map 76 from the maps 64-67. The resolution of the intermediate map 76 is equal to the resolution of the maps 66 and 67. Then, the pooling unit 122v pools the intermediate map 76 only in the horizontal direction to generate a map 93. That is, the vertical resolution of the map 93 is equal to the vertical resolution of the map 66. Similarly, the non-linearization processing unit 121v ′ generates an intermediate map 77 from the maps 64-67. Then, the pooling unit 122v generates a map 94 from the intermediate map 77.
Since the generation methods of the maps 91 to 94 are common, the generation of the map 93 will be described in detail below as a representative.

  FIG. 13 is a diagram for explaining a method for generating the map 93. Here, it is assumed that the number of pixels of the input image is C × R. The number of pixels of the maps 64 and 65 input to the vertical direction map generation unit 12v ′ is C × R / 2 by the processing of the horizontal direction map generation unit 12h in the first stage. Further, the number of pixels of the maps 66 and 66 input to the vertical direction map generation unit 12v ′ is C / 2 × R by the process of the vertical direction map generation unit 12v in the first stage.

  The non-linearization processing unit 121v ′ convolves the maps 66 and 67 to generate intermediate maps 66 ′ and 67 ′, respectively. The number of pixels in the intermediate maps 66 ′ and 67 ′ is C / 2 × R, similar to the number of pixels in the maps 66 and 67. The convolution calculation here is, for example, the process shown in FIG.

  On the other hand, the non-linearization processing unit 121v 'performs convolution operations on the maps 64 and 65 to generate intermediate maps 64' and 65 ', respectively. Here, the non-linearization processing unit 121v ′ doubles the number of pixels in the horizontal direction of the maps 64 and 65 and doubles the number of pixels in the vertical direction as follows to obtain the intermediate maps 64 ′ and 65 ′ as follows. Are generated respectively.

  FIG. 14 is a diagram illustrating a method for generating the map 64 ′. In the figure, the number of pixels of the filter is 3 × 3. As shown in FIG. 6A, the non-linearization processing unit 121v ′ first sets the filter so that the center of the filter overlaps the pixel (1, 1) of the map 64. Then, the non-linearization processing unit 121v ′ calculates the inner product of the pixel value of the portion where the filter of the map 64 is set and the filter coefficient by using the upper left pixel (1, 1) and the lower pixel (1) of the intermediate map 64 ′. , 2). That is, one inner product is set to two pixels arranged in the vertical direction of the intermediate map 64 '.

Subsequently, as illustrated in FIG. 14B, the non-linearization processing unit 121 v ′ shifts the filter to the right by two pixels, and sets the filter so that the center of the filter overlaps the pixel (3, 1) of the map 64. Then, the non-linearization processing unit 121v ′ calculates the inner product of the pixel value of the portion where the filter of the map 64 is set and the filter coefficient, as the pixel (2, 1) of the intermediate map 64 ′ and the lower pixel (2, 2). ) Value.
In this way, by processing one row of the map 64 while shifting the filter to the right by two pixels, the values of the two columns of the intermediate map 64 ′ are determined.

  After the processing for the first row of the map 64 is finished, as shown in FIG. 14C, the non-linearization processing unit 121v ′ shifts the filter down by one pixel, and the center of the filter is the pixel (1, 2) of the map 64. Are set so that they overlap with each other, and the inner product is calculated as the value of the pixel (1, 3) and the pixel (1, 4) below it in the intermediate map 64 ′.

  By performing the processing while shifting the filter in this way, an intermediate map 64 ′ having the number of pixels of C / 2 × R is generated from the map 64 having the number of pixels of C × R / 2.

  More generally, in order to generate the intermediate map 64 ′ by multiplying the number of horizontal pixels and the number of vertical pixels of the map 64 by 1 / p times and q times (p and q are arbitrary integers), respectively, nonlinearity is required. The conversion processing unit 121v ′ converts the inner product of the pixel value of the part where the filter of the map 64 is set and the filter coefficient into the value of the corresponding pixel in the intermediate map 64 ′ and (q−1) pixels below it. Set. By performing this process while shifting the filter by p pixels in the horizontal direction of the map 64 and shifting the filter by one pixel in the vertical direction of the map 64, an intermediate map 64 'can be generated.

As described above, the intermediate maps 64 ′ and 65 ′ in FIG. 13 are generated. Then, the non-linearization processing unit 121v ′ adds the pixel values of the intermediate maps 64 ′ to 67 ′ to generate the intermediate map 76 having the number of pixels of C / 2 × R (two-dot chain line in FIG. 13). . Although not shown, a bias is added after addition. Then, the pooling unit 122v performs a pooling process on the intermediate map 76, and a map 93 having the number of pixels of C / 4 × R is generated. Similarly, the non-linearization processing unit 121v ′ can generate the map 94 of FIG.
Further, the non-linearization processing unit 121h ′ can generate the maps 91 and 92 by switching the horizontal direction and the vertical direction described above.

  That is, the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the map 66 are multiplied by p and 1 / q, respectively (p and q are arbitrary integers), and an intermediate map (not shown. To generate the inner product of the pixel value of the portion of the map 66 where the filter is set and the filter coefficient, the non-linearization processing unit 121h ′ generates the corresponding pixel in the intermediate map and (p) -1) Set to the value of one pixel. An intermediate map can be generated by performing this process while shifting the filter by one pixel in the horizontal direction of the map 66 and shifting the filter by q pixels in the vertical direction of the map 66.

  As described above, the horizontal direction processing unit 120h also uses the map generated by the vertical direction processing unit 120v, and the vertical direction processing unit 120v also uses the map generated by the horizontal direction processing unit 120h. The improvement in detection accuracy will be described.

  FIG. 15 is a diagram schematically illustrating the processing operation of the neural network processing unit 1 ′ according to the second embodiment. For simplification, in FIG. 13, an intermediate map 76 is obtained from one map 64 output from the first horizontal map generation unit 12h and one map 66 output from the first vertical map generation unit 12v. It shows that it generates. The detection object is a person.

  When a person is shown in the input image, the feature of the person crushed in the vertical direction is arranged on the map 64, and the feature of the person crushed in the horizontal direction is arranged on the map 66. In the present embodiment, an intermediate map 76 is generated from these two maps 64 and 66. The pixel 76a of the intermediate map 76 is generated from the inner product in the rectangular area 64a of the map 64 and the inner product in the rectangular area 66a of the map 66. When the pixel 76a is generated using only the rectangular area 66a, the pixel 76a is generated only from the image of the left foot of the person on the map 66.

  However, it may be difficult to accurately detect the position of a person from an image of only the tip of a foot that is a small part of the human body. This is because many regions similar to the tip of the foot may exist in the image.

  On the other hand, in the present embodiment, the pixel 76a is generated using not only the rectangular area 66a but also the rectangular area 64a. Therefore, the pixel 76a is generated from an image in a range that is wider in the vertical direction than the tip of the foot (specifically, from the base of the foot to the waist and the stomach). As a result, the person is detected using information on a wider area of the person, and the detection accuracy is improved.

  As described above, in the second embodiment, the horizontal direction processing unit 120h generates the row vector 51 using the map generated by the vertical direction processing unit 120v, and the map generated by the horizontal direction processing unit 120h is also used. Using the vertical processing unit 120v, the column vector 52 is generated. Therefore, the position of the detection target can be detected with higher accuracy.

(Third embodiment)
In the third embodiment, the specifying unit 3 will be described in detail. In this embodiment, it is assumed that there is one type of detection object.

  FIG. 16 is a diagram illustrating the specification of the left and right end positions of the detection target by the specifying unit 3. An example is shown in which two row vectors 51a and 51b are output as a result of processing an input image.

  The row vector 51a includes information on the upper half area 51A of the input image by the pooling processing of the pooling unit 122h in the horizontal processing unit 120h. Similarly, the row vector 51b includes information on the lower half area 51B of the input image. That is, the row vector 51a and the row vector 51b correspond to the upper half area 51A and the lower half area 51B of the input image, respectively.

  Therefore, when the row vector 51a includes one value (or a value close to one value, the same applies hereinafter) (xla and xra columns in the example of FIG. 16), the left end candidate of the detection target in the upper half area 51A of the input image And / or right edge candidates are included. On the other hand, when the row vector 51a does not include 1 value, it means that the left and right end candidates are not included in the upper half area 51A of the input image.

  Similarly, when 1 is included in the row vector 51b (xlb and xrb columns in the example of FIG. 16), the lower half region 51B of the input image includes the left end candidate and / or right end candidate of the detection target. means. When the row vector 51b does not include 1 value, it means that the left and right end candidates are not included in the lower half area 51B of the input image.

  FIG. 17 is a diagram for explaining the identification of the upper and lower ends of the detection target by the identification unit 3. An example is shown in which three column vectors 52a to 52c are output as a result of processing an input image.

  The column vector 52a includes information on the left region 52A of the input image divided into three equal parts in the vertical direction by the pooling processing of the pooling unit 122v in the vertical direction processing unit 120v. Similarly, the column vectors 52b and 52c include information on the central region 52B and the right region 52C, respectively, of the input image divided into three equal parts in the vertical direction. That is, the column vectors 52a to 52c correspond to the left region 52A, the central region 52B, and the right region 52C, respectively, of the input image divided into three equal parts in the vertical direction.

  Therefore, when 1 is included in the column vector 52a, it means that the upper end candidate and / or the lower end candidate of the detection target is included in the left region 52A of the input image. On the other hand, when the column vector 52a does not include 1 value, it means that the upper and lower end candidates are not included in the left region 52A of the input image.

  Similarly, when 1 is included in the column vector 52b (the ytb and ybb columns in the example of FIG. 17), it means that the upper end candidate and / or the lower end candidate of the detection target is included in the central region 52B of the input image. To do. When the column vector 52b does not include 1 value, it means that the upper and lower end candidates are not included in the central region 52B of the input image.

  In addition, when the column vector 52c includes one value (the ytc and ybc columns in the example of FIG. 17), it means that the upper end candidate and / or the lower end candidate of the detection target is included in the right region 52C of the input image. . When the column vector 52c does not include 1 value, it means that the upper and lower end candidates are not included in the right region 52C of the input image.

  FIG. 18 is a diagram illustrating a first example of the processing operation of the specifying unit 3. As an example of processing an input image, two row vectors 51a and 51b and three column vectors 52a to 52c are output.

  In FIG. 18, the values of two common columns xl and xr of the row vectors 51a and 52b are 1. The pixel positions corresponding to these columns xl and xr are the left and right candidates for the detection target. Further, the value of two common rows yt and yb of the column vectors 52a to 52c is 1. Pixel positions corresponding to these rows yt and yb are upper and lower end candidates of the detection target.

  FIG. 18 is a simple example, and the specifying unit 3 has a detection target in a rectangular area surrounded by coordinates (xl, yt), (xr, yt), (xl, yb), and (xr, yb). Can be identified.

  In practice, the row vector 51 and the column vector 52 may contain noise. Therefore, each value in the row vector 51 and the column vector 52 may be a value between 0 and 1. A column or row whose value is closer to 1 is more likely to correspond to the edge of the detection object, but a value near a column or row near the edge of the detection object may also be close to 1. Therefore, the specifying unit 3 may remove noise as follows.

  FIG. 19 is a diagram illustrating noise removal by the specifying unit 3. This figure shows an example of removing noise from two row vectors 51a and 52b. FIG. 20 is a flowchart showing a noise removal procedure.

  First, the specifying unit 3 selects the maximum value in each column of one row vector 51a (step S1 in FIG. 20). In the example of FIG. 19, it is assumed that the value of the column x1 is the maximum value. Then, the specifying unit 3 compares the maximum value with a predetermined threshold (step S2 in FIG. 20).

  When the maximum value is smaller than the threshold (NO in step S2), the specifying unit 3 determines that the row vector 51a does not have the left end and the right end of the detection target, and sets the parameter flag to false. Then, the specifying unit 3 proceeds to the process for the next row vector 51b (step S5).

If the maximum value is greater than or equal to the threshold value (YES in step S2), the specifying unit 3 sets the column that takes the maximum value (column x1 in the example of FIG. 19) as the left and right end candidates of the detection target (step S3 in FIG. 20). Set the parameter flag to true. Then, the specifying unit 3 sets a mask in a predetermined area near the column (in the example of FIG. 19, two columns on the left and right of the column x1) (step S4 in FIG. 20). Next, the specifying unit 3 selects the maximum value among the columns in which the mask of the row vector 51a is not set (step S5). In the example of FIG. 19, it is assumed that the value of the column x2 is the maximum value.
Thereafter, steps S2 to S5 are repeated until the selected maximum value is smaller than the threshold value (until the parameter flag is set to false).

When the processing of the row vector 51a is completed, the same processing is performed on the row vector 51b (step S6). The same applies to noise removal for the column vector 52.
When two left and right end candidates are detected, the specifying unit 3 can set the left side as a left end candidate and the right side as a right end candidate.

  By setting the mask, the left end candidate and the right end candidate of the detection target are detected at positions separated by at least the number of pixels corresponding to the size of the mask. Similarly, the upper end candidate and the lower end candidate of the detection target are detected at positions separated by the number of pixels or more according to the mask size. Hereinafter, it is assumed that the left and right end candidates of the detection target are detected from the row vector 51 and the upper and lower end candidates are detected from the column vector 52 in this way.

  According to this process, the left and right edge candidates may not be found, only one may be found, or three or more may be found. When it is known by an application that one detection object fits in the input image, the threshold processing in step S2 is not performed, and two columns may be set as the left and right end candidates in the order in which the maximum values are selected. . The same applies to the upper and lower end candidates.

  FIG. 21 is a diagram illustrating a second example of the processing operation of the specifying unit 3. In the figure, left and right end candidates xla and xra are detected in the row vector 51a, and left and right end candidates xlb and xrb are detected in the row vector 51b. Also, upper and lower end candidates ytb and ybb are detected in the column vector 52b, and upper and lower end candidates ytc and ybc are detected in the column vector 52c. In this case, when a total of eight lines are drawn at positions corresponding to the candidates, four crosses (cross points), that is, (xla, ytb), (xra, ytc), (xlb, ybb), (xrb) , Ybc). These four points are the upper left, upper right, lower left and lower right candidate points of the detection object.

However, the sides of the rectangle obtained by connecting these four points are not parallel to the horizontal direction and the vertical direction. Therefore, for example, the specifying unit 3 connects four points (xl, yt), (xr, yt), (xl, yb), (xr, yb) obtained by using the average value of each point as in the following equation. The rectangle can be identified as the position of the detection object.
xl = (xla + xlb) / 2
xr = (xra + xrb) / 2
yt = (ytb + ytc) / 2
yb = (ybb + ybc) / 2

FIG. 22 is a diagram illustrating a third example of the processing operation of the specifying unit 3. In the figure, left and right end candidates xla and xra are detected in the row vector 51a, and left and right end candidates xlb and xrb are detected in the row vector 51b. Further, upper and lower end candidates ytb and ybb are detected in the column vector 52b. Also in this case, four crosses (cross points) are formed, that is, (xla, ytb), (xra, ytb), (xlb, ybb), and (xrb, ybb). These four points are the upper left, upper right, lower left and lower right candidate points of the detection object.
And the specific | specification part 3 can pinpoint the rectangle which connected 4 points | pieces obtained, for example using an average value with the position of a detection target object.

  As illustrated in FIGS. 21 and 22, when the cross is formed at four points, the specifying unit 3 can determine that the detection target is included in the input image.

  FIG. 23 is a diagram illustrating a fourth example of the processing operation of the specifying unit 3. In the figure, the left and right end candidates xa are also detected in the row vector 51a, and the left and right end candidates xb are detected in the row vector 51b. In addition, upper and lower end candidates yt and yb are detected in the column vector 52a. Therefore, a cross (cross point) is formed at two points, that is, (xa, yt) and (xb, yb).

  In this case, it can be interpreted that the upper end, the lower end, and the right end of the detection target are included in the input image, and only the left end is outside the input image. Therefore, the specifying unit 3 assumes that only a part of the detection target (vehicle in the example of FIG. 23) is included in the area 52A of the input image. For example, the upper right coordinates ((xa + xb) / 2, yt) and lower right coordinates ((xa + xb) / 2, yb).

  FIG. 24 is a diagram illustrating a fifth example of the processing operation of the specifying unit 3. In the figure, upper and lower end candidates yt and yb are detected in the column vector 52c. On the other hand, left and right edge candidates are not detected in the row vectors 51a and 51b. Thus, no cross is formed. In this case, the specifying unit 3 interprets that the upper and lower end candidates yt and yb are erroneous detections, and can determine that the detection target is not included in the input image.

  FIG. 25 is a diagram illustrating a sixth example of the processing operation of the specifying unit 3. In the figure, left and right end candidates xla and xra are detected in the row vector 51a, and left and right end candidates xlb and xrb are detected in the row vector 51b. On the other hand, the upper and lower end candidates are not detected in the column vectors 52a to 52c. Thus, no cross is formed.

  However, in this case, it can be interpreted that the left and right ends of the detection target are included in the input image and the upper and lower ends are outside the input image. Therefore, the specifying unit 3 assumes that a part of the detection target (a person in the example of FIG. 25) is included in the input image. For example, the left end coordinate (xla + xlb) / 2 and the right end coordinate (xra + xrb) / 2 can be specified.

  In FIGS. 23 to 25, the cross formed is less than four points, and in this case, various interpretations can be established. For example, in FIG. 24, the specifying unit 3 interprets that the right and left end candidates should be detected in the row vectors 51 a and 51 b as detection errors, and uses only the upper end coordinates yt and the lower end coordinates yb of the detection target. It is also possible to specify. In FIG. 25, the specifying unit 3 can interpret that the left and right end candidates are erroneous detections and determine that the detection target is not included in the input image.

  Therefore, depending on the application of the detection apparatus 100 and the shape of the detection target (for example, a person is assumed to be vertically long), the interpretation when the formed cross is less than 4 points is determined in advance. Just keep it.

  Incidentally, the specifying unit 3 may further calculate the positional relationship between the in-vehicle camera and the detection target based on (at least a part of) the upper and lower left and right positions of the detected detection target.

  FIG. 26 is a diagram illustrating a method for calculating a distance Z in the traveling direction between the in-vehicle camera 101 and the detection target object (preceding vehicle in this case) 102. In the figure, a vehicle 110 includes a vehicle main body 111, an in-vehicle camera 101 attached to the vehicle main body 111, and a detection device 100. The in-vehicle camera 101 is assumed to be installed at a known height h (for example, 130 cm). In addition, a virtual image plane is set at the position of the focal length f pixel of the in-vehicle camera 101. In this image plane, the center is the origin, the horizontal direction (traveling direction of the vehicle 110) is the x-axis, and the vertical direction (vertical downward) is the y-axis.

Then, it is assumed that the coordinate of the lower end position of the detection object 102 specified by the specifying unit 3 of the detection apparatus 100 is yb. At this time, as shown in the figure, the specifying unit 3 can calculate the distance Z in the traveling direction between the in-vehicle camera 101 and the detection target object 102 based on the following equation from the similarity of triangles.
Z = fh / yb

FIG. 27 is a diagram for explaining a method for calculating the lateral distances X _L and X _R between the in-vehicle camera 101 and the detection target object 102. It is assumed that coordinate axes are set as in FIG. 26, and the coordinates of the left end position and the right end position of the detection target 102 specified by the specifying unit 3 are xl and xr, respectively.
At this time, as shown in the figure, the specifying unit 3 can calculate the lateral distances X _L and X _R between the in-vehicle camera 101 and the detection target object 102 based on the following formula from the similarity relationship of the triangles.
X _L = Zxl / f
X _R = Zxr / f

  Thus, in the third embodiment, since the specifying unit 3 performs an appropriate interpretation according to the row vector 51 and the column vector 52, only a part of the detection target is included in the input image. However, the position of the detection target can be specified flexibly.

(Fourth embodiment)
In the first to third embodiments described above, the detection object is one type (for example, a person) in mind. In contrast, in the fourth embodiment described below, the positions of a plurality of types of detection objects (for example, people and vehicles) are specified.

FIG. 28 is a block diagram showing a schematic configuration of a neural network processing unit 1 ″ according to the fourth embodiment. When there are n types of detection objects (referred to as detection objects 1 to n), as a difference from the first embodiment (FIG. 3), the rows output by the horizontal processing unit 120h and the vertical processing unit 120v, respectively. The number of maps of the vector 51 and the column vector 52 is n. The row vector 51 and the column vector 52 in the i (i = 1 to n) th map indicate the detection result of the detection object i.
Note that a plurality of row vectors 51 and column vectors 52 can be output per map in accordance with the number of pixels of the input image and the number of stages of the pooling unit 122h.

  In the learning process, a plurality of sample images showing each of the detection objects 1 to n are used. In order to generate correct data, n input images 1 to n are input to the neural network processing unit 1 ″ for one sample image. When the detection target object i is included in the sample image, all the pixels of the input image k (k = 1 to n and k ≠ i) are 0 values, and the input image i is 1 only at the top, bottom, left, and right ends of the detection target. The other pixels are 0 values. Others are the same as in the first embodiment.

  In the detection process, the position of the detection target is specified for each map. That is, when detecting the detection target i, the specifying unit 3 specifies the position of the detection target i based on the row vector 51 and the column vector 52 in the i-th map.

  As described above, in the fourth embodiment, the number of maps of the row vector 51 and the column vector 52 output from the neural network processing unit 1 ″ is n, and the learning process is performed using the sample image including each detection target. Thus, n types of detection objects can be detected separately. Note that this embodiment can also be applied to the second and third embodiments.

(Fifth embodiment)
In the fifth embodiment described below, the neural network structure itself is the same as in the first embodiment, but the output row vector and column vector forms are different from those in the first to fourth embodiments. is there. Then, the detection target in the input image is not necessarily limited to the rectangles at the top, bottom, left and right ends, but is specified by an arbitrary shape.

FIG. 29 is a diagram for explaining the outline of the processing operation of the neural network processing unit 1 according to the fifth embodiment, and corresponds to FIG. When the detection target is the same rectangle as in FIG. 1B in the input image, the neural network processing unit 1 uses a row vector 51 ′ in which the values of only the xl to xr columns are 1 and the values of the other columns are 0. Also, a column vector 52 ′ in which only the values in the yt to yb rows are 1 and the values in the other rows are 0 is output. That is, in the row vector 51 ′ and the column vector 52 ′, the value of the row and the column corresponding to the position where the detection target is present is 1, and the value of the column and the row corresponding to the position where there is no detection target is 0. .
In order to output such a row vector 51 ′ and column vector 52 ′, the learning process may be performed as follows.

  FIG. 30 is a diagram for explaining an input image for generating correct data from a sample image, and corresponds to FIG. In the present embodiment, an input image shown in FIG. 28B is generated manually with respect to the sample image shown in FIG. That is, in the input image, the value of a pixel with a detection target is 1 and the value of a pixel with no detection target is 0.

  FIG. 31 is a diagram illustrating an input image for generating correct answer data from another sample image. In the sample image shown in FIG. 31A, a plurality of detection objects are included. In this case, in the input image ((b) in the figure), the value of a pixel having at least one detection object is set to 1, and the values of other pixels are set to 0. In the case of such an input image, the region where the pixel value is 1 is not limited to a rectangle.

  The input image as described above is input to the neural network processing unit 1. The learning neural network parameters are as described above. When the detection process is performed on the input image from the in-vehicle camera by performing the learning process in this manner and generating the detection neural network parameters, the row vector 51 ′ and the column vector 52 ′ as shown in FIG. Is output.

  In the present embodiment, the specifying unit 3 can determine that there is a detection target at the position of the column and row where the values in the row vector 51 ′ and the column vector 52 ′ are 1 without making a complicated interpretation. .

  Depending on the application, it is not necessary to specify the position of each detection object in detail, and it may be necessary to grasp only a rough position. For example, even if the detection result is “a group of people on the left side”, it can sufficiently contribute to safe driving. Therefore, especially when there are a plurality of detection objects, it is particularly effective to use this embodiment.

(Sixth embodiment)
In a sixth embodiment to be described next, it is assumed that the detection target is vertically long, such as a motorcycle or a person. The neural network processing unit in the detection device first detects the horizontal position of the detection target in the input image, and then detects the vertical position.

  FIG. 31 is a block diagram illustrating a schematic configuration of a neural network processing unit 1 ″ ″ according to the sixth embodiment. The neural network processing unit 1 ″ ″ performs detection processing as follows. First, it is assumed that there are no two or more detection objects. In the following, differences from the above-described embodiment will be mainly described.

  The map generated from the input image by the non-linear processing unit 11 is not input to the vertical processing unit 120v, but is input only to the horizontal processing unit 120h. The horizontal processing unit 120h and the vertical processing unit 120v do not exchange information directly.

The horizontal processing unit 120h outputs one row vector 51. In the row vector 51, as shown in FIG. 29, the value of the column corresponding to the position where the detection target is located is 1. Various methods are conceivable for reducing the number of output row vectors 51 to one. For example, an n-stage pooling unit 122h that pools the number of pixels in the vertical direction to ½ may be provided, and an input image having 2 ⁿ pixels in the vertical direction may be input to the horizontal processing unit 120h. An input image that has been resized or cut out in advance so that the number of pixels in the vertical direction is 2 ⁿ pixels may be input. Alternatively, the number of pixels in the vertical direction of the input image may be arbitrary, and the reduction rate in the pooling unit 122h may be adjusted according to the number of pixels in the vertical direction so that one row vector 51 is finally output.

  The neural network processing unit 1 ″ ″ according to the present embodiment includes a region specifying unit 13 that specifies a region including the detection target object from the input image based on the row vector. Hereinafter, it is assumed that the area specifying unit 13 specifies a rectangular area.

  FIG. 32 is a diagram for explaining the processing operation of the area specifying unit 13. The area specifying unit 13 specifies the following rectangular area. The center of the rectangular area coincides with the center of the column that is a single value in the row vector 51. The number (width) of pixels in the horizontal direction of the rectangular area may be a predetermined fixed value (for example, 64 pixels), or the input image can be selected from predetermined options (for example, three of 32 pixels, 128 pixels, and 512 pixels). You may select according to the number of pixels, and may increase, so that there are many continuation of 1 value. The number of vertical pixels (height) of the rectangular area matches the number of vertical pixels of the input image.

  Referring back to FIG. 31, the rectangular area specified by the area specifying unit 13 is input to the vertical processing unit 120v as a map based on the input image. One or more nonlinear processing units may be provided between the region specifying unit 13 and the vertical direction processing unit 120v. If the row vector 51 has no value, that is, if the detection target is not detected, the area specifying unit 13 does not specify a rectangular area, and therefore the vertical processing unit 120v does not perform processing.

  The vertical processing unit 120v to which the rectangular area is input processes the rectangular area and outputs one column vector 52. In the column vector 52, as shown in FIG. 29, the value of the row corresponding to the position where the detection target is located is 1.

If the number of pixels in the horizontal direction of the rectangular area is a fixed value, the number of pooling sections 122v corresponding to the number may be provided. When the number of pixels in the horizontal direction of the rectangular area is selected from a plurality of options, the vertical processing units 120v having different numbers of stages of the pooling unit 122v are provided for the number of options, and any one of them processes the rectangular area. May be. For example, when the options are 32 (= 2 ⁵ ) pixels, 128 (= 2 ⁷ ) pixels, and 512 (= 2 ⁹ ) pixels, the vertical processing unit 120 v having the five-stage pooling section 122 v and the seven-stage pooling section 122 v The vertical processing unit 120v and the vertical processing unit 120v including the nine-stage pooling unit 122v are provided, and the vertical processing unit 120v corresponding to the selected number of horizontal pixels performs processing. Conceivable. If the number of pixels in the horizontal direction is arbitrary, the reduction rate in the pooling unit 122v may be adjusted.

  Based on the row vector 51 and the column vector 52 obtained one by one as described above, the specifying unit 3 (FIG. 2) detects a detection target from the input image.

  When there are two or more detection objects, the area specifying unit 13 specifies a rectangular area for each detection object, and the vertical direction processing unit 120v may perform processing for each of them.

  The advantage of processing in this way will be described. When the detection target is a two-wheeled vehicle or a pedestrian, it is unlikely that the detection target exists in the vertical direction. Therefore, first, the row vector 51 is generated, the horizontal position of the detection target is specified, and the rectangular area is specified, so that the detection target is positioned near the center. Therefore, in the processing in the vertical processing unit 120v, the characteristics of the detection target are sufficiently extracted, and improvement in detection accuracy can be expected. Furthermore, when it is determined that the detection target does not exist based on the row vector 51, the vertical processing unit 120v does not need to perform processing, and the amount of calculation can be reduced.

  In the neural network processing unit 1 ″ ″ of the present embodiment, the neural network parameters in the horizontal direction processing unit 120 h and the neural network parameters in the vertical direction processing unit 120 v can be learned separately. That is, correct data of the row vector 51 is generated by the processing of the horizontal processing unit 120h using the learning processing neural network parameters described in the first embodiment. Then, the learning unit 2 (FIG. 2) may generate a neural network parameter for detection processing so that the row vector 51 approaches the correct data. The same applies to the neural network parameters of the vertical processing unit 120v.

  As described above, in the sixth embodiment, the row vector 51 is first generated by performing the pooling in the vertical direction, and the rectangular region including the detection target is specified based on the row vector 51. Subsequently, a column vector 52 is generated by performing horizontal pooling on the rectangular area. Since the features of the detection object are included in the rectangular area, the detection accuracy can be improved. Further, when it is found that there is no object to be detected at the time when the pooling in the vertical direction is performed, the pooling in the horizontal direction is not performed, so that the amount of calculation can be reduced.

  In this embodiment, since it is assumed that the detection target is vertically long, first, the vertical pooling is performed. However, when the detection target is horizontal, the horizontal pooling may be performed first. Good.

  In the first to sixth embodiments described above, detection of a person, a vehicle, or the like from an image from a vehicle-mounted camera has been mainly described, but the present invention can also be applied to other uses.

  The embodiment described above is described for the purpose of enabling the person having ordinary knowledge in the technical field to which the present invention belongs to implement the present invention. Various modifications of the above embodiment can be naturally made by those skilled in the art, and the technical idea of the present invention can be applied to other embodiments. Therefore, the present invention should not be limited to the described embodiments, but should be the widest scope according to the technical idea defined by the claims.

1, 1 ′, 1 ″, 1 ′ ″ Neural network processing unit 2 Learning unit 3 Identification unit 11 Nonlinearization processing unit 12h, 12h ′ Horizontal direction map generation unit 12v, 12v ′ Vertical direction map generation unit 121h, 121h ′ , 121v, 121v ′ Non-linearization processing unit 122h, 122v Pooling unit 120h Horizontal direction processing unit 120v Vertical direction processing unit 13 Area setting unit 51, 51a, 51b, 51 ′ Row vector 52, 52a-52c, 52 ′ Column vector 100 detection Device 101 Car-mounted camera 102 Target object 110 Vehicle 111 Vehicle body

Claims (19)

A horizontal that outputs a row vector indicating a horizontal position of a detection object in the input image while performing a non-linearization process and a pooling process on the map based on the input image and maintaining a horizontal resolution of the input image. A direction processing unit;
A non-linearization process and a pooling process are performed on the map based on the input image, and a column vector indicating the vertical position of the detection target in the input image is output while maintaining the vertical resolution of the input image. And a vertical direction processing unit. The horizontal processing unit has a first pooling unit that performs a pooling process only in the vertical direction on the map based on the input image,
The neural network processing apparatus according to claim 1, wherein the vertical direction processing unit includes a second pooling unit that performs a pooling process only in a horizontal direction on a map based on the input image. The horizontal direction processing unit includes a first horizontal direction map generation unit and a second horizontal direction map generation unit,
The vertical direction processing unit includes a first vertical direction map generation unit and a second vertical direction map generation unit,
The first horizontal direction map generation unit performs a non-linearization process and a pooling process only in the vertical direction on the map based on the input image to generate a first output map,
The first vertical map generation unit performs a non-linearization process and a horizontal pooling process on the map based on the input image to generate a second output map,
The second horizontal map generation unit performs a non-linearization process and a pooling process on the first output map and the second output map, and outputs a third output having the same horizontal resolution as the first output map. Generate a map
The second vertical map generation unit performs a non-linearization process and a pooling process on the first output map and the second output map, and outputs a fourth output having the same vertical resolution as that of the second output map. Generate a map
The row vector is generated based on the third output map;
The neural network processing device according to claim 1, wherein the column vector is generated based on the fourth output map. The second horizontal map generation unit includes:
A convolution operation is performed on the first output map and the second output map to generate a first intermediate map and a second intermediate map having the same resolution as the first output map, and the first intermediate map and the second output map are generated. A first non-linearization processing unit that adds two intermediate maps to generate a third intermediate map;
A third pooling unit that generates a third output map by performing a pooling process only in a vertical direction on the third intermediate map;
Have
The second vertical map generation unit includes:
A convolution operation is performed on the first output map and the second output map to generate a fourth intermediate map and a fifth intermediate map having the same resolution as that of the second output map, respectively. A second non-linearization processing unit that adds the five intermediate maps to generate a sixth intermediate map;
A fourth pooling unit that performs a pooling process only in the horizontal direction on the sixth intermediate map to generate the fourth output map;
The neural network processing apparatus according to claim 3, comprising: The horizontal and vertical resolutions of the second intermediate map are p1 times and 1 / q1 times (p1 and q1 are integers), respectively, of the horizontal and vertical resolutions of the second output map,
The first non-linearization processing unit includes an inner product of a certain pixel value in the second output map and a filter coefficient of the first filter, the second intermediate corresponding to a certain part in the second output map. The process of setting the values of the pixels in the map and the (p1-1) pixels to the right thereof is performed while shifting the first filter by q1 pixels in the vertical direction of the second output map. Two intermediate maps,
The horizontal and vertical resolutions of the fourth intermediate map are 1 / p2 times and q2 times (p2 and q2 are integers), respectively, the horizontal and vertical resolutions of the first output map,
The second non-linearization processing unit includes the fourth intermediate corresponding to an inner product of a certain pixel value of the first output map and a filter coefficient of the second filter with a certain portion of the first output map. The process of setting the values of the pixels in the map and the (q2-1) pixels below it is performed while shifting the second filter by p2 pixels in the horizontal direction of the first output map. The neural network processing apparatus according to claim 4, wherein four intermediate maps are generated. An image in which the pixels corresponding to the upper, lower, left, and right edges of the detection object in the sample image have the first value and the other pixels have the second value is the input image,
The horizontal direction processing unit and the vertical direction processing unit perform the non-linearization process including a convolution operation using a filter whose center is the first value and the other is the second value, and the correct data is the correct answer data. The neural network processing apparatus according to claim 1, wherein the neural network processing apparatus outputs a row vector and the column vector. In the sample image, a pixel where the detection target is a first value and another pixel is a second value is the input image,
The horizontal direction processing unit and the vertical direction processing unit perform the non-linearization process including a convolution operation using a filter whose center is the first value and the other is the second value, and the correct data is the correct answer data. The neural network processing apparatus according to claim 1, wherein the neural network processing apparatus outputs a row vector and the column vector.   The neural network processing apparatus according to claim 6 or 7, wherein a neural network parameter used in the non-linearization process is generated based on the sample image and the correct answer data. The row vector in which the column corresponding to the left end and the right end of the detection target object is the first value, and the column corresponding to the left side from the left end and the column corresponding to the right end from the right end is the second value. And
The column in which the vertical processing unit corresponds to the first value in the row corresponding to the upper end and the lower end of the detection target, and the row corresponding to the upper side from the upper end and the lower side from the lower end is the second value. The neural network processing apparatus of claim 8, wherein the neural network parameters are generated to output a vector.   10. The sample image according to claim 6, wherein the sample image includes a sample image including a first detection target and a sample image including a second detection target different from the first detection target. Neural network processing device.   The neural network processing apparatus according to claim 6, wherein the sample image includes a sample image including only a part of the detection target. An area specifying unit for specifying an area including the detection target object from the input image based on the row vector;
The neural network processing apparatus according to claim 1, wherein the vertical processing unit performs nonlinear processing and pooling processing on the specified region as a map based on the input image.   The neural network processing device according to claim 12, wherein when it is determined that the detection target is not included in the input image based on the row vector, the vertical processing unit does not perform nonlinear processing and pooling processing. . A neural network processing device according to any one of claims 1 to 13,
And a specifying unit that specifies a position of the detection target in the input image based on the row vector and the column vector output by the neural network device. The specific part is:
A column in the row vector, the column of which satisfies the first threshold condition as a left end candidate or a right end candidate of the detection object,
The detection apparatus according to claim 14, wherein a row in the column vector whose value satisfies a second threshold condition is set as an upper end candidate or a lower end candidate of the detection target. The left end candidate and right end candidate of the detection target specified by the specifying unit are separated by a number of first pixels or more,
The detection device according to claim 14 or 15, wherein an upper end candidate and a lower end candidate of the detection target specified by the specifying unit are separated by a second pixel number or more. A vehicle body,
A camera attached to the vehicle body,
A vehicle comprising: the detection device according to claim 14, which processes an image from the camera as the input image. Performing a non-linearization process and a pooling process on a map based on the input image, and outputting a row vector indicating a horizontal position of the detection target in the input image while maintaining a horizontal resolution of the input image When,
A non-linearization process and a pooling process are performed on the map based on the input image, and a column vector indicating the vertical position of the detection target in the input image is output while maintaining the vertical resolution of the input image. And a neural network processing method. Performing a non-linearization process and a pooling process on a map based on the input image, and outputting a row vector indicating a horizontal position of the detection target in the input image while maintaining a horizontal resolution of the input image When,
A non-linearization process and a pooling process are performed on the map based on the input image, and a column vector indicating the vertical position of the detection target in the input image is output while maintaining the vertical resolution of the input image. And steps to
Identifying the position of the detection object in the input image based on the row vector and the column vector.