JP5515871B2 - Object recognition apparatus and program - Google Patents

Description

  The present invention relates to an object recognition apparatus and program, and more particularly to an object recognition apparatus and program for controlling an imaging apparatus so that an image suitable for recognition is captured when a specific object is recognized from the captured image.

  2. Description of the Related Art In recent years, an increasing number of vehicles are equipped with an object recognition device that performs image processing on images around a vehicle imaged by an in-vehicle camera, recognizes a specific object such as a pedestrian or vehicle, and presents a recognition result to a driver. In such an object recognition apparatus, it is necessary to control so as to obtain an image suitable for recognition.

  For example, an exposure control method for an in-vehicle monitoring camera has been proposed in which photometry is performed by a photometric sensor and the exposure time of an electronic shutter of a CCD is determined based on the photometric signal (see Patent Document 1). In the exposure control method described in Patent Document 1, auto bracketing shooting is performed based on an exposure time based on a photometric signal, and the image data that is the proper exposure from the three pieces of image data input to the memory or the closest to the proper exposure. Image data is selected, input to the image processing unit, set to a new reference exposure time, and the next auto bracketing shooting is performed.

  In addition, reference information in which an object is detected from the image including the determination mark and the visibility evaluation value indicating the visibility state is associated with the reliability of the detection performance of the object detection means in the visibility state is stored in the reliability determination information storage unit in advance. And storing, extracting the image feature of the determination marker region from the image including the determination indicator, calculating the visibility evaluation value based on the extracted image feature, and storing the calculated visibility evaluation value in the reliability determination information storage means An image processing apparatus has been proposed in which the reliability of the detection performance of the object detection unit is determined by collating with the reference information, and a predetermined process is executed based on the detection result and the determination result (see Patent Document 2). .

  Further, in a camera system having a normal mode and a camera shake reduction (hjr) mode, a first exposure time-gain product is generated by multiplying a normal mode exposure time by a gain to obtain a first exposure time-gain. The exposure time and gain for the hjr mode are determined by modifying the exposure time and gain in the normal mode so as to reduce the difference between the product of the second and the second exposure time-gain product for the hjr mode. A camera system has been proposed (see Patent Document 3).

JP 2007-295093 A JP 2001-84377 A Japanese translation of PCT publication 2010-504718

  However, there is a problem that an image suitable for recognition is not necessarily obtained by appropriate exposure obtained by the exposure control method described in Patent Document 1. For example, if the result of the photometric signal is low, the exposure time can be increased. However, if the exposure time is increased, the image is unsuitable for recognition with blurring. Although it is possible to increase the gain by shortening the exposure time, in this case, the noise of the image increases, and the image is also unsuitable for recognition.

  The image processing apparatus described in Patent Document 2 evaluates whether an image is suitable for recognition based on edge strength, frequency component, color, and the like, but the relationship between the evaluation and imaging parameters is described. Therefore, there is a problem that an image suitable for recognition cannot be obtained.

  In the camera system described in Patent Document 3, the exposure time and gain in the camera shake reduction mode are determined based on the product of the exposure time and gain in the normal mode, but can be obtained by the exposure time and gain in the normal mode. If the image is not suitable for recognition, there is a problem that images that are not suitable for recognition continue to be obtained.

  The present invention has been made to solve the above-described problems, and can reduce an image blur and noise to obtain an image suitable for recognition for recognizing a specific object. And to provide a program.

  In order to achieve the above object, the object recognition apparatus of the present invention provides a specific image included in a captured image obtained by amplifying charges accumulated and stored with an exposure time set by an imaging unit with a set gain. Recognizing means for recognizing an image showing the object, an extracting means for extracting characteristics of a partial image including an image showing the specific object recognized by the recognizing means and a peripheral image of the image showing the specific object, Recognition of an image indicating the specific object recognized by the recognition means based on the relationship between the defined image characteristics and the ease of recognition by the recognition means, and the characteristics of the partial image extracted by the extraction means An evaluation value calculating means for calculating an evaluation value indicating the ease of the evaluation, an evaluation value calculated by the evaluation value calculating means is increased, and a product of the exposure time and the gain is a predetermined value. Is configured to include a, an imaging control means for controlling the serial exposure time and the gain.

  According to the object recognition apparatus of the present invention, the specific object included in the captured image obtained by the recognition unit amplifying the charge captured and accumulated with the exposure time set by the imaging unit with the set gain. The extraction unit extracts the characteristics of the partial image including the image indicating the specific object recognized by the recognition unit and the peripheral image of the image indicating the specific object. Then, the evaluation value calculation means determines the specific object recognized by the recognition means based on the relationship between the predetermined image characteristics and the ease of recognition by the recognition means, and the characteristics of the partial image extracted by the extraction means. An evaluation value indicating the ease of recognizing the displayed image is calculated. Then, the imaging control unit controls the exposure time and the gain so that the evaluation value calculated by the evaluation value calculation unit is high and the product of the exposure time and the gain becomes a predetermined value. Here, the predetermined value is a value in which the product of the exposure time and the gain is a constant value with the currently set product of the exposure time and the gain, and a value in which the constant value has a slight width. be able to.

  As described above, the relationship between the predetermined image characteristic and the evaluation value indicating the ease of recognition and the evaluation value calculated based on the characteristic of the partial image are high, and the product of the exposure time and the gain is predetermined. Since the exposure time and gain as values are calculated, image blur and noise can be reduced, and an image suitable for recognition in order to recognize a specific object can be obtained.

  Further, the imaging control means can control the exposure time and the gain when the evaluation value calculated by the evaluation value calculation means is lower than a predetermined evaluation value threshold.

  Further, the object recognition apparatus of the present invention includes a reliability calculation unit that calculates a specific object-likeness of an image showing the specific object as a recognition reliability based on a recognition result of the recognition unit. The imaging control means can control the exposure time and the gain when the reliability calculated by the reliability calculation means is lower than a predetermined reliability threshold.

  Further, the object recognition apparatus of the present invention may be configured to include a storage control unit that controls the storage unit to store the evaluation values calculated by the evaluation value calculation unit in time series, and the imaging control unit Is based on the time-series evaluation values stored in the storage means by the storage control means, and when the change amount of the evaluation value is larger than a predetermined evaluation value change amount threshold, the exposure time and the gain are Can be controlled.

The front Symbol storage control means, wherein the reliability reliability calculated by the calculation means is controlled to be stored in the storage means in time series, the image pickup control means in said storage means by said storage control means Based on the stored time-series reliability, the exposure time and the gain are controlled when the amount of change in the direction in which the reliability decreases is larger than a predetermined reliability change amount threshold. be able to.

  Further, the object recognition apparatus of the present invention can be configured to include detection means for detecting the movement of the vehicle on which the imaging means is mounted, and the imaging control means can detect the movement of the vehicle detected by the detection means. The exposure time and the gain can be controlled when the magnitude of is greater than a predetermined motion threshold.

  As described above, since the control by the imaging control unit is performed based on the evaluation value, the reliability, the amount of change in the evaluation value, the amount of change in the reliability, and the magnitude of the vehicle motion, it is unnecessary when the imaging control is not necessary. Processing can be suppressed.

  Further, the object recognition apparatus of the present invention is characterized in that the predetermined image characteristic used when the evaluation unit calculates the evaluation value based on the time-series evaluation value stored in the storage unit by the storage control unit. And updating means for updating the relationship between the recognition means and the ease of recognition by the recognition means. Thereby, a more appropriate evaluation value can be calculated.

  Further, the characteristics of the partial image, the average of the histogram of the edge direction of the partial image and the variance of the frequency, the ratio of the sum of the power spectrum of the partial image and the sum of the power spectrum of the high-frequency component of the partial image, Or it can be set as the ratio between the maximum value and the minimum value of the histogram of the luminance value of the partial image.

  The recognition unit recognizes an image indicating a specific object by comparing a histogram of the edge direction of the captured image and a histogram of the edge direction of the image indicating the specific object stored in advance, and the extracting unit By using the histogram of the edge direction of the partial image obtained by the recognition means, the average frequency and the variance of the frequency of the edge direction histogram can be extracted as the characteristics of the partial image. In this way, the amount of calculation can be reduced by reusing the histogram in the edge direction obtained by the recognition means by the extraction means.

The object recognition program also displays an image indicating a specific object included in a captured image obtained by amplifying a computer with a set gain, the charge captured and accumulated with the exposure time set by the imaging means. Recognizing recognition means, extracting means for extracting characteristics of a partial image including an image showing a specific object recognized by the recognizing means and a peripheral image of the image showing the specific object, characteristics of the predetermined image and the recognition An evaluation value indicating the ease of recognizing the image representing the specific object recognized by the recognizing means based on the relationship with the ease of recognizing by the means and the characteristics of the partial image extracted by the extracting means. evaluation value calculating means for calculating, storing control means for controlling so as to store the time series in the storage means the evaluation value calculated by the evaluation value calculating means, and said storage control hand Based on the evaluation value of the time series stored in the storage unit by, when the amount of change in the evaluation value is greater than a predetermined amount of change in the evaluation value threshold, evaluation value calculated by the evaluation value calculating means is high And a program for functioning as an imaging control means for controlling the exposure time and the gain so that a product of the exposure time and the gain becomes a predetermined value.

  The storage medium for storing the program of the present invention is not particularly limited, and may be a hard disk or a ROM. Further, it may be a CD-ROM, a DVD disk, a magneto-optical disk or an IC card. Furthermore, the program may be downloaded from a server or the like connected to the network.

  As described above, according to the present invention, the evaluation value calculated based on the relationship between the predetermined image characteristic and the evaluation value indicating the ease of recognition and the characteristic of the partial image is increased, and the exposure is performed. Since the exposure time and gain at which the product of time and gain becomes a predetermined value are calculated, it is possible to reduce image blurring and noise and obtain an image suitable for recognition for recognizing a specific object. An effect is obtained.

It is a block diagram which shows the structure of the object recognition apparatus which concerns on 1st Embodiment. It is a figure which shows an example of an evaluation value map. It is a figure for demonstrating calculation of exposure time and a gain. It is a flowchart which shows the content of the object recognition process routine in the object recognition apparatus of 1st Embodiment. It is an image figure which shows the flow of extraction of an image characteristic, and evaluation value calculation. It is a block diagram which shows the structure of the object recognition apparatus which concerns on 2nd Embodiment. It is a figure for demonstrating the fluctuation | variation of the evaluation value by rapid vehicle motion. It is a flowchart which shows the content of the object recognition process routine in the object recognition apparatus of 2nd Embodiment. It is a figure for demonstrating the other example of an image characteristic. It is a figure for demonstrating the other example of an image characteristic.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, a case where the present invention is applied to an object recognition apparatus mounted on a vehicle will be described.

  As illustrated in FIG. 1, the object recognition device 10 according to the first embodiment includes an imaging device 12 that captures a range including a recognition target region, and a specific object based on a captured image output from the imaging device 12. The computer 16 that executes the object recognition processing routine for recognizing the image and the display device 18 for displaying the processing result of the computer 16 are provided.

  The imaging device 12 images a range including a recognition target region with a set exposure time, and amplifies an imaging unit (not shown) that generates an image signal, and an analog signal generated by the imaging unit with a set gain. A gain adjustment unit (not shown), an A / D conversion unit (not shown) that converts an image signal that is a gain-adjusted analog signal into a digital signal, and temporarily stores the A / D converted image signal Image memory (not shown).

  The computer 16 includes a CPU that controls the entire object recognition apparatus 10, a ROM or HDD as a storage medium that stores a program for an object recognition processing routine, which will be described later, and various types of information, a RAM that temporarily stores data as a work area, and these It is comprised including the bus which connects. In the case of such a configuration, a program for realizing the function of each component is stored in a storage medium such as a ROM or HDD, and each function is realized by executing the program by the CPU. To.

  If the computer 16 is described with functional blocks divided for each function realizing means determined based on hardware and software, it is included in the captured image captured by the imaging device 12 and input to the computer 16 as shown in FIG. A specific object recognition unit 20 for recognizing an image indicating a specific object to be detected, and an imaging control unit 40 for controlling the imaging device 12 so as to obtain an image that can be easily recognized based on a recognition result in the specific object recognition unit 20. And can be represented by a configuration including

  The specific object recognition unit 20 further compares a window image extraction unit 22 that extracts a window image of a predetermined region from the input captured image, and the window image extracted by the window image extraction unit 22 and the identification model. The recognition unit 24 for recognizing whether or not the window image is a specific object, the identification model storage unit 26 in which the identification model is stored, and the display device 18 are controlled so as to display the recognition result superimposed on the captured image. And a display control unit 28.

  The window image extraction unit 22 cuts an image from a captured image while moving a predetermined size window (referred to as a search window) by a predetermined movement amount (referred to as a search step) per step. Here, the cut image is referred to as a window image, and the size of the window image (that is, the size of the search window) is referred to as a window size. A plurality of types of window sizes are set for recognizing objects of various sizes, and the window image extraction unit 22 extracts window images using search windows of all set window sizes. In addition, the window image extraction unit 22 converts the extracted window image into an image having a preset number of pixels (for example, an image of 16 horizontal x 32 vertical pixels).

  The recognizing unit 24 recognizes whether or not the window image is an image indicating a specific object by using a technique such as SVM (Support Vector Machine) or HOG (Histogram of Oriented Gradient).

  The identification model storage unit 26 stores an identification model generated by learning in advance according to the recognition method by the recognition unit 24. Although the case where the identification model storage unit 26 is provided in the computer 16 will be described here, the identification model is stored in the storage unit of another external device and connected to the other external device via a network or communication unit. And it is good also as a structure which reads the identification model memorize | stored in the memory | storage means of another external apparatus.

  The imaging control unit 40 further extracts an image characteristic extraction unit 42 that extracts an image characteristic of a window image recognized as an image showing a specific object by the recognition unit 24, and an image of the window image extracted by the image characteristic extraction unit 42. Based on the characteristics and a predetermined evaluation value map, an evaluation value calculation unit 44 that calculates an evaluation value indicating the ease of recognizing the window image, and an imaging device based on the evaluation value calculated by the evaluation value calculation unit 44 And an imaging control value calculation unit 46 that calculates an imaging control value for controlling the exposure time and gain set in the above and outputs the imaging control value to the imaging device 12.

  The image characteristic extraction unit 42 recognizes the window image recognized as an image showing a specific object by the recognition unit 24, the size of the window image, the position of the window image on the captured image, and the attribute of the recognized object (pedestrian , Motorcycles, vehicles, etc.). Here, the window image is acquired as one piece of information included in the recognition result, but a partial image including an image indicating a specific object and its peripheral image may be acquired. Further, the reason for including the peripheral image is to evaluate the ease of recognition using the image including the background because the ease of recognition varies depending on the complexity of the background.

  Further, the image characteristic extraction unit 42 detects an edge from the acquired window image, and generates a histogram of the edge direction indicating the frequency according to the direction of the detected edge. In general, it is expected that an image suitable for recognition will include edges in various directions in an appropriate manner. Extract as

  The evaluation value calculation unit 44 calculates an evaluation value for the image characteristics of the window image extracted by the image characteristic extraction unit 42 using a predetermined evaluation value map. As shown in FIG. 2, the evaluation value map defines the relationship between image characteristics and evaluation values, and the evaluation value is higher as the image is suitable for recognition. An evaluation value map plots multiple images with known recognition results and their image characteristics on a graph with the frequency averaged on the horizontal axis and the frequency variance on the vertical axis, and many image characteristics of well-captured images Created by setting a multi-step area with a high evaluation value for the plotted area, and a low evaluation value for the area where many image characteristics of the image with blurred image and noise are generated. can do. For example, as shown in FIG. 2, it is possible to create a map of contour lines having a higher evaluation value at the center and a lower evaluation value toward the outside.

  The relationship between image characteristics and evaluation values will be described in more detail using this evaluation value map. For example, it is assumed that the window image in which the image characteristics are plotted at the position indicated by A in the figure is an image having a high evaluation value suitable for recognition. Even if the image was shot at the same brightness as the image with a high evaluation value, the longer the exposure time and the lower the gain, the more the image is affected by the movement of the vehicle and the image is strongly blurred. The image becomes a low value. The amount of edges detected from such an image is small, and the edge direction histogram is biased due to the directionality of the blur. That is, the average frequency becomes smaller and the frequency variance becomes larger. In this case, it is plotted at a position indicated by B in FIG. On the other hand, even an image shot at the same brightness as an image with a high evaluation value, as the exposure time is short and the gain is high, the noise of the entire image increases, resulting in an image with a low evaluation value that is not suitable for recognition. . Since the edges detected from such an image are evenly distributed in various directions, the frequency dispersion is low and is plotted at a position indicated by C in FIG.

  The imaging control value calculation unit 46 calculates the exposure time and the gain as the imaging control value based on the evaluation value map so that the evaluation value becomes high. Specifically, as shown in FIG. 3, in order to keep the brightness of the image constant, the change in the evaluation value when the exposure time and the gain are changed so that the product of the exposure time and the gain is constant. The straight line shown is made to correspond on the evaluation value map. Then, the exposure time and gain corresponding to the point where the evaluation value is maximum on this straight line are calculated as the imaging control value. Note that the product of the exposure time and the gain being constant is not limited to a strictly constant case, and the product of the exposure time and the gain is a predetermined value or a predetermined value depending on the setting of the evaluation value in the evaluation value map. Including the range of ± α. The imaging control value calculation unit 46 outputs the calculated imaging control value to the imaging device 12. In the imaging device 12, the exposure time and gain calculated by the imaging control value calculation unit 46 are set, and imaging is performed with the set exposure time and gain.

  Next, an object recognition processing routine executed by the computer 16 of the object recognition apparatus 10 according to the first embodiment will be described with reference to FIG.

  In step 100, a captured image captured by the imaging device 12 is acquired. Next, in step 102, a search window of, for example, 16 × 32 pixels is displayed on the captured image, for example, a predetermined region (for example, an upper left corner region) of the captured image. And a 16 × 32 pixel window image is extracted from the captured image using the set search window.

  Next, in step 104, using the identification model stored in the identification model storage unit 26, it is recognized whether or not the window image is an image showing a specific object using a technique such as SVM or HOG.

  Next, in step 106, based on the recognition result in step 104, it is determined whether or not the window image is an image showing a specific object. If the determination is affirmative, the process proceeds to step 108, and if the determination is negative, the process ends.

  In step 108, as shown in FIG. 5, as the recognition result in step 104, the window image recognized as an image showing a specific object, the size of the window image, the position of the window image on the captured image, recognition The attribute (such as a pedestrian, a two-wheeled vehicle, or a vehicle) of the selected object is acquired. Then, an edge is detected from the acquired window image, and a histogram of the edge direction indicating the frequency according to the direction of the detected edge is generated. Then, as the image characteristics of the window image, the average frequency and the frequency variance of the histogram in the edge direction are extracted.

  Next, in step 110, the image characteristic extracted in step 108 is plotted on a predetermined evaluation value map, and an evaluation value E indicating the ease of recognition corresponding to the image characteristic is calculated.

  Next, in step 112, by determining whether or not the evaluation value E calculated in step 110 is smaller than a predetermined evaluation value threshold Eth, the window image extracted in step 102 is an image suitable for recognition. It is determined whether or not. If E <Eth, it is determined that the image is not suitable for recognition, and the process proceeds to step 114. If E ≧ Eth, it is determined that the image is suitable for recognition, and the process ends. To do.

  In step 114, a straight line indicating a change in evaluation value when the exposure time and gain are changed so that the product of the exposure time and gain is constant is made to correspond on the evaluation value map, and the evaluation value is maximum on this straight line. The exposure time and gain corresponding to the points are calculated as imaging control values, the calculated imaging control values are output to the imaging device 12, and the process ends.

  Thereby, the captured image of the next frame becomes a captured image suitable for recognition captured by the imaging device 12 in which the exposure time and the gain are set so that the evaluation value becomes high.

  In the window image extraction process in step 102, the window image is extracted while scanning the entire captured image for each size of the search window, and the recognition process is performed on each window image extracted in step 104. In some cases, a window image indicating a plurality of specific objects is recognized from one captured image. In that case, the processing of step 108 and step 110 is executed for each window image to calculate the evaluation value E. If any one of the evaluation values E is smaller than the evaluation value threshold Eth, the determination in step 112 is affirmed. It is possible to make a determination, or when all the evaluation values E are smaller than the evaluation value threshold Eth, an affirmative determination can be made in step 112 above.

  As described above, according to the object recognition apparatus of the first embodiment, a specific object is indicated in the evaluation value map indicating the relationship between the predetermined image characteristic and the evaluation value indicating the ease of recognition. An image characteristic of a window image recognized as an image is plotted to calculate an evaluation value, and the exposure time and gain at which the evaluation value is maximized while maintaining the brightness of the image are calculated. Noise can be reduced and a captured image suitable for recognition in order to recognize a specific object can be obtained.

  In the first embodiment, the case where the imaging control value is calculated when the evaluation value is smaller than the evaluation value threshold has been described. However, the threshold value determination of the evaluation value is performed by omitting the process of step 112 above. Alternatively, the imaging control value may be calculated so that the evaluation value becomes high.

  Next, a second embodiment will be described. It differs from the first embodiment in that imaging control processing is performed when a sudden movement occurs in the vehicle. In addition, about the structure same as the object recognition apparatus 10 of 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  As shown in FIG. 6, the object recognition device 210 according to the second embodiment includes an imaging device 12, a vehicle motion sensor 14 that detects the motion of the vehicle such as an acceleration sensor and a yaw rate sensor, a computer 216, and a display device. 18. When the computer 216 is described in terms of functional blocks divided for each function realization means determined based on hardware and software, as shown in FIG. 6, the computer 216 is included in a captured image captured by the imaging device 12 and input to the computer 216. A specific object recognition unit 20 that recognizes an image indicating a specific object and an image suitable for recognition can be obtained based on the recognition result of the specific object recognition unit 20 when a sudden vehicle motion is detected. And an imaging control unit 240 that controls the imaging device 12.

  The imaging control unit 240 further includes an image characteristic extraction unit 42, an evaluation value calculation unit 44, a reliability calculation unit 48 that calculates the reliability of recognition based on the recognition result in the recognition unit 24, and an evaluation value calculation. A time-series storage control unit 50 for controlling the evaluation value calculated by the unit 44 and the reliability calculated by the reliability calculation unit 48 to be stored in a predetermined storage area in time series, and whether to start imaging control A configuration including a control start determination unit 52 that determines whether or not, an evaluation value map update unit 54 that updates an evaluation value map based on evaluation values stored in time series, and an imaging control value calculation unit 46 Can be expressed as

  The reliability calculation unit 48 is based on the recognition result in the recognition unit 24 and is high when the recognition result is correct, and low when there is a possibility of erroneous recognition due to blurring of the image. R is calculated. The recognition unit 24 uses a method such as SVM or HOG to compare the window image with the identification model to calculate a score indicating the likelihood of a specific object, and performs recognition based on whether the score is equal to or greater than a predetermined threshold. In this case, this score can be used as the reliability R as it is. Further, in the method of performing recognition by configuring a plurality of discriminators in multiple stages, the reliability R may be calculated according to how many classifiers have been discriminated.

  The time-series storage control unit 50 recognizes a window image recognized as an image showing a specific object, the size of the window image, the position of the window image on the captured image, and the attribute of the recognized object (pedestrian, two-wheeled vehicle, Vehicle recognition), the evaluation value E calculated by the evaluation value calculation unit 44, and the reliability R calculated by the reliability calculation unit 48 are associated with each other and stored in a predetermined storage area in time series. Control to do. When the attributes of the object are the same, tracking is performed so as to minimize the degree of change in position and size on the image, and time-series association is performed. For example, information on the speed and pattern of movement for each attribute of the object is stored in advance, and tracking is performed according to the attribute of the object with reference to the stored information. Further, the movement of the object may be predicted by a Kalman filter or the like during tracking. Further, in time series association, optimization may be performed using a technique such as RANSAC (Random Sample Consensus) so that the sum of errors in the positions of the associated objects is minimized.

  The control start determination unit 52 determines whether or not the time-series change amount ΔE of the evaluation value E stored in the predetermined storage area has exceeded a predetermined evaluation value change amount threshold value ΔEth, and the reliability stored in the predetermined storage area. A time series change amount of the degree R and whether or not the change amount ΔR in the direction in which the reliability R decreases exceeds a predetermined reliability change amount threshold value ΔRth, and a detection value detected by the vehicle motion sensor 14 Whether or not the imaging control is to be started is determined based on whether or not the vehicle motion magnitude M indicated by is greater than a predetermined motion threshold Mth. As shown in FIG. 7, when an abrupt vehicle motion occurs, the evaluation value E also varies greatly. At the same time, the reliability R tends to decrease. Sudden vehicle movement is movement that causes a shake in a captured image. For example, assuming a movement of a vehicle when turning right or left, a movement threshold Mth is set as a threshold at which such a sudden vehicle movement can be detected. deep. Also, for the evaluation value change threshold ΔEth and the reliability change threshold ΔRth, appropriate values are determined based on the evaluation value and the reliability change when a similar sudden vehicle motion occurs.

  In addition, a sudden vehicle motion can be detected using any one of a change in the evaluation value E, a change in the reliability R, and the magnitude M of the vehicle motion, but a change in an object or a background image, etc. Therefore, there is a possibility that the evaluation value E changes greatly even if the magnitude M of the vehicle movement does not exceed the movement threshold value Mth, or the reliability R decreases even if the evaluation value E does not change. Therefore, here, these three values are used in an integrated manner to determine an appropriate control start timing.

  The evaluation value map update unit 54 starts imaging control by abrupt vehicle motion by expanding, reducing, deforming, rotating, or the like the shape of the evaluation value map based on the time-series evaluation value E stored in a predetermined storage area. The evaluation value map is updated corresponding to the event that is to be performed. Specifically, the evaluation value map is enlarged when the variation of the evaluation value E is large, or when the evaluation value E is biased, it is expanded (deformed) only in the biased direction. The imaging control value calculation unit 46 calculates the imaging control value using the evaluation value map updated by the evaluation value map update unit 54.

  Next, an object recognition processing routine executed by the computer 216 of the object recognition apparatus 210 according to the second embodiment will be described with reference to FIG. In addition, about the process same as 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  Through step 100 to step 110, an evaluation value E indicating the ease of recognition corresponding to the image characteristics is calculated.

  Next, in step 250, based on the recognition result in step 104, the recognition reliability R is high when the recognition result is correct and low when there is a possibility of erroneous recognition due to blurring of the image. calculate.

  Next, in step 252, the window image recognized as an image showing a specific object, the size of the window image, the position of the window image on the captured image, and the attributes of the recognized object (pedestrian, two-wheeled vehicle, vehicle And the like, the evaluation value E calculated in step 110, and the reliability R calculated in step 250 are associated with each other and controlled so as to be stored in a predetermined storage area in time series. .

  Next, in step 254, a detection value detected by the vehicle motion sensor 14 is acquired. Next, in step 256, a time-series change amount ΔE of the evaluation value E stored in a predetermined storage area is determined in advance. Whether the evaluation value change amount threshold value ΔEth has been exceeded or not, whether or not the change amount ΔR in the direction of decreasing reliability R stored in a predetermined storage area in a time series is a predetermined reliability Whether to start the imaging control depending on whether or not the change amount threshold value ΔRth has been exceeded and whether or not the magnitude M of the vehicle motion indicated by the detection value detected by the vehicle motion sensor 14 is greater than a predetermined motion threshold value Mth. Determine whether or not. When imaging control is started, the process proceeds to step 258, and when imaging control is not started, the process is terminated.

  In step 258, based on the time-series evaluation value E stored in a predetermined storage area, the shape of the evaluation value map is enlarged, reduced, deformed, rotated, etc., and imaging control is started by a rapid vehicle motion. The evaluation value map is updated in response to the event, and in step 260, the imaging control value is calculated using the updated evaluation value map.

  As described above, according to the object recognition apparatus of the second embodiment, a specific object is indicated on the evaluation value map indicating the relationship between the predetermined image characteristic and the evaluation value indicating the ease of recognition. An image characteristic of a window image recognized as an image is plotted to calculate an evaluation value, and the exposure time and gain at which the evaluation value is maximized while maintaining the brightness of the image are calculated. Noise can be reduced and a captured image suitable for recognition in order to recognize a specific object can be obtained. In addition, since it is determined whether or not imaging control is started by detecting a sudden vehicle motion, unnecessary processing when it is not necessary to perform imaging control can be suppressed.

  In the first and second embodiments, the case where the average of the frequency of the histogram in the edge direction and the variance of the frequency are used as the image characteristics has been described. However, the present invention is not limited to this. For example, the edge direction having the maximum frequency, the ratio between the maximum value and the minimum value of the frequency, and the like may be included, and the high-dimensional evaluation value map may be configured together with the frequency average and the frequency variance. .

  Also, as shown in FIG. 9, the sum of the power spectra of the entire partial image (in this embodiment, the window image) including the image showing the specific object and its peripheral image, and the sum of the power spectra of the high-frequency components of the partial image. The ratio may be used as an image characteristic. Since noise generated in the captured image is added to the captured image as a high-frequency component independent of the captured scene, the amount of noise can be estimated by frequency conversion such as Fourier transform. Therefore, the ratio between the sum of the power spectrum of the entire partial image and the sum of the power spectrum of the high frequency components of the partial image in the power spectrum after frequency conversion of the partial image is made to correspond to the evaluation value indicating the ease of recognition. Can be used as an image characteristic.

  Also, as shown in FIG. 10, the ratio of the minimum value to the maximum value of the histogram of the luminance value of the partial image may be used as the image characteristic. Since noise generated in the captured image is evenly added to the luminance value, the influence of the noise on the luminance value having the minimum frequency is larger than the influence of the noise on the luminance value having the maximum frequency. Therefore, since the amount of noise can be estimated by calculating the ratio of the minimum value of the luminance value of the partial image to the maximum value of the histogram, this ratio can be used as the image characteristic.

  Also, if the recognition method in the recognition unit uses an edge direction histogram, the image characteristic extraction unit may reuse the edge direction histogram obtained in the recognition unit to extract the image characteristic. Also good. Thereby, the amount of calculation can be reduced.

DESCRIPTION OF SYMBOLS 10,210 Object recognition apparatus 12 Imaging apparatus 14 Vehicle motion sensor 16, 216 Computer 18 Display apparatus 20 Specific object recognition part 22 Window image extraction part 24 Recognition part 26 Identification model memory | storage part 28 Display control part 40, 240 Imaging control part 42 Image Characteristic extraction unit 44 Evaluation value calculation unit 46 Imaging control value calculation unit 48 Reliability calculation unit 50 Time series storage control unit 52 Control start determination unit 54 Evaluation value map update unit

Claims (10)

Recognizing means for recognizing an image showing a specific object included in a picked-up image obtained by amplifying charges accumulated by the exposure time set by the image pickup means with a set gain;
Extraction means for extracting characteristics of a partial image including an image showing the specific object recognized by the recognition means and a peripheral image of the image showing the specific object;
Based on the relationship between the characteristics of the predetermined image and the ease of recognition by the recognition means, and the characteristics of the partial image extracted by the extraction means, the image showing the specific object recognized by the recognition means Evaluation value calculating means for calculating an evaluation value indicating ease of recognition;
Storage control means for controlling to store the evaluation values calculated by the evaluation value calculation means in time series in the storage means;
Based on the time-series evaluation values stored in the storage means by the storage control means, the evaluation value calculation means calculates when the evaluation value change amount is larger than a predetermined evaluation value change amount threshold value. An imaging control means for controlling the exposure time and the gain so that an evaluation value becomes high and a product of the exposure time and the gain becomes a predetermined value;
An object recognition apparatus including:   The object recognition apparatus according to claim 1, wherein the imaging control unit controls the exposure time and the gain when the evaluation value calculated by the evaluation value calculation unit is lower than a predetermined evaluation value threshold. Based on the recognition result of the recognition means, including a reliability calculation means for calculating the specific object likelihood of the image showing the specific object as the recognition reliability;
3. The object recognition apparatus according to claim 1, wherein the imaging control unit controls the exposure time and the gain when the reliability calculated by the reliability calculation unit is lower than a predetermined reliability threshold. . Before SL storage control means, wherein the reliability reliability calculated by the calculation means is controlled to be stored in the storage means in time series,
The imaging control unit is configured such that, based on the time-series reliability stored in the storage unit by the storage control unit, a change amount that has changed in a direction in which the reliability decreases is greater than a predetermined reliability change amount threshold value. The object recognition apparatus according to claim 3 , wherein the exposure time and the gain are controlled. Including detection means for detecting movement of the vehicle on which the imaging means is mounted;
Said imaging control unit, when the magnitude of the motion of the vehicle detected is greater than a predetermined movement threshold by said detecting means, any one of claims 1 to 4 for controlling the exposure time and the gain 1 Item recognition apparatus. Based on the time-series evaluation values stored in the storage means by the storage control means, the characteristics of the predetermined image used when calculating the evaluation value by the evaluation value calculation means and the recognition by the recognition means are easy. object recognition apparatus according to any one of claims 1 to 5 comprising updating means for updating the relationship Sato. The characteristic of the partial image is obtained by averaging the frequency of the histogram in the edge direction of the partial image and the variance of the frequency, the ratio of the sum of the power spectrum of the partial image and the sum of the power spectrum of the high frequency components of the partial image, or The object recognition apparatus according to any one of claims 1 to 6 , wherein a ratio between a maximum value and a minimum value of a histogram of luminance values of partial images is set. The recognition means recognizes an image showing a specific object by comparing a histogram of the edge direction of the captured image and a histogram of the edge direction of an image showing the specific object stored in advance,
The extraction means extracts an average frequency and a variance of the frequency of the edge direction histogram as characteristics of the partial image using the edge direction histogram of the partial image obtained by the recognition means. The object recognition device according to any one of claims 7 to 9. Computer
Recognizing means for recognizing an image indicating a specific object included in a picked-up image obtained by amplifying charges accumulated by an exposure time set by the image pickup means with a set gain;
Extraction means for extracting characteristics of a partial image including an image showing the specific object recognized by the recognition means and a peripheral image of the image showing the specific object;
Based on the relationship between the characteristics of the predetermined image and the ease of recognition by the recognition means, and the characteristics of the partial image extracted by the extraction means, the image showing the specific object recognized by the recognition means Evaluation value calculation means for calculating an evaluation value indicating ease of recognition ;
Storage control means for controlling the evaluation values calculated by the evaluation value calculation means to be stored in the storage means in time series; and
Based on the time-series evaluation values stored in the storage means by the storage control means, the evaluation value calculation means calculates when the evaluation value change amount is larger than a predetermined evaluation value change amount threshold value. An object recognition program for causing an evaluation value to increase and functioning as an imaging control unit that controls the exposure time and the gain so that a product of the exposure time and the gain becomes a predetermined value. The object recognition program for functioning a computer as each means which comprises the object recognition apparatus of any one of Claims 1-8 .

Images