JP6859910B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging device including an imaging unit that images an object and a control unit that controls the operation of the imaging unit.

The distance image sensor, which is an image pickup device, has the following problems. That is, the reflection characteristics of the object to be imaged by the distance image sensor are not uniform. Therefore, in the range image sensor, there is a trade-off that if the intensity of the output laser is increased, a leak occurs in the detection of a highly reflective object, and if the intensity of the laser is weakened, a leak occurs in the detection of a low reflection object.

In addition, in the range image sensor, if the intensity of the laser is increased to enable the detection of an object existing in the distance, the reflected light from the object existing near the object becomes saturated and cannot be detected, and the object existing near the object becomes undetectable. There is a trade-off that if the intensity of the laser is weakened to enable detection, the reflected light from a distant object becomes too small and cannot be detected.

As a conventional technique for solving such a trade-off, a High Dynamic Range function utilized in an imaging device such as a camera can be mentioned. In the following, High Dynamic Range will be abbreviated as HDR. The HDR function is a technique for expanding the dynamic range of an image by synthesizing a plurality of captured images having different exposure amounts, that is, different sensitivities to generate one image, and is disclosed in, for example, Patent Document 1. There are roughly two types of methods.

The first method is to generate a recorded image of one frame by synthesizing two frames of images alternately captured with a long exposure time and a short exposure time, and is called interplane HDR. However, in face-to-face HDR, the frame rate of the recorded image is halved of the frame rate in normal imaging.

The second method is to acquire a plurality of images having different exposure amounts by imaging one frame by reading out the images with different exposure times for each predetermined unit area, thereby expanding the dynamic range. Is called in-plane HDR. However, in in-plane HDR, two images are generated by imaging one frame. Therefore, for example, when reading is performed with different exposure times for each line, the vertical resolution of each image is usually set. It becomes 1/2 of the vertical resolution in the imaging of.

Japanese Unexamined Patent Publication No. 2014-146850

As mentioned above, the existing HDR function has the problem of reducing the frame rate or resolution. Therefore, if the existing HDR function is applied to the distance image sensor, the above-mentioned trade-off problem is solved, but another problem that the frame rate or the resolution is reduced arises. When the frame rate or resolution of the image data is reduced, there may be a problem that the accuracy of various processes executed using the image data is lowered or the processes cannot be performed normally.

In particular, since the distance image sensor is used for safety and security applications, it is necessary to be able to detect a person's motion and track the flow line at high speed and with high accuracy. Therefore, in a distance image sensor, a decrease in the frame rate is not preferable, and a decrease in resolution may affect the extraction process of feature points when recognizing a person, so an event that should be avoided as much as possible. Is considered to be.

The present invention has been made in view of the above circumstances, and an object of the present invention is to generate image data having an expanded dynamic range without reducing the frame rate, and to reduce the resolution using the image data. It is an object of the present invention to provide an image pickup apparatus capable of suppressing the influence on the image.

The imaging device according to claim 1 includes an imaging unit that images an object and a control unit that controls the operation of the imaging unit. Then, the control unit acquires a plurality of image data having different sensitivities by imaging one frame by making the sensitivity in imaging different for each predetermined unit region, and synthesizes the plurality of image data to obtain a dynamic range. Is provided with an HDR processing unit that executes in-plane HDR processing for generating enlarged image data.

According to the above configuration, by executing the in-plane HDR processing, it is possible to obtain image data having an expanded dynamic range without reducing the frame rate. However, as described above, the in-plane HDR processing has a problem that the resolution is reduced.

The decrease in the resolution of the image data becomes a problem in various processes performed using the image data, particularly in the object recognition process for determining what the imaged object (object) is. This is because, in the object recognition process, it is common to make a judgment using feature points. The feature amount of an object (for example, a person) obtained by general feature amount calculation such as HOG (Histograms of Oriented Gradients) has a large proportion of feature points on the outer periphery of the object. Therefore, it can be said that accurately grasping the outer shape, which is the shape of the outer circumference of the object, leads to improvement in the accuracy of object recognition. When the resolution of the image data is lowered, the feature points on the outer circumference of the object obtained from the image data are reduced, making it difficult to accurately grasp the outer shape of the object, and as a result, the accuracy of object recognition may be lowered. ..

On the other hand, in the above configuration, for example, when the in-plane HDR processing is performed so as to make the sensitivity different for each predetermined unit line, the vertical resolution of the generated image data is lowered. Further, in the above configuration, if the in-plane HDR processing is performed so as to make the sensitivity different for each predetermined unit sequence, for example, the horizontal resolution of the generated image data is lowered. That is, in the above configuration, the direction in which the resolution of the image data decreases when the in-plane HDR processing is executed depends on the HDR processing direction, which is the direction in which the sensitivity is different in the in-plane HDR processing.

In addition, the feature amount appearing in one direction (for example, the vertical direction) and the feature amount appearing in the other direction (for example, the horizontal direction) orthogonal to the one direction of the outer circumference of the object must not be exactly the same amount. There are many. For example, when the object is a person, the feature amount appearing in the alignment direction of the person's head, torso, and legs is larger than the feature amount appearing in the direction of the shoulder width of the person, which is the direction orthogonal to the alignment direction. Then, in the direction in which a large amount of features originally appears, even if the resolution is lowered, a relatively large amount of features can be obtained, so that no major problem occurs. However, in the direction in which the feature amount is originally small, if the resolution is lowered, the feature amount may not be obtained at all.

In this means, paying attention to such points, the following measures are added in order to prevent the feature points on the outer circumference of the object obtained from the image data from being extremely reduced due to the decrease in the resolution of the image data. ing. That is, the control unit further includes a direction control unit that controls the HDR processing direction. Then, the direction control unit controls so that the direction in which more features of the object appear is the HDR processing direction.

According to the above configuration, in the image data generated by executing the in-plane HDR processing, the resolution in the direction in which the feature amount of the object appears more is lowered, but the feature amount of the object is in the direction in which the feature amount is smaller. The resolution is maintained at the same level as the resolution in normal imaging. As a result, the feature points on the outer circumference of the object obtained from the image data decrease in the direction in which a large amount of features originally appears, but do not decrease in the direction in which the amount of features originally appearing is small.

Therefore, in the above configuration, the feature points on the outer circumference of the object obtained from the image data are not extremely reduced, and as a result, the accuracy of processing such as object recognition can be maintained satisfactorily. Therefore, according to the above configuration, it is possible to generate image data with an expanded dynamic range without reducing the frame rate, and it is possible to minimize the influence of the reduction in resolution on the processing performed using the image data. Excellent effect can be obtained.

In the imaging device according to claim 2, the imaging unit has a configuration of capturing a distance image of an object, and the image data is distance image data representing the distance to the object. That is, the imaging device according to claim 2 is configured as a distance image sensor. Even in an imaging device configured as a distance image sensor in this way, the effect of the reduction in resolution on processing such as object recognition processing performed using image data is small without reducing the frame rate by the above-mentioned ingenuity. It can be suppressed. Therefore, according to the above configuration, it is possible to detect a person's motion and track the flow line at high speed and with high accuracy, which is suitable for safety applications and security applications.

In the imaging device according to claim 3, the object is a person. In this case, the direction in which a person's feature amount appears more is the direction in which the person's head, torso, and legs are aligned, and conversely, in the direction of the person's shoulder width, the feature amount originally obtained is small. Therefore, if the HDR processing direction is controlled to match the direction of the person's shoulder width, the feature of the outer circumference in the person's shoulder width direction obtained from the image data generated by executing the in-plane HDR processing. The amount may be extremely reduced.

Therefore, in this case, the direction control unit controls so that the alignment direction of the human head, torso, and legs in the image data is the HDR processing direction. As a result, it is possible to reliably suppress an extreme decrease in the feature amount on the outer circumference of the person obtained from the captured image data, and as a result, it is possible to maintain good recognition accuracy of the person as the object.

If all the objects present in the image captured by the image pickup device have the same direction in which more features appear, the HDR processing direction should be uniformly controlled so that the features appear more in the same direction. For example, it is possible to suppress an extreme decrease in the feature amount of all the objects. However, when more features of each object appear in different directions, if the HDR processing direction is uniformly controlled, the extreme decrease of the features of a predetermined object can be suppressed, but the features of another object can be controlled. Extreme reductions may become uncontrollable.

Therefore, in the imaging device according to claim 4, the control unit further includes an area setting unit for setting a target area for executing the in-plane HDR processing. Then, in this case, the direction control unit controls the HDR processing direction for each target area set by the area setting unit. According to the above configuration, it is possible to set the HDR processing direction individually for each target area, so that even if the direction in which more features of each object appear is different, the optimum direction is set accordingly. By controlling the HDR processing direction for each target area, it is possible to suppress an extreme decrease in the feature amount of all the objects.

The direction in which more features of the object appear is not always constant, and may change as the object moves. Therefore, even if the HDR processing direction is controlled so that the feature amount of the object appears more at a certain point in time, the feature amount of the object becomes larger in the HDR processing direction due to the subsequent movement of the object. It can be in a different direction than it often appears.

Therefore, in the imaging device according to claim 5, the control unit further includes an external shape determining unit that determines the external shape of the imaged object based on the image data. In this case, the area setting unit detects the moving speed and moving direction of the object based on the image data, and sets a region in which a predetermined margin is added to the outer shape determined by the outer shape determining unit as the target area. , The target area is sequentially moved according to the detected moving speed and moving direction.

In this way, the target area for which the HDR processing is executed is set individually for each object existing in the image to be captured, and moves so as to follow the movement of the object. Therefore, even if the direction in which more features of the object appear changes due to the movement of the predetermined object, the direction in which more features of the object appear after the change is the HDR processing direction. Can be controlled as such.

Further, in this case, the target area is an area in which a predetermined margin is added to the outer shape of the object, and the area occupied by one target area can be suppressed to the minimum necessary. Therefore, even if there are many objects in the captured image, it is possible to set the target area for each of these objects. As a result, it is possible to suppress an extreme decrease in the feature amount of all the objects existing in the captured image, and to maintain good recognition accuracy of the object recognition process for all the objects.

As described above, when the object recognition process for the object in the target area is normally completed as a result of setting the target area and individually controlling the HDR processing direction, the object has a large amount of features. There is no problem even if it cannot be obtained. Therefore, in the imaging device according to claim 6, the area setting unit cancels the setting of the target area when a command for releasing the target area is given after setting the target area. Then, when the setting of the target area by the area setting unit is canceled, the direction control unit uniformly controls the HDR processing direction for all the areas of the image.

According to such a configuration, for example, when the object recognition process for the object in the target area is normally completed, a command for releasing the target area can be given to the target area setting unit. After that, since the HDR processing direction is uniformly controlled for all the regions of the image, the effect of reducing the processing load in the control unit can be obtained.

The figure which shows typically the structure of the image pickup apparatus which concerns on 1st Embodiment Diagram to explain the direction in which more features of the target person appear The figure which shows typically the installation mode in which the image pickup part was attached to the side wall. The figure which shows typically the image data when the in-plane HDR processing is executed so that the high sensitivity and the low sensitivity can be switched line by line. The figure which shows typically the installation mode which the image pickup part was attached to the ceiling. The figure which shows typically the image data when the in-plane HDR processing is executed so that the high sensitivity and the low sensitivity can be switched for each row. The figure which shows typically the structure of the image pickup apparatus which concerns on 2nd Embodiment The figure which shows typically the image data when three target areas are set. The figure which shows typically the structure of the image pickup apparatus which concerns on 3rd Embodiment The figure which shows typically the contents of the processing executed by the control part The figure which shows typically the contents of the area setting process executed by the area setting part. The figure which shows typically the image data when two target areas are set.

Hereinafter, a plurality of embodiments will be described with reference to the drawings. In each embodiment, substantially the same configuration is designated by the same reference numerals and the description thereof will be omitted.
(First Embodiment)
Hereinafter, the first embodiment will be described with reference to FIGS. 1 to 6.

The image pickup apparatus 1 shown in FIG. 1 is configured as a Time of Flight (TOF) type distance image sensor that measures a distance with a laser. In this case, the image pickup device 1 is used for safety applications, security applications, and the like, and a person is assumed as the object to be imaged.

The image pickup apparatus 1 includes an image pickup unit 2 that images an object and a control unit 3 that controls the operation of the image pickup unit 2. The imaging unit 2 is configured to capture a distance image of an object. That is, the image pickup unit 2 receives the laser emission unit 4 that emits the laser light, the emission control unit 5 that controls the emission operation of the laser light by the laser emission unit 4, and the reflected light of the emitted laser light to obtain a distance image. It includes a distance image sensor unit 6 that outputs a signal to be represented, and an A / D conversion unit 7 that A / D converts a signal output from the distance image sensor unit 6.

In this case, the distance image sensor unit 6 is mainly composed of a CMOS image sensor, and its light receiving sensitivity (hereinafter, simply referred to as sensitivity) can be changed on a pixel-by-pixel basis. In the present embodiment, the HDR processing direction, which is the direction in which the sensitivity is different in the in-plane HDR processing described later, is controlled by utilizing the above-mentioned function of the distance image sensor unit 6.

The control unit 3 controls the emission of the laser beam via the emission control unit 5, and performs distance calculation, image processing, and the like using the digital signal output from the A / D conversion unit 7, and is an image representing a distance image. Generate data. In this case, the image data is distance image data representing the distance to the object. In the present embodiment, the control unit 3 is separate from the image pickup unit 2, and is composed of, for example, a so-called personal computer.

The control unit 3 includes an HDR processing unit 8, a direction control unit 9, an object detection unit 10, and an object recognition unit 11. In the present embodiment, the HDR processing unit 8, the direction control unit 9, the object detection unit 10, and the object recognition unit 11 are realized by the CPU of the control unit 3 executing a program stored in a ROM or the like. In other words, it is realized by software. The HDR processing unit 8, the direction control unit 9, the object detection unit 10, and the object recognition unit 11 can also be realized by dedicated hardware.

The HDR processing unit 8 can execute the in-plane HDR processing. In the in-plane HDR processing, a plurality of image data having different sensitivities are acquired by imaging one frame by making the sensitivity in imaging different for each predetermined unit region, and the dynamic range is increased by synthesizing the plurality of image data. Enlarged image data is generated.

The unit area includes a plurality of pixel areas arranged in the horizontal direction (row direction), that is, a unit row such as one row, or a plurality of pixel areas arranged in the vertical direction (column direction), that is, a unit such as one column. There are columns. Further, in this case, the sensitivity in imaging can be switched in two stages by using the function of the distance image sensor unit 6 described above. In the following, the higher sensitivity of the two levels of sensitivity will be referred to as "high sensitivity", and the lower sensitivity will be referred to as "low sensitivity".

The direction control unit 9 controls the HDR processing direction, which is the direction in which the sensitivity is different in the in-plane HDR processing. Specifically, the direction control unit 9 controls so that the direction in which more features of the object appear is the HDR processing direction. As described above, in the image pickup apparatus 1, a person is assumed as an object. As shown in FIG. 2, the length of the alignment direction D1 of the head 12, the body 13 and the legs 14 of the outer circumference of the person is compared with the length of the shoulder width direction D2 of the person which is orthogonal to the alignment direction. Is long. Therefore, the feature amount appearing in the alignment direction D1 on the outer circumference of the person is larger than the feature amount appearing in the shoulder width direction D2 orthogonal to the alignment direction D1.

In consideration of such a point, the direction control unit 9 controls so that the arrangement direction D1 in which more human features appear is the HDR processing direction. Hereinafter, a specific example of controlling the HDR processing direction by the direction control unit 9 will be described with reference to FIGS. 3 to 6. In FIGS. 4 and 6, a plurality of pixel regions set to high sensitivity are schematically indicated by solid arrows, and a plurality of pixel regions set to low sensitivity are schematically indicated by dashed arrows. There is.

[1] First Specific Example of Control of HDR Processing Direction As shown in FIG. 3, in the first specific example, the image pickup unit 2 of the image pickup apparatus 1 is attached to the side wall 15. In this case, the imaging unit 2 images a predetermined region in the horizontal direction, specifically, in the right direction in FIG. In the image captured by the image pickup device 1 installed in this way, in most cases, the person H, which is the object, is projected so that the arrangement direction D1 is the vertical direction of the image (vertical direction in FIG. 4).

Therefore, in such a case, for example, when the image pickup apparatus 1 is installed, the direction control unit 9 uniformly switches the entire region of the image so that the vertical direction is the HDR processing direction. In this way, in the in-plane HDR processing executed by the HDR processing unit 8, as shown in FIG. 4, for example, high sensitivity and low sensitivity can be switched for each line. Although the vertical resolution of the image data generated by executing such in-plane HDR processing is lowered, the horizontal resolution is maintained at the same level as the resolution in normal imaging.

[2] Second Specific Example of Control of HDR Processing Direction As shown in FIG. 5, in the second specific example, the image pickup unit 2 of the image pickup apparatus 1 is attached to the ceiling 16. In this case, the imaging unit 2 images a predetermined region in the vertical direction, specifically, the downward direction in FIG. In the image captured by the image pickup device 1 installed in this way, the person H, which is the object, is projected so that the arrangement direction D1 is the vertical direction (vertical direction in FIG. 6) of the image, or the arrangement direction. It may appear so that D1 is in the horizontal direction of the image (horizontal direction in FIG. 6).

Therefore, in this case, the flow of people (direction of traffic of people) in the region to be imaged by the image pickup apparatus 1 may be observed in advance, and the HDR processing direction may be controlled based on the observation result. For example, if it is possible to obtain an observation result that most of the flow of people is in the left-right direction in FIG. 5, for example, when the image pickup apparatus 1 is installed, the direction control unit 9 controls the entire area of the image. Switching is performed uniformly so that the horizontal direction becomes the HDR processing direction.

In this way, in the in-plane HDR processing executed by the HDR processing unit 8, as shown in FIG. 6, for example, high sensitivity and low sensitivity can be switched for each row. Although the horizontal resolution of the image data generated by executing such in-plane HDR processing is lowered, the vertical resolution is maintained at the same level as the resolution in normal imaging.

The object detection unit 10 can execute an object detection process for determining whether or not an object exists in the captured image based on the generated image data. This object detection process is generally a process performed without using feature points. The object recognition unit 11 can execute an object recognition process for determining what an object exists in the captured image based on the generated image data. This object recognition process is generally a process performed using feature points. In the present embodiment, since the object is a person, the object recognition process is a person recognition process for determining whether or not the object existing in the captured image is a person.

As described above, the following effects can be obtained according to the present embodiment.
The control unit 3 of the image pickup apparatus 1 acquires a plurality of image data having different sensitivities by imaging one frame by making the sensitivity in imaging different for each predetermined unit region, and synthesizes the plurality of image data. The HDR processing unit 8 that executes the in-plane HDR processing that generates the image data whose dynamic range is expanded is provided. According to the above configuration, by executing the in-plane HDR processing, it is possible to obtain image data having an expanded dynamic range without reducing the frame rate.

However, as described above, the in-plane HDR processing has a problem that the resolution is reduced. The decrease in the resolution of the image data becomes a problem in various processes performed using the image data and the object recognition process executed by the object recognition unit 11. This is because, in the object recognition process, it is common to make a judgment using feature points. The feature amount of an object (for example, a person) obtained by general feature amount calculation such as HOG (Histograms of Oriented Gradients) has a large proportion of feature points on the outer periphery of the object. Therefore, it can be said that accurately grasping the outer shape, which is the shape of the outer circumference of the object, leads to improvement in the accuracy of object recognition. When the resolution of the image data is lowered, the feature points on the outer circumference of the object obtained from the image data are reduced, making it difficult to accurately grasp the outer shape of the object, and as a result, the accuracy of object recognition may be lowered. ..

In the configuration of the present embodiment, as shown in FIG. 4, when the in-plane HDR processing is performed so as to make the sensitivity different for each line, the vertical resolution of the generated image data is lowered. Further, in the configuration of the present embodiment, as shown in FIG. 6, when the in-plane HDR processing is performed so as to make the sensitivity different for each row, the horizontal resolution of the generated image data is lowered. That is, in the configuration of the present embodiment, the direction in which the resolution of the image data decreases when the in-plane HDR processing is executed depends on the HDR processing direction, which is the direction in which the sensitivity is different in the in-plane HDR processing.

In addition, the feature amount appearing in one direction (for example, the vertical direction) and the feature amount appearing in the other direction (for example, the horizontal direction) orthogonal to the one direction of the outer circumference of the object must not be exactly the same amount. There are many. When the object is a person assumed in the present embodiment, as shown in FIG. 2, the feature amount appearing in the arrangement direction D1 is larger than the feature amount appearing in the direction D2 orthogonal to the arrangement direction D1. Become. Then, in the direction in which a large amount of features originally appears, even if the resolution is lowered, a relatively large amount of features can be obtained, so that no major problem occurs. However, in the direction in which the feature amount is originally small, if the resolution is lowered, the feature amount may not be obtained at all.

In this embodiment, paying attention to such points, the following measures are added in order to prevent the feature points on the outer circumference of the object obtained from the image data from being extremely reduced due to the decrease in the resolution of the image data. Has been done. That is, the control unit 3 includes a direction control unit 9 that controls the HDR processing direction. Then, the direction control unit 9 controls so that the direction in which more features of the object appear is the HDR processing direction.

According to the above configuration, in the image data generated by executing the in-plane HDR processing, the resolution in the direction in which the feature amount of the object appears more is lowered, but the feature amount of the object is in the direction in which the feature amount is smaller. The resolution is maintained at the same level as the resolution in normal imaging. As a result, the feature points on the outer circumference of the object obtained from the image data decrease in the direction in which a large amount of features originally appears, but do not decrease in the direction in which the amount of features originally appearing is small.

Therefore, in the above configuration, the feature points on the outer circumference of the object obtained from the image data are not extremely reduced, and as a result, the recognition accuracy of the object recognition process executed by the object recognition unit 11 is maintained well. Can be done. Therefore, according to the present embodiment, it is possible to generate image data having an expanded dynamic range without reducing the frame rate, and to reduce the influence of the reduction in resolution on the processing performed using the image data. An excellent effect can be obtained.

In the imaging device 1, the imaging unit 2 has a configuration of capturing a distance image of an object, and the image data is distance image data representing the distance to the object. That is, the image pickup apparatus 1 of the present embodiment is configured as a distance image sensor. Even in the image pickup device 1 configured as the distance image sensor in this way, the effect of the reduction in resolution on the processing such as the object recognition processing performed using the image data without reducing the frame rate by the above-mentioned ingenuity It can be kept small. Therefore, the image pickup apparatus 1 of the present embodiment can detect a person's motion and track the flow line with high speed and high accuracy, and is suitable for safety applications and security applications.

In the image pickup apparatus 1, a person is assumed as an object to be imaged. As shown in FIG. 2, the direction in which more human features appear is the alignment direction D1 of the human head 12, the torso 13 and the legs 14, and conversely, the direction D2 of the human shoulder width is originally obtained. The amount of features to be used is small. Therefore, if the HDR processing direction is controlled to match the person's shoulder width direction D2, the outer circumference of the person's shoulder width direction D2 obtained from the image data generated by executing the in-plane HDR processing. There is a risk that the feature amount of will be extremely reduced.

Therefore, in the present embodiment, the direction control unit 9 controls so that the alignment direction D1 of the human head 12, the body 13 and the legs 14 in the image data is the HDR processing direction. As a result, it is possible to reliably suppress an extreme decrease in the feature amount on the outer circumference of the person obtained from the captured image data, and as a result, it is possible to maintain good recognition accuracy of the person as the object.

(Second Embodiment)
Hereinafter, the second embodiment will be described with reference to FIGS. 7 and 8.
As shown in FIG. 7, the control unit 22 included in the image pickup apparatus 21 of the present embodiment sets a target area to be executed for in-plane HDR processing in addition to the configuration provided by the control unit 3 of the first embodiment. The area setting unit 23 is provided. The area setting unit 23 can set a plurality of target areas in the entire area of the image. In this case, the direction control unit 9 controls the HDR processing direction for each target area set by the area setting unit 23. The area setting unit 23 is realized by software like the HDR processing unit 8.

As shown in FIG. 8, when the three target areas R1, R2 and R3 are set by the area setting unit 23, the direction control unit 9 can control the HDR processing direction for each of the target areas R1 to R3. .. In FIG. 8, as in FIGS. 4 and 6, a plurality of pixel regions set to high sensitivity are schematically indicated by solid arrows, and a plurality of pixel regions set to low sensitivity are indicated by dashed arrows. It is shown schematically.

In this case, the flow of people (direction of traffic of people) in the target areas R1 to R3 may be observed in advance, and each HDR processing direction may be controlled based on the observation result. For example, if it is possible to obtain an observation result that most of the flow of people in the target areas R1 and R3 is in the vertical direction in FIG. 8 and most of the flow of people in the target area R2 is in the horizontal direction in FIG. , The direction control unit 9 may control the HDR processing direction as follows. That is, as shown in FIG. 8, the direction control unit 9 controls so that the vertical direction is the HDR processing direction in the target areas R1 and R3, and is controlled so that the horizontal direction is the HDR processing direction in the target area R2. do it.

In this way, in the in-plane HDR processing executed by the HDR processing unit 8, as shown in FIG. 8, in the target areas R1 and R3, high sensitivity and low sensitivity are switched line by line, and the target area R2 Then, high sensitivity and low sensitivity can be switched for each row. In the image data generated by executing such in-plane HDR processing, the vertical resolution is lowered for the target areas R1 and R3, but the horizontal resolution is maintained at the same level as the resolution in normal imaging. Further, in the above image data, although the horizontal resolution of the target region R2 is lowered, the vertical resolution is maintained at the same level as the resolution in normal imaging.

When all the objects existing in the image captured by the image pickup apparatus 21 have the same direction in which the feature amount appears more, the HDR processing direction is uniformly controlled so as to be the direction in which the feature amount appears more. By doing so, it is possible to suppress an extreme decrease in the feature amount of all the objects. However, when more features of each object appear in different directions, if the HDR processing direction is uniformly controlled, the extreme decrease of the features of a predetermined object can be suppressed, but the features of another object can be controlled. Extreme reductions may become uncontrollable.

Therefore, in the image pickup apparatus 21 of the present embodiment, the control unit 22 includes an area setting unit 23 for setting a target area for executing the in-plane HDR processing. Then, in this case, the direction control unit 9 controls the HDR processing direction for each target area. According to such a configuration, it is possible to set the HDR processing direction individually for each target area, so that even if the direction in which more features of each object appear is different, the optimum direction is set accordingly. By controlling the HDR processing direction for each target area so as to be, it is possible to suppress an extreme decrease in the feature amount of all the objects.

(Third Embodiment)
Hereinafter, the third embodiment will be described with reference to FIGS. 9 to 12.
As shown in FIG. 9, the control unit 32 included in the image pickup apparatus 31 of the present embodiment determines the outer shape of the object imaged based on the image data with respect to the control unit 22 of the second embodiment. The difference is that 33 is added, and an area setting unit 34 is provided instead of the area setting unit 23. The external shape determination unit 33 and the area setting unit 34 are realized by software like the HDR processing unit 8.

The area setting unit 34 can execute the area setting process for setting the target area. In the area setting process, the moving speed of the object is detected and the moving direction is predicted based on the image data. Further, in the area setting process, an area in which a predetermined margin is added to the outer shape determined by the outer shape determining unit 33 is set as the target area, and the target area is set according to the detected moving speed and the predicted moving direction. Areas are moved sequentially.

Next, the operation on the above configuration will be described.
[1] Contents of processing executed by the control unit 32 The control unit 32 executes the processing as shown in FIG. First, in step S100, the HDR processing direction is set to the first direction for all areas of the image. The first direction in this case is the vertical direction.

In step S200, it is determined whether or not a moving object exists, that is, whether or not a moving object is detected, based on the generated image data. Here, if it is determined that the moving body has been detected, the result is "YES" in step S200, and the process proceeds to step S300. In step S300, the outer shape of the detected moving body (object) is determined based on the generated image data.

In step S400, an area setting process for setting the target area is executed. The contents of the area setting process will be described later. In step S500, it is determined whether or not the HDR processing direction coincides with the direction in which the feature amount of the target object (moving body) appears more for all of the set target areas. It should be noted that such a determination can be made based on the result of scanning the image data in the horizontal direction and the vertical direction.

Here, if it is determined that the HDR processing direction coincides with the direction in which the feature amount of the object appears more for all the target areas, the result is "YES" in step S500, and the process proceeds to step S600. On the other hand, if it is determined that the HDR processing direction does not match the direction in which the feature amount of the object appears more for at least one target area, the result is "NO" in step S500, and the process proceeds to step S600 through step S700. ..

In step S700, the HDR processing direction is the HDR processing direction for the target area in which the HDR processing direction does not match the direction in which the feature amount of the object appears more, that is, "the HDR processing direction ≠ the direction in which the feature amount of the object appears more". Is switched to the second direction. As a result, the HDR processing direction in all the target areas is set to the direction in which more features of the target object appear. In step S600, a person recognition process for determining whether or not the object existing in the captured image is a person is executed.

[2] Contents of the area setting process The specific contents of the area setting process are as shown in FIG. 11, for example. First, in step S410, a target area in which a certain margin is secured with respect to the outer shape of the object detected in step S300 is set. The target area set here is a square area, for example, the target areas R31 and R32 shown in FIG.

In step S420, the moving speed of the object is detected based on the image data. In step S430, the moving direction of the object is predicted based on the image data. The detection of the moving speed of the object and the prediction of the moving direction of the object are performed, for example, for the position of the object in the image data generated in a predetermined sampling cycle and the object in the image data generated before that, for example, one sampling cycle. It can be done based on the position of. In step S440, the target area is sequentially moved according to the moving speed of the object detected in step S420 and the moving direction of the object predicted in step S430.

The direction in which more features of the object appear is not always constant, and may change as the object moves. Therefore, even if the HDR processing direction is controlled so that the feature amount of the object appears more at a certain point in time, the feature amount of the object becomes larger in the HDR processing direction due to the subsequent movement of the object. It can be in a different direction than it often appears.

Therefore, the control unit 32 of the image pickup apparatus 31 of the present embodiment includes an outer shape determination unit 33 that determines the outer shape of the imaged object based on the image data. In this case, the area setting unit 34 detects the moving speed of the object and predicts the moving direction based on the image data. Then, the area setting unit 34 sets the area in which a predetermined margin is added to the outer shape determined by the outer shape determination unit 33 as the target area, and sets the target area according to the detected movement speed and the predicted movement direction. Move in sequence. That is, in this case, the position of the target area dynamically changes according to the position of the target object.

By doing so, as shown in FIG. 12, the target areas R31 and R32 to be executed for HDR processing are individually set for each person H31 and H32 which are objects existing in the image to be captured. .. Further, the set target areas R31 and R32 move so as to follow the movements of the people H31 and H32, respectively. As a result, even when the persons H31 and H32 move, the persons H31 and H32 are housed in the target areas R31 and R32, respectively. Therefore, even if the direction in which more features of the object appear changes due to the movement of the predetermined object, the direction in which more features of the object appear after the change is the HDR processing direction. Can be controlled as such.

Further, the target area set in this case is an area in which a predetermined margin is added to the outer shape of the person H31 and H32 which are the objects, as in the target areas R31 and R32 shown in FIG. The area occupied by one target area can be minimized. Therefore, even if there are many objects in the captured image, it is possible to set the target area for each of the objects by making the best use of the limited area. As a result, it is possible to suppress an extreme decrease in the feature amount of all the objects existing in the captured image, and to maintain good recognition accuracy of the object recognition process for all the objects.

It is also possible to dynamically change the margin of the target area. Specifically, based on the detected moving direction of the object, it is preferable to dynamically change the margin of the target area so that the margin on the moving direction side becomes large. In this way, even when the object moves at high speed, the moved object can be reliably contained in the target area.

Further, according to the configuration of the present embodiment, the following effects can be obtained. That is, the flow of people (direction of traffic of people) in the entire region imaged by the image pickup apparatus 31 is observed in advance, and the first direction in step S100 is determined based on the observation result. For example, if the observation result that most of the flow of people is in the vertical direction in FIG. 12 can be obtained, the first direction is set to the vertical direction, and the observation result that most of the flow of people is in the horizontal direction in FIG. 12 is obtained. If it can be obtained, the first direction may be the horizontal direction. In this way, most of the people imaged by the image pickup apparatus 31 are accurate in the person recognition process of step S700 without executing step S600 in the state where the HDR processing direction is set to the first direction. Can be recognized well.

However, in reality, there may be rare objects that move in a direction different from most objects. Even in such a case, according to the present embodiment, the HDR processing direction can be switched to a second direction different from the first direction for the target area in which a margin is secured for the outer shape of the object moving in the different directions. .. As a result, all the objects (people) imaged by the image pickup apparatus 31 including the objects moving in such a rare direction can be accurately recognized by the person recognition process in step S700.

In this case, after the recognition of the object moving in the rare direction is normally completed, the setting of the target area is canceled and all the HDR processing directions are switched back to the first direction, that is, The HDR processing direction may be uniformly controlled to be the first direction for all areas of the image. The setting of the target area can be released by giving a command from the object recognition unit 11 to the area setting unit 34 to release the target area.

In this way, it may not be possible to obtain a large amount of features for the object. However, since it is already recognized that the object is a human being, it is not necessary to obtain a large amount of its features. Therefore, as described above, after the object is recognized as a person, the setting of the target area is canceled and the HDR processing direction is uniformly controlled to be the first direction for all the areas of the image. However, there is no problem, and an effect that the increase in the processing load in the control unit 32 can be suppressed to the minimum necessary can be obtained.

(Other embodiments)
It should be noted that the present invention is not limited to each of the above-described embodiments and described in the drawings, and can be arbitrarily modified, combined, or extended without departing from the gist thereof.
The numerical values and the like shown in each of the above embodiments are examples, and are not limited thereto.
The present invention is not limited to the image pickup device 1 configured as a distance image sensor, and can be applied to all image pickup devices such as a camera that generates ordinary image data.

1, 21, 31 ... Imaging device, 2 ... Imaging unit, 3, 22, 32 ... Control unit, 8 ... HDR processing unit, 9 ... Direction control unit, 23, 34 ... Area setting unit, 33 ... External shape determination unit.

Claims (6)

An imaging device including an imaging unit that images an object and a control unit that controls the operation of the imaging unit.
The control unit
By making the sensitivity in the imaging different for each predetermined unit area, a plurality of image data having different sensitivities can be acquired by imaging one frame, and the dynamic range is expanded by synthesizing the plurality of image data. An HDR processing unit that executes in-plane HDR processing that generates data,
A direction control unit that controls the HDR processing direction, which is a direction in which the sensitivity is different in the in-plane HDR processing,
With
The direction control unit is an imaging device that controls so that the direction in which more features of the object appear is the HDR processing direction. The imaging unit has a configuration for capturing a distance image of the object.
The imaging device according to claim 1, wherein the image data is distance image data representing a distance to the object. The object is a person
The imaging device according to claim 1 or 2, wherein the direction control unit controls so that the alignment direction of the person's head, torso, and legs in the image data is the HDR processing direction. The control unit further includes an area setting unit for setting a target area for executing the in-plane HDR processing.
The imaging device according to any one of claims 1 to 3, wherein the direction control unit controls the HDR processing direction for each target area set by the area setting unit. The control unit further includes an outer shape determining unit that determines the outer shape of the object imaged based on the image data.
The area setting unit
The moving speed and moving direction of the object are detected based on the image data,
The imaging device according to claim 4, wherein a region to which a predetermined margin is added to the outer shape is set as the target region, and the target region is sequentially moved according to the moving speed and the moving direction. After setting the target area, the area setting unit releases the setting of the target area when a command for releasing the target area is given.
The imaging device according to claim 5, wherein the direction control unit uniformly controls the HDR processing direction for all areas of the image when the setting of the target area by the area setting unit is released.

Images