JP2018156408A - Image recognizing and capturing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image recognizing and capturing apparatus which can reduce a calculation amount in image recognition and can improve recognition rate.SOLUTION: An image recognizing and capturing apparatus has an image sensor 1 for capturing an image, and a 3D sensor 2 for acquiring a depth map indicating, as a distance to an image object, each of pixels in an area corresponding to a captured area of an image. The image recognizing and capturing apparatus has an object extracting means 3 for extracting a recognition object on the depth map from the depth map based on a distance. The image recognizing and capturing apparatus has an object image extracting means 4 for extracting, from the image, a partial image which is an object based on the area on the depth map of the objects extracted from the depth map, and an image recognizing means 5 for recognizing the partial image and identifying the extracted object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image recognition imaging apparatus that performs image recognition by acquiring a two-dimensional image and a distance image.

  In general, in a surveillance camera, an image is taken and displayed on a monitor, and this image is used by a person to monitor the image in real time, or used to store an image and confirm an incident after the occurrence of the incident. In recent years, with the development of AI (artificial intelligence) technology, it is possible to automatically detect the presence of a specific person or automatically detect the intrusion of a suspicious person into a restricted area by image recognition. is there.

  On the other hand, in an automobile, in order to automate the avoidance of a danger such as a collision, it has been proposed to use a distance image including a distance to a subject to be photographed in each pixel arranged vertically and horizontally (see, for example, Patent Document 1). ) Thereby, the position and movement of the obstacle can be detected. In this case, a stereo camera having two cameras is used for distance measurement and image acquisition. In recent years, 3D cameras that combine a 3D sensor (depth sensor) for acquiring a distance image and a 2D camera have begun to spread. In stereo cameras and 3D cameras, a 2D image of a visible light in a shooting target range and a distance It is possible to obtain both an image and a so-called 3D image of visible light.

Japanese Patent Laid-Open No. 2006-1464

  By the way, since recognition of a two-dimensional image using a 2D camera is performed on an image in a state where a person or an object in a three-dimensional space is projected two-dimensionally, AI technology (deep machine learning or the like) is used. However, the improvement of the processing speed and the recognition rate will end. That is, in object extraction from a two-dimensional visible image, different colors and different luminance parts are recognized as the same object, so the entire screen is processed and the object is recognized from changes in color and luminance. In addition, it requires a huge amount of calculation and requires a long time for processing, and the detection accuracy is about 90% in image recognition using deep machine learning.

  On the other hand, in the 3D camera, it is possible to relatively easily recognize whether there is an obstacle that the host vehicle collides with by knowing the distance. That is, it is easy to detect a person or an object at a collision position as an obstacle. That is, the 3D sensor can recognize the shape of the object, but the object may not be identified only by the shape read by the 3D sensor.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an image recognition imaging apparatus capable of reducing the amount of calculation in image recognition and improving the recognition rate.

In order to solve the above problems, an image recognition imaging apparatus of the present invention includes an imaging unit that captures an image,
Distance image acquisition means for acquiring a distance image in which each pixel in a range corresponding to the imaging range of the image is represented by a distance to a shooting target;
Distance image recognition object extraction means for extracting a recognition object on the distance image based on the distance from the distance image;
Recognition object image extraction means for extracting a partial image to be the recognition object from the image based on a range on the distance image of the recognition object extracted from the distance image;
And image recognition means for recognizing the partial image and identifying the recognition object.

  According to such a configuration, in a distance image, a group of pixels having a distance value close to each other and a single cluster can be recognized as an object that is a recognition target object on the distance image. In this case, for example, an object can be extracted by simple calculation as compared with the case of extracting an object from a two-dimensional color visible image.

  The distance image basically corresponds to the shooting range of the image, and the position of the distance image corresponding to each position of the image includes distance information. For example, when the image and the distance image have the same number of pixels, the pixel of the distance image corresponding to the pixel at the specific position of the image obtained by photographing a part of the object is up to a part of the object by the color. It represents distance information. Further, the image capturing range corresponds to the range of the range image. In other words, it is preferable that the image and the distance image are basically taken from the same range, but if the correspondence between the position of the image and the distance image is obtained, even if either image is larger than the other image, Good. In addition, it is preferable that it was what was imaged simultaneously. Therefore, the recognition target on the distance image can be assigned on the image, and the same range on the image as the range of the object extracted on the distance image can be set as the object on the image.

  An object is identified by recognizing the partial image that is the range of the object on the image. In this case, the amount of calculation can be greatly reduced because the image recognition is performed only for the extracted partial images, compared to the conventional image recognition performed on the entire image. In addition, since the object has already been separated from the background on the image, many operations for separating the object from the image are not required, and the amount of calculation can be reduced. In addition, since the separation of the object is performed based on the distance, the separation accuracy can be increased.

  In image recognition, a person, a car, or the like can be identified as an attribute of an object based on the shape, color, brightness, or the like of the extracted object. In addition to image recognition using deep machine learning, it can identify not only humans but also men, women, adults, children, etc., and facial features such as nose height, eyes and mouth. The size of eyes and the color of eyes can be classified by job. In addition, it is possible to identify the type and year of an automobile. Further, if data of feature points of a specific person is stored in advance, it is possible to identify whether or not the recognition object recognized as a person is a specific person. In addition, when the object is separated from the distance image, the three-dimensional shape and size of the object can be recognized from the relationship of the distance, and the image can be recognized in consideration of the three-dimensional shape and size of the object. Separation of an object according to a distance from an image and image recognition using data of a three-dimensional shape and size in addition to the image can improve object identification capability and identification accuracy including object detection and identification.

  The said structure of this invention WHEREIN: It is preferable that the said distance image acquisition means has a distance measurement means to measure the distance of each pixel of the said distance image.

  According to such a configuration, the above-described image and the distance image can be obtained from an imaging unit that is a monocular camera that captures an image and a distance measuring unit that generates a distance image such as a depth sensor and a 3D sensor. As the depth sensor and the 3D sensor, for example, a TOF (Time Of Flight) type can be used.

  In the configuration of the present invention, it is preferable that the distance image acquisition unit obtains a distance image based on the parallax between the two imaging units.

  According to such a configuration, a distance image can be obtained from the parallax of a so-called stereo camera.

  In the configuration of the present invention, it is preferable that an imaging unit, a distance image acquisition unit, a distance image recognition object extraction unit, a recognition object image extraction unit, and an image recognition unit are provided in one housing.

  According to such a configuration, processing required for image recognition is simplified, so that a large arithmetic processing device with a high processing speed is not required, and the image recognition imaging device is housed in a relatively small casing such as a monitoring camera. be able to. Note that the image recognition imaging apparatus may be connected to an external server so as to be able to perform data communication, and the server may be configured to store data or perform more advanced image recognition processing.

  According to the present invention, image recognition can be performed easily and with high accuracy.

1 is a block diagram illustrating an image recognition imaging apparatus according to a first embodiment of the present invention. 3 is a flowchart showing an image recognition method. It is a figure for demonstrating the image recognition method similarly. It is a figure for demonstrating the image recognition method similarly. It is a block diagram which shows the image recognition imaging device of the 2nd Embodiment of this invention.

Embodiments of the present invention will be described below.
The image recognition and imaging apparatus of the present embodiment is a combination of an image recognition apparatus and a camera mainly related to monitoring, such as a surveillance camera and an in-vehicle camera, and identifies a person, a car, and the like in a shooting range. ing.

The image recognition imaging apparatus of the present embodiment includes an image sensor 1 that is an imaging unit, a 3D sensor 2 that is a distance image acquisition unit, and an object (including a person) that is a recognition target from the distance image obtained by the 3D sensor 2. ) That is, the object extraction means 3 as a distance image recognition object extraction means for extracting a recognition object and the recognition object image extraction means for extracting a partial image as the recognition object from the image of the image sensor 1. Object image extraction means 4, image recognition means 5 for recognizing the extracted partial image (object image), control means 6 for controlling these, and memory for storing data such as images, distance images, recognition results, etc. Means 7.
The control means 6 is connected to an external server 9 (host PC) via a communication network 8 such as the Internet so that data communication is possible.

The image sensor 1 is a so-called image sensor (image sensor), and is used as a camera including a lens that forms an image to be photographed on the image sensor 1.
The 3D sensor 2 is of the above-described TOF method, for example, scans an ultrashort pulse of an infrared laser in the imaging range, measures the time until the reflected light of the light hitting the object returns, and this time Is multiplied by the speed of light to obtain the distance of each pixel in the shooting range. Note that the resolution of the image sensor 1 and the resolution of the 3D sensor 2 may or may not match, and the position of each part of the imaging range of the image sensor 1 and each of the imaging range of the 3D sensor 2 It is only necessary that the positions of the portions correspond to each other, and it is only necessary to know where an arbitrary position in the shooting range of the image sensor 1 is located in the shooting range of the 3D sensor 2. Basically, the image sensor 1 and the 3D sensor 2 are configured to simultaneously capture an overlapping range as an image and a distance image.

  The object extraction unit 3 extracts an object from a distance image that is distance information acquired by the 3D sensor 2. At this time, it is determined that a group of pixels (neighboring pixels) in which distance values are close to each other on the distance image and are substantially in a lump is an object to be extracted. At this time, a group of pixels with approximate distance values are extracted as one object. In this case, the object can be extracted basically only by the distance of each pixel in the distance image. Therefore, the distance image captured only once, not the image of a monitoring camera or the like that constantly or repeatedly captures the same range. For example, an object can be extracted with high accuracy even from a distance image captured by the in-vehicle 3D sensor 2. Note that a portion where the pixel distance value is equal to or greater than the predetermined distance may not be extracted as an object using the background. Further, in the 3D sensor 2 that is fixed like a surveillance camera or the 3D sensor 2 in which the movement range such as rotation is determined, the same range is always shot or the same range is shot repeatedly, so the 3D sensor 2 When shooting a pixel, the distance that does not change for a certain period of time (the longest distance in the case of change) is stored as the background distance of the pixel, and the group of pixels whose distance has changed from the background distance is recognized as an object It is good. In this case, a group of pixels whose distances change over time may be detected as an object. Note that it is possible to separate the background and the object from the change over time of the image even in a two-dimensional image, but in the case of a distance image, the part that has changed so that the distance is basically shorter than the background Can be identified as an object.

In the object extraction means 3, the position of the range of the object is determined on the distance image by the 3D sensor 2.
The object image extraction unit 4 converts the range of the above-described object determined on the distance image into a range on a visible two-dimensional image by the image sensor 1 and extracts a partial image within this range. That is, the range of the object extracted on the distance image is assigned to the image, and the partial image that becomes the range of the object is extracted from the image. For example, a coordinate system may be provided for each of the distance image and the image, the coordinates on the image may be converted to the coordinates on the distance image, or both coordinate systems may be the same.

  The image recognition means 5 performs image recognition of a partial image that is an object on the image extracted from the image as described above. In this case, since a portion that is an object has already been extracted from the distance image, it is recognized whether the extracted partial image is, for example, a person or a car. At this time, for example, the stored feature points of the person or the car and the feature points detected from the partial image are compared to determine whether the person is a car or the like. Further, based on an algorithm acquired by deep machine learning, a person may be identified as a dwarf, an adult, a woman, or a man. For actual image recognition, for example, open CV (Open Computer Vision Library), which is a library of functions related to image recognition, and recognition by deep machine learning are used to perform human detection, human tracking, and face recognition. be able to. The latest OpenCV library includes a machine learning function, for example, also includes a deep learning module, and can identify people, vehicles, and the like. In the present embodiment, since the area on the image that becomes the object has already been determined using the distance image, for example, it is not necessary to determine the area of the image that becomes the identified person or car, and has already been extracted. Therefore, it is only necessary to identify whether the area of the object is a person or a car, so that an operation for processing the entire image and specifying a portion to be an object is not required, and the amount of calculation is small. In image recognition, the attributes of an object are recognized, and the type of an object such as a person or a car, and if it is a person, the attribute of the object such as an adult, dwarf, male, female, race, facial features, etc. To detect. In the case of a car, the model, year, color, etc. are identified as attributes. In the present embodiment, the three-dimensional shape and the size can be recognized from the object data on the distance image when the object is extracted, and the three-dimensional shape and the size are used when identifying the attribute of the object. Thus, it is easy to determine adults and children, small cars and large cars.

The control means 6 controls photographing by the image sensors 1 and 3D sensor 2, object extraction from the distance image by the object extraction means 3, extraction of a partial image from the image by the object image extraction means 4, and image recognition by the image recognition means 5. To do. The control means 6 comprises an arithmetic processing device, and the arithmetic processing device may function as the object extraction means 3, the object image extraction means 4, and the image recognition means 5. For example, the image recognition means 5 may be realized by executing a machine learning model (image recognition algorithm) that has undergone deep machine learning on the arithmetic processing unit.
The storage unit 7 is a storage device including a hard disk, a flash memory, and the like, and stores data such as images, distance images, and image recognition results.

Next, an image recognition method by the image recognition imaging apparatus will be described with reference to the flowchart of FIG. 2 and FIGS. 3 and 4.
A distance image b is photographed by the 3D sensor 2 (step S1). At this time, a two-dimensional visible (color) image a is simultaneously captured by the image sensor 1. For example, as shown in FIG. 3, the photographed image a includes an adult man, a car, and a dwarf woman. Further, in the distance image b shown in FIG. 4, the difference in distance between the pixels can be expressed as an image, and an image can be obtained as shown in FIG. 4 by assigning light and darkness or color to the distance. . In FIG. 4, pixels having a predetermined distance or more are represented by white, for example.

  Next, an object is extracted from the distance image as a group of pixels that are close to each other (step S2). In this case, as described above, pixels having a predetermined distance or more may be used as the background, and a group of pixels having a smaller distance value and approximate distance may be extracted as objects. In addition, even if it is a lump of pixels, when it is separated into pixels of clearly different distances, they are handled as different objects. The object extracted on the distance image can represent the arrangement at the position of each pixel within the range of the object as a group of pixels. Here, distance image portions of an adult male, a car, and a dwarf female are extracted.

  Next, on the image a in FIG. 4 taken by the image sensor 1, a range that is the same position as the range of the object extracted in the distance image b in FIG. 4 is extracted as a partial image that becomes an object image (step S3). . As a result, an object on the two-dimensional color image is extracted. Here, the object is not separated by identifying and extracting the object range on the image, but the object is separated by the difference in distance on the distance image, and the object on the separated distance image is displayed on the image. Since the object is merely fitted to the object, the position of the object is only known, and the object is not identified. Therefore, since the object is extracted only by the distance on the distance image and the object on the image is extracted based on the position, the calculation amount is extremely large compared with the case of identifying the object and extracting the object. Few.

  Next, image recognition of a partial image as an object image extracted from the image is performed (step S4). In this case, since the object has been extracted as described above, in the image recognition, there is no need to process the entire image and extract the object. That is, since it is not necessary to detect and extract an object from the image, image recognition is performed only on the already extracted partial image, and the amount of calculation can be reduced. In addition, since image recognition can be performed on the assumption that the extracted partial image is one object or a plurality of adjacent objects, for example, the outer edge portion of the extracted portion is used as the outer edge portion of the object. Therefore, it is possible to reduce the amount of calculation in image recognition of partial images. As described above, a person can be identified by well-known person detection, wrinkle recognition, or the like. In addition, an object other than a person can also identify, for example, a car or a bicycle by registering feature points of various objects as part of the algorithm. In addition, the three-dimensional shape and size of an object that can be read from a distance image can be used as the feature point of the object, and the recognition accuracy of the object can be improved.

  Data such as the attribute of the object as the image recognition result is transmitted to the server 9 (step S5). Further, various data related to the above-described image recognition, for example, data such as a distance image used for image recognition, an image, a range of an extracted object, and the like are also sent to the server 9. In this case, more advanced image recognition may be performed on the server 9 side. In this case, the server 9 may include the object extraction unit 3, the object image extraction unit 4, and the image recognition unit 5. It is possible to perform more advanced processing by performing image recognition processing on the server side having high computing ability. Further, when image recognition processing is performed by the server 9, a system in which a plurality of image sensors 1 and 3D sensors 2 are connected may be used. In this case, the calculation amount of each image recognition and imaging apparatus can be reduced even in the server. For example, in a system including a large number of image sensors 1 and 3D sensors 2, a single high-function server is used. You can also. Further, the image recognition in the normal time may be performed on the image recognition imaging apparatus side, and the image recognition may be performed on the server when the image recognition is performed with the past data when the incident occurs. In this case, it is not necessary to save all the images and distance images. For example, only the extracted object image (partial image) is saved to reduce the storage capacity required for the server 9. Can do.

  According to such an image recognition imaging apparatus, after extracting an object from a distance image, the object image is extracted based on the position on the image corresponding to the position of the extracted object on the distance image, and then extracted. The amount of calculation can be reduced by performing image recognition on the object image, and the accuracy of separating the object from the background can be improved by performing object extraction on the distance image, for example, The object detection accuracy can be increased to about 99%. In addition, since the distance to the object is known, the size can be easily and accurately calculated as the attribute of the object. Based on this size, it becomes easy to determine other attributes of the object, for example, whether it is an adult or a dwarf, or the type of car.

Next, a second embodiment of the present invention will be described.
As shown in FIG. 5, in the image recognition and imaging apparatus according to the present embodiment, the distance image acquisition unit has two cameras (camera 1 and camera 2) 10 as imaging units, and left and right images captured by these cameras 10. Distance image detection means 11 for calculating a distance for each pixel from the parallax of the image and generating a distance image. The two cameras 10 and the distance image detecting means 11 constitute a stereo camera 12 with a 3D sensor function. The stereo camera 12 with the 3D sensor function can obtain the distance image generated by the distance image detection means 11 and the image photographed by the camera 10. Note that two images with parallax can be obtained by the stereo camera 12, but either one or both may be used.
In the second embodiment, the configuration other than the configuration of the distance image acquisition unit and the imaging unit is the same as that of the image recognition imaging device of the first embodiment described above, and the object extraction unit 3 and the object image extraction unit 4 The image recognition means 5, the control means 6, and the storage means 7 are provided, and are connected to an external server 9 via a communication network 8.

Also in the second embodiment, image recognition can be performed by the same method as in the first embodiment, except that a distance image is obtained by a known method from the parallax of a pair of images taken by the stereo camera 12. It is possible to achieve the same operational effects as those of the image recognition and imaging apparatus of the first embodiment.
The distance image acquisition means is not limited to the TOF type 3D sensor or the stereo camera, but may be another type of 3D sensor as long as it can generate a distance image corresponding to the shooting range of the imaging means. Good.

1 Image sensor (imaging means)
2 3D sensor (distance image acquisition means, distance measurement means)
3 Object extraction means (distance image recognition object extraction means)
4 Object image extraction means (recognition object image extraction means)
5 Image recognition means 10 Camera (imaging means)
11 Distance image detection means (distance image acquisition means)
12 Stereo camera (distance image acquisition means)

Claims (4)

An imaging means for capturing an image;
Distance image acquisition means for acquiring a distance image in which each pixel in a range corresponding to the imaging range of the image is represented by a distance to a shooting target;
Distance image recognition object extraction means for extracting a recognition object on the distance image based on the distance from the distance image;
Recognition object image extraction means for extracting a partial image to be the recognition object from the image based on a range on the distance image of the recognition object extracted from the distance image;
An image recognition imaging apparatus comprising: an image recognition unit that performs image recognition of the partial image and identifies the recognition object.   The image recognition imaging apparatus according to claim 1, wherein the distance image acquisition unit includes a distance measurement unit that measures a distance of each pixel of the distance image.   The distance image acquisition unit obtains a range image based on the parallax between the two imaging units.   The image pickup means, the distance image acquisition means, the distance image recognition object extraction means, the recognition object image extraction means, and the image recognition means are provided in one casing. The image recognition imaging device described.

Images