JP2019200769A - Learning device, method for learning, and program - Google Patents

Abstract

To provide a learning device which can identify an object by an image more accurately and also can perform identification processing more rapidly.SOLUTION: A learning device 12 has a memory and a processing circuit. The processing circuit: (a) acquires a first calculation taken image including an imaging target and surrounding environments of the imaging target from the memory, the first calculation taken image having a plurality of first pixels; (b) acquires a taken image including the imaging target and surrounding environments of the imaging target from the memory, the taken image having a plurality of second pixels; (c) acquires the result of identification of the imaging target and the surrounding environments of the imaging target in the taken image; (d) generates an identification model for identifying the first calculation taken image on the basis of the result of the identification of the taken image with reference to the correspondence relation between a plurality of first pixels and a plurality of second pixels; and (e) outputs the identification model to an image recognition device 10 for identifying a second calculation taken image.SELECTED DRAWING: Figure 2

Description

  The present disclosure relates to a learning device, a learning method, and a program.

  Technology for recognizing surrounding objects and recognizing the environment is important in autonomous driving vehicles and robots. 2. Description of the Related Art In recent years, for example, a technique called deep learning has attracted attention for object identification in autonomously driven vehicles and robots. Deep learning is machine learning using a multi-layered neural network, and a large amount of learning data is used in learning. By using such deep learning, it is possible to realize higher-precision identification performance as compared with the conventional method. In such object identification, image information is particularly effective. Non-Patent Document 1 discloses a technique for greatly improving conventional object identification capability by deep learning using image information as input. Further, in order to identify with high accuracy, the input image needs to have a high resolution. This is because, for example, a low-resolution image cannot be captured at a sufficient resolution with respect to a distant subject, and the identification performance deteriorates when the input image has a low resolution.

  On the other hand, Non-Patent Document 2 discloses a technique for further improving the deep learning identification ability by inputting depth information by a three-dimensional range finder in addition to image information. By using depth information, it is possible to separate the near and far subjects. Therefore, the identification performance can be improved even for a distant subject by using the depth information. In order to restore a high-resolution image while capturing a low-resolution image, for example, a technique called compression sensing as disclosed in Non-Patent Document 3 is known.

A. Krizhevsky, I. Sutskever and G. E. Hinton, `` ImageNet Classication with Deep Convolutional Neural Networks '', NIPS'12 Proceedings of the 25th International Conference on Neural Information Processing Systems, 2012, P1097-1105 Andreas Eitel et al., `` Multimodal Deep Learning for Robust RGB-D Object Recognition '', 2015 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), 2015 Y. Oike and A. E. Gamal, “A 256 × 256 CMOS Image Sensor with ΔΣ-Based Single-Shot Compressed Sensing”, 2012 IEEE International Solid-State Circuits Conference (ISSCC) Dig. Of Tech. Papers, 2012, P386-387 M. Salman Asif, Ali Ayremlou, Ashok Veeraraghavan, Richard Baraniuk and Aswin Sankaranarayanan, "FlatCam: Replacing Lenses with Masks and Computation", International Conference on Computer Vision Workshop (ICCVW), 2015, P.663-666 Yusuke Nakamura, Takeshi Shimano, Kazuyuki Tajima, Mayu Sao and Taku Hoshizawa, `` Lensless Light-field Imaging with Fresnel Zone Aperture '', 3rd International Workshop on Image Sensors and Imaging Systems (IWISS2016) ITE-IST2016-51, 2016, no .40, P.7-8

  However, the techniques disclosed in Non-Patent Documents 1 to 3 have a problem that it is difficult to achieve both improvement in identification accuracy of an object using an image and improvement in identification processing speed.

  Therefore, the present disclosure provides a learning device and the like that improve the identification accuracy of an object using an image and improve the identification processing speed.

  In order to solve the above-described problem, an aspect of the learning device of the present disclosure is a learning device including a memory and a processing circuit, and the processing circuit includes: (a) an imaging object and the imaging object from the memory A first calculated captured image including a surrounding environment of the first calculated captured image, the first calculated captured image includes a plurality of first pixels, and (b) the imaging object and the periphery of the imaging object from the memory A captured image including an environment is acquired, the captured image includes a plurality of second pixels, and (c) an identification result of the imaging object included in the captured image and a surrounding environment of the imaging object is acquired. , (D) identification for identifying the first calculated captured image based on the identification result of the captured image with reference to the correspondence relationship between the plurality of first pixels and the plurality of second pixels Generate a model and (e) identify the second computed captured image The image identification apparatus, and outputs the identification model.

  The comprehensive or specific aspect described above may be realized by a system, an apparatus, a method, an integrated circuit, a recording medium such as a computer program or a computer-readable recording disk, and the system, apparatus, method, and integrated circuit. The present invention may be realized by any combination of a computer program and a recording medium. The computer-readable recording medium includes a nonvolatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory).

  According to the learning device or the like of the present disclosure, it is possible to improve the identification accuracy of an object using an image and improve the identification processing speed.

  Additional benefits and advantages of one aspect of the present disclosure will become apparent from the specification and drawings. This benefit and / or advantage may be provided individually by the various aspects and features disclosed in this specification and the drawings, and not all are required to obtain one or more thereof.

FIG. 1 is a schematic diagram illustrating an example of a functional configuration of an identification system including an image identification device according to an embodiment. FIG. 2 is a schematic diagram illustrating an example of a functional configuration of an identification system according to a modification of the embodiment. FIG. 3 is a schematic diagram illustrating an example of a hardware configuration of an identification system according to a modification of the embodiment. FIG. 4 is a flowchart illustrating an example of a main processing flow of the learning device according to the modification of the embodiment. FIG. 5 is a diagram illustrating an example of a light field camera using a multi-pinhole. FIG. 6 is a schematic diagram illustrating an example of a normally captured subject image (captured image). FIG. 7 is a schematic diagram illustrating an example of a subject image (calculated captured image) captured using a light field camera including a multi-pinhole mask. FIG. 8A is a schematic diagram illustrating a captured image in which an identification area frame is superimposed and displayed. FIG. 8B is a schematic diagram showing only the identification area frame. FIG. 9 is a schematic diagram illustrating an example of an identification correct answer given as a mask on an image. FIG. 10 is a flowchart illustrating an example of an operation flow of the image identification device according to the embodiment. FIG. 11 is a schematic diagram illustrating an example of a functional configuration of the identification unit. FIG. 12 is a schematic diagram of an example of a coded aperture mask that uses a random mask as a coded stop. FIG. 13 is a schematic diagram illustrating another example of the functional configuration of the identification unit. FIG. 14A is a schematic diagram showing that the optical axis of the second image acquisition unit and the optical axis of the first image acquisition unit approximately match. FIG. 14B is a schematic diagram showing that each optical axis of the stereo camera constituting the second image acquisition unit and the optical axis of the first image acquisition unit approximately coincide with each other. FIG. 15 is a schematic diagram showing that a beam splitter is used to match the optical axis of the first image acquisition unit with the optical axis of the second image acquisition unit.

  As described in the “Background Technology” section, the use of machine learning such as deep learning has made it possible to implement a highly accurate identification technique using a mechanical device. Attempts have been made to apply such identification technology to automatic driving of a vehicle and movement of a robot. Since the vehicle and the robot are moving bodies, it is necessary to recognize surrounding objects from the captured image of the camera while moving. For this reason, a high identification processing speed is required.

  The technique disclosed in Non-Patent Document 1 requires a high-resolution image in order to obtain high identification accuracy. In order to acquire high-resolution image information, it is necessary to use an expensive camera, and there is a problem that the object identification system itself is expensive. In addition, acquiring a high-resolution image not only requires an expensive camera, but also increases the processing amount of the high-resolution image, which may cause a delay in processing.

  Non-Patent Document 2 discloses a technique regarding a highly accurate identification system that uses depth information. Such a system requires an expensive three-dimensional range finder in order to acquire depth information, and there is a problem that the cost increases. Furthermore, in this technique, since it is necessary to process a captured image and depth information in association with each other, the processing amount increases. This is because the depth information obtained by the three-dimensional range finder includes point group information including a large number of points obtained by scanning using a radar, for example, and thus has a large data size. That is, using depth information from such a three-dimensional range finder in addition to image information as an input causes a problem that the network size of the neural network increases and the identification processing speed decreases.

  In the technique disclosed in Non-Patent Document 3, the amount of processing for restoring a high-resolution image from a low-resolution image is enormous. The present inventors according to the present disclosure have found the above-described problems in the techniques of Non-Patent Documents 1 to 3, and studied a technique for improving the identification processing speed while improving the identification accuracy, as shown below. Invented technology.

  A learning device according to an aspect of the present disclosure is a learning device including a memory and a processing circuit. The processing circuit includes: (a) a first object including an imaging object and a surrounding environment of the imaging object from the memory; The first calculated captured image has a plurality of first pixels, and (b) acquires a captured image including the imaged object and the surrounding environment of the imaged object from the memory. The captured image has a plurality of second pixels, and (c) obtains an identification result of the imaging object included in the captured image and the surrounding environment of the imaging object, and (d) the plurality of the plurality of pixels. An identification model for identifying the first calculated captured image is generated on the basis of the identification result of the captured image with reference to the correspondence relationship between the first pixel and the plurality of second pixels, and (e ) In the image identification device for identifying the second calculated captured image, the identification And outputs a model.

  Since other information such as depth information can be added to the calculated captured image, the image itself can be simply used as an input to identify the object, and the data size of the three-dimensional range finder is large. There is no need to use point cloud information or the like as input. For this reason, it can suppress that the network size of a neural network becomes large, and can improve the identification processing speed. In addition, since the process of restoring the high resolution image from the low resolution image is not required, the identification processing speed can be improved. Moreover, since other information, such as depth information, can be used by the calculated captured image, the identification accuracy can be improved. As described above, it is possible to improve the identification accuracy of an object using an image and improve the identification processing speed.

  However, the calculated captured image is an image that cannot be visually recognized by a person in the same manner as the state of the real space. When machine learning is performed using the first calculated captured image as an input, the person actually executes the first calculated captured image. Since it cannot be visually recognized in the same manner as the state of the space, it is difficult to input the identification result for the first calculated captured image as an identification correct answer when performing machine learning. Therefore, even when machine learning is performed by using the first calculated captured image as an input, an identification result for a normal captured image that can be visually recognized by a person in the same manner as the state of the real space is input as an identification correct answer. Since the captured image is an image that can be visually recognized by a person in the same way as the state of the real space, it is possible to easily acquire the identification result such as the position of the imaging target included in the captured image and the surrounding environment of the imaging target Because. An identification model for identifying the first calculated captured image by using the first calculated captured image as input and performing machine learning based on the identification result of the captured image different from the first calculated captured image. In order to generate the image, it is known which position (pixel) in the captured image corresponds to the imaging object included in the first calculated captured image and the position (pixel) of the surrounding environment of the imaging object. There is a need. For this reason, in this aspect, the correspondence relationship between the positions of the first calculated captured image and the captured image (specifically, the plurality of first pixels included in the first calculated captured image and the plurality of captured images includes (Corresponding relationship with the second pixel).

  For example, the identification result may include a position of the imaging object and the surrounding environment of the imaging object in a plane.

  According to this, since the identification model is generated based on the imaging object and the position of the imaging object in the plane of the surrounding environment, the imaging object included in the second calculated captured image using the identification model. And the position in the plane of the surrounding environment of an imaging target object can be identified.

  For example, the identification result may include a position in the depth direction of the imaging object and the surrounding environment of the imaging object.

  According to this, since the identification model is generated based on the imaging object and the position in the depth direction of the surrounding environment of the imaging object, the imaging object included in the second calculated captured image using the identification model The position in the depth direction of the surrounding environment of the object and the imaging object can be identified.

  For example, the identification result may include category information to which the imaging object and the surrounding environment of the imaging object belong.

  According to this, since the identification model is generated based on the category information of the imaging object and the surrounding environment of the imaging object, using the identification model, the imaging object included in the second calculated captured image and The category information of the surrounding environment of the imaging object can be identified. For example, it can be identified whether the object to be imaged is a person, a car, a bicycle, a signal, or the like.

  For example, the first calculated captured image and the second calculated captured image may be images including parallax information in which a plurality of surroundings of the imaging target and the imaging target are superimposed. Specifically, the first calculated captured image and the second calculated captured image are obtained by using the multi-pinhole camera, the coded aperture camera, the light field camera, or the lensless camera, and the imaging object and the imaging object. It may be an image obtained by imaging the surrounding environment.

  According to this, depth information can be added to an image itself by superimposing a plurality of imaging objects and surrounding environments of the imaging objects.

  For example, the captured image may be an image obtained by imaging the imaging object and the surrounding environment of the imaging object with a multi-view stereo camera.

  By using the captured image obtained by the multi-view stereo camera, it is possible to estimate the imaging object included in the captured image and the position in the depth direction of the surrounding environment of the imaging object. Therefore, the position in the depth direction can be input as the identification correct answer as the identification result for the captured image.

  For example, the optical axis of the camera used for capturing the first calculated captured image and the optical axis of the camera used for capturing the captured image may be substantially the same. Specifically, the optical axis of the camera used for capturing the first calculated captured image and the optical axis of the camera used for capturing the captured image are determined by passing through a beam splitter, a prism, or a half mirror. You may do it.

  According to this, when converting the identification correct answer with respect to the captured image into the identification correct answer with respect to the first calculated captured image, each optical axis is substantially matched (or matched), thereby reducing errors caused by the conversion. , More accurate identification can be realized. Since the optical axis of the camera used for capturing the first calculated captured image and the optical axis of the camera used for capturing the captured image substantially coincide with each other, the first calculated captured image and the captured image are at substantially the same position. This is because an image obtained when imaging (environment) is obtained.

  A learning method according to an aspect of the present disclosure includes: (a) a first calculated captured image including a plurality of first pixels, the first calculated captured image including an imaging target object and a surrounding environment of the imaging target object. An image is acquired; (b) a captured image including the imaging object and a surrounding environment of the imaging object, the captured image having a plurality of second pixels; (c) included in the captured image And (d) referring to a correspondence relationship between the plurality of first pixels and the plurality of second pixels, and Based on the identification result, an identification model for identifying the first calculated captured image is generated, and (e) the identification model is output to an image identification device for identifying the second calculated captured image.

  According to this, it is possible to provide a learning method that improves the identification accuracy of an object using an image and improves the identification processing speed.

  A program according to an aspect of the present disclosure is a program for causing a computer to execute the learning method described above.

  According to this, the program which improves the identification accuracy of the object using an image and improves the identification processing speed can be provided.

  The comprehensive or specific aspect described above may be realized by a system, an apparatus, a method, an integrated circuit, a recording medium such as a computer program or a computer-readable recording disk, and the system, apparatus, method, and integrated circuit. The present invention may be realized by any combination of a computer program and a recording medium. The computer-readable recording medium includes a non-volatile recording medium such as a CD-ROM.

[Embodiment]
Hereinafter, embodiments will be described with reference to the drawings. It should be noted that each of the embodiments described below shows a comprehensive or specific example. Numerical values, shapes, components, arrangement positions and connection forms of components, steps (steps), order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements. Further, in the following description of the embodiment, an expression with “substantially” such as substantially coincidence may be used. For example, “substantially coincident” not only means that they are completely coincident, but also means that they are substantially coincident, that is, include a difference of, for example, several percent. The same applies to expressions involving other “abbreviations”. Each figure is a mimetic diagram and is not necessarily illustrated strictly. Furthermore, in each figure, the same code | symbol is attached | subjected to the substantially same component, and the overlapping description may be abbreviate | omitted or simplified.

  An image identification device according to an embodiment will be described.

  FIG. 1 is a schematic diagram illustrating an example of a functional configuration of an identification system 1 including an image identification device 10 according to an embodiment.

  The identification system 1 includes a camera that captures a captured image including a captured object and a surrounding environment of the captured object, and a processing circuit that identifies the captured object in the captured captured image using an identification model. The identification model and the calculated captured image will be described later. The identification system 1 includes an image identification device 10 having the processing circuit and an imaging unit 11 as the camera. The image identification device 10 includes an acquisition unit 101, an identification unit 102, and an output unit 103. The identification system 1 detects a subject included in the image using the image acquired by the imaging unit 11, and outputs a detection result. The detection of the subject in the image is also called “identification”.

  The identification system 1 may be mounted on a moving body such as a vehicle and a robot, or may be mounted on a fixed object such as a surveillance camera system. In the present embodiment, the identification system 1 will be described as being mounted on an automobile that is an example of a mobile object. In this case, both the imaging unit 11 and the image identification device 10 may be mounted on the moving body. Alternatively, the imaging unit 11 may be mounted on the moving body, and the image identification device 10 may be disposed outside the moving body. An example of a target on which the image identification device 10 is arranged is a computer device or a terminal device of an operator of a moving object. Examples of the terminal device are an operation terminal device dedicated to a mobile body, or a general-purpose portable terminal device such as a smartphone, a smart watch, and a tablet. Examples of the computer device are a car navigation system, an ECU (Engine Control Unit), a server device, or the like.

  When the image identification device 10 and the imaging unit 11 are arranged apart from each other, the image identification device 10 and the imaging unit 11 may communicate via wired communication or wireless communication. For the wired communication, for example, a wired LAN (Local Area Network) such as a network conforming to the Ethernet (registered trademark) standard and any other wired communication may be applied. For wireless communication, a mobile communication standard used in a mobile communication system such as the third generation mobile communication system (3G), the fourth generation mobile communication system (4G), or LTE (registered trademark), Wi-Fi ( Wireless LAN such as registered trademark (Wireless Fidelity), and short-range wireless communication such as Bluetooth (registered trademark) and ZigBee (registered trademark) may be applied.

  The imaging unit 11 captures, that is, acquires a computational imaging image including an imaging object and the surrounding environment of the imaging object. Specifically, the imaging unit 11 captures (acquires) an image including parallax information in which a plurality of imaging objects and surrounding environments of the imaging object are superimposed as a calculated captured image. The calculated captured image acquired by the imaging unit 11 is also referred to as a second calculated captured image. The second calculated captured image is an image used when identifying an object. Note that the calculated captured image is also called a calculated image. For example, the imaging unit 11 may acquire the second calculated captured image every first cycle that is a predetermined cycle, or may acquire the second calculated captured image continuously as a moving image. The imaging unit 11 may acquire a second calculated captured image associated with the time. An example of hardware of the imaging unit 11 is a camera, specifically, a multi-pinhole camera, a coded aperture camera, a light field camera, a lensless camera, or the like. In the case of such a camera, the imaging unit 11 can simultaneously acquire a plurality of images of the subject in one imaging operation, as will be described later. Note that the imaging unit 11 may acquire the plurality of images by a plurality of imaging operations, for example, by changing an imaging region of the imaging element included in the imaging unit 11, that is, a light receiving region. The imaging unit 11 outputs the acquired second calculated captured image to the acquisition unit 101 of the image identification device 10.

  Note that the imaging unit 11 acquires not only the second calculated captured image used when identifying the object but also the first calculated captured image used during learning described in FIG. The calculated captured image may be output to the first image acquisition unit 121 (see FIG. 2) of the learning device 12.

  Here, the calculated captured image and the normal captured image will be described. A normal captured image is an image captured through an optical system. A normal captured image is usually acquired by imaging light from an object collected by an optical system. An example of the optical system is a lens. By swapping the object and the image point in the image and placing the object at the image point, the image point at the position of the original object in the same optical system as before the object and the image point in the image are replaced The positional relationship between an object point and an image point that can be used is called a conjugate. In this specification, an image captured in such a conjugate state is referred to as a normal captured image (or captured image). When a person views the object directly in an environment where the object exists, the person perceives the object in a state almost the same as a normal captured image. In other words, a person visually recognizes a normal captured image captured by a normal digital camera in the same manner as a real space state.

  On the other hand, the calculated captured image is an image in which a plurality of images are shifted and superimposed by using, for example, a multi-pinhole, and is an image that cannot be visually recognized by a person in the same manner as the state of the real space. However, the calculated captured image may be an image that cannot be visually recognized by a person, but if computer processing is used, it is possible to acquire information included in the image of the imaging target and the surrounding environment. The computed captured image can be visualized so that a person can recognize it by restoring the image. Examples of computed captured images are light field images captured using multi-pinholes or microlenses, compressed sensing images captured by weighted addition of pixel information in space-time, or encoded with an encoded aperture It is an encoded image such as a coded aperture image (encoded aperture image) imaged using a mask. For example, Non-Patent Document 3 shows an example of a compressed sensing image. Another example of the calculated captured image is an image captured using a lensless camera that does not have an imaging optical system by refraction as shown in Non-Patent Document 4 and Non-Patent Document 5. Since any of the above calculated captured images is a known technique, a detailed description thereof will be omitted.

  For example, a light field image includes depth information in addition to an image value for each pixel. The light field image is an image acquired by the image sensor through a plurality of pinholes or microlenses arranged in front of the image sensor. The plurality of pinholes and microlenses are arranged in a plane along the light receiving surface of the image sensor, for example, arranged in a lattice shape. The imaging device simultaneously acquires a plurality of images through each of a plurality of pinholes or microlenses in one imaging operation as a whole. The plurality of images are images taken from different viewpoints. The distance in the depth direction of the subject can be acquired from the positional relationship between the plurality of images and the viewpoint. An example of the image sensor is an image sensor such as a complementary metal oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor.

  The compressed sensing image is a target image for compressed sensing. An example of a compression sensing target image is an image captured by a lensless camera. The lensless camera does not have an image forming optical system by refraction, and acquires an image via a mask arranged in front of the image sensor. The mask includes a plurality of regions having different transmittances, for example, in a lattice shape. By photographing through such a mask, light rays (light field images) from various directions can be coded and imaged by the mask. In compressed sensing, by using this mask information, it is possible to acquire an image of only the light beam in the desired direction or an omnifocal image focused at all distances from the coded light field image. Furthermore, depth information can be acquired.

  An image captured by setting such a mask as a diaphragm at the opening of the camera is called a coded aperture image (coded aperture image).

  As described above, the calculated captured image (the first calculated captured image and the second calculated captured image) is an image including parallax information in which a plurality of imaging objects and a plurality of surrounding environments of the imaging objects are superimposed. Specifically, it is an image obtained by imaging the imaging object and the surrounding environment of the imaging object by a multi-pinhole camera, a coded aperture camera, a light field camera, or a lensless camera.

  The acquisition unit 101 of the image identification device 10 acquires the second calculated captured image from the imaging unit 11 and outputs it to the identification unit 102. Further, the acquiring unit 101 may acquire a discriminator used by the discriminating unit 102 for discrimination, or may output the acquired discriminator to the discriminating unit 102. When the image identification device 10 is mounted on a moving body, the acquisition unit 101 may acquire the speed of the moving body from the moving body. The acquisition part 101 may acquire the speed of a moving body in real time, and may acquire it regularly. For example, when the moving body includes a speedometer, the acquiring unit 101 may acquire the speed from the speedometer, or the computer provided in the moving body may obtain the speed from a computer that receives speed information from the speedometer. You may get it. Further, for example, when the moving body does not include a speedometer, the acquiring unit 101 acquires speed-related information from an inertial measurement device such as a GPS (Global Positioning System) device, an accelerometer, and an angular velocity meter provided in the moving body. May be.

  The identification unit 102 acquires the second calculated captured image from the acquisition unit 101. The identification unit 102 includes, for example, a classifier acquired from the acquisition unit 101. The discriminator is an identification model for acquiring object information from an image, and is data used by the discriminator 102 for discrimination. The classifier is constructed using machine learning. It is possible to construct a discriminator with improved discrimination performance by machine learning using the calculated captured image as learning data. The calculated captured image used for machine learning as the learning data is also referred to as a first calculated captured image. In the present embodiment, the machine learning model applied to the discriminator is a machine learning model using a neural network such as deep learning, but may be another learning model. For example, the machine learning model may be a machine learning model using Random Forest, Genetic Programming, or the like.

  The identification unit 102 acquires information on the object (the imaging target object and the surrounding environment of the imaging target object) in the second calculated captured image using the classifier. Specifically, the identification unit 102 identifies an object included in the second calculated captured image, and acquires the position of the object in the second calculated captured image. That is, the object information includes the presence / absence of the object and the position of the object. The position of the object may include a planar position on the image and a position in the depth direction of the image. For example, the identification unit 102 identifies whether or not an object exists for each at least one pixel of the second calculated captured image using a classifier. The identification unit 102 acquires the position of at least one pixel that is identified as the presence of the object as the position of the object in the second calculated captured image. Here, the identification of the object in this specification includes detecting a pixel in which the object exists in the second calculated captured image.

  For example, when the identification system 1 is mounted on a car, examples of the object are a person, a car, a bicycle, or a signal. Note that the identifying unit 102 may identify a predetermined type of object or a plurality of types of objects using the second calculated captured image. The identification unit 102 may identify an object in units of categories such as a moving object including a person, a car, or a bicycle. At this time, a discriminator corresponding to the type (category) of the object to be identified may be used. The discriminator is recorded in, for example, a memory (for example, a first memory 203 described later) included in the image discrimination device 10.

  For example, the light field image includes depth information of the subject of each pixel in addition to the image value. Further, as described in Non-Patent Document 2, the use of subject depth information as learning data is effective in improving the discrimination capability of the discriminator. For example, it is possible to recognize that an object that is small in the image is a subject that exists in the distance, and it is possible to prevent the object from being recognized as dust (that is, ignored). For this reason, a discriminator constructed by machine learning using a light field image can improve its discrimination performance. Similarly, machine learning using a compressed sensing image and a coded aperture image is also effective in improving the identification performance of the classifier.

  Further, the identification system 1 may include a learning device 12 for generating a classifier as shown in FIG. 2 described later. In this case, the identification unit 102 of the image identification device 10 uses the classifier generated by the learning device 12, in other words, the learning is completed.

  The output unit 103 outputs the identification result of the identification unit 102. When the identification system 1 further includes a display, the output unit 103 outputs an instruction to output the identification result to the display. Alternatively, the output unit 103 may include a communication unit, and output the identification result via a communication unit in a wired or wireless manner. As described above, the object information includes the presence / absence of the object and the position of the object, and automatic operation or the like is performed according to the identification result of the object information. By outputting the information, the user can recognize the situation around the moving body on which the identification system 1 is mounted.

  The components of the image identification apparatus 10 including the acquisition unit 101, the identification unit 102, and the output unit 103 as described above are a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), and a RAM (Random Access Memory). ) And a processing circuit including a memory such as a ROM (Read-Only Memory). Some or all of the functions of the above components may be achieved by the CPU or DSP executing a program recorded in the ROM using the RAM as a working memory. Further, some or all of the functions of the above components may be achieved by a dedicated hardware circuit such as an electronic circuit or an integrated circuit. A part or all of the functions of the constituent elements may be configured by a combination of the software function and a hardware circuit.

  Next, as a case where the identification system includes a learning device, a modified example of the identification system 1 according to the embodiment will be described with reference to FIG.

  FIG. 2 is a schematic diagram illustrating an example of a functional configuration of an identification system 1A according to a modification of the embodiment.

  As illustrated in FIG. 2, the identification system 1 </ b> A according to the modification includes an image identification device 10, an imaging unit 11, and a learning device 12. The learning device 12 includes a first image acquisition unit 121, a second image acquisition unit 122, an identification correct answer acquisition unit 123, and a learning unit 124. The image identification device 10, the imaging unit 11, and the learning device 12 may be mounted on one device, or may be mounted on a plurality of devices. When the image identification device 10, the imaging unit 11, and the learning device 12 are separately installed in a plurality of devices, information may be exchanged between the devices via wired communication or wireless communication. The applied wired communication and wireless communication may be any of those exemplified above.

  FIG. 3 is a schematic diagram illustrating an example of a hardware configuration of an identification system 1A according to a modification of the embodiment.

  As illustrated in FIG. 3, the learning device 12 includes a second input circuit 221, a third input circuit 222, a second arithmetic circuit 223, and a second memory 224. Further, the image identification device 10 includes a first input circuit 201, a first arithmetic circuit 202, a first memory 203, and an output circuit 204.

  The first input circuit 201, the first arithmetic circuit 202, and the output circuit 204 are an example of a processing circuit included in the image identification device 10, and the first memory 203 is an example of a memory included in the image identification device 10. Referring to FIGS. 1 and 2, the first input circuit 201 corresponds to the acquisition unit 101. The first arithmetic circuit 202 corresponds to the identification unit 102. The output circuit 204 corresponds to the output unit 103. Thus, since the acquisition unit 101, the identification unit 102, and the output unit 103 correspond to the first input circuit 201, the first arithmetic circuit 202, and the output circuit 204, the first acquisition unit 101, the identification unit 102, and the output The unit 103 is also an example of a processing circuit included in the image identification device 10. The first memory 203 includes a computer program for the first input circuit 201, the first arithmetic circuit 202, and the output circuit 204 to execute processing, a second calculated captured image acquired by the acquisition unit 101, and an identification unit 102. The discriminator to be used is stored. The first memory 203 may be composed of a single memory, or may be composed of a plurality of memories of the same type or different types. The first input circuit 201 and the output circuit 204 may include a communication circuit.

  The second input circuit 221, the third input circuit 222, and the second arithmetic circuit 223 are examples of processing circuits included in the learning device 12, and the second memory 224 is an example of memory included in the learning device 12. Referring to FIGS. 2 and 3, the second input circuit 221 corresponds to the first image acquisition unit 121. The second input circuit 221 may include a communication circuit. The third input circuit 222 corresponds to the second image acquisition unit 122. The third input circuit 222 may include a communication circuit. The second arithmetic circuit 223 corresponds to the identification correct answer acquisition unit 123 and the learning unit 124. The second arithmetic circuit 223 may include a communication circuit. Thus, the first image acquisition unit 121, the second image acquisition unit 122, the identification correct answer acquisition unit 123, and the learning unit 124 correspond to the second input circuit 221, the third input circuit 222, and the second arithmetic circuit 223. Therefore, the first image acquisition unit 121, the second image acquisition unit 122, the identification correct answer acquisition unit 123, and the learning unit 124 are also examples of processing circuits included in the learning device 12. The second memory 224 includes a computer program for executing processing by the second input circuit 221, the third input circuit 222, and the second arithmetic circuit 223, the first calculated captured image acquired by the first image acquisition unit 121, the first The captured image acquired by the two image acquisition unit 122, the identification correct answer acquired by the identification correct acquisition unit 123, the classifier generated by the learning unit 124, and the like are stored. The second memory 224 may be composed of a single memory, or may be composed of a plurality of memories of the same type or different types.

  The first input circuit 201, the first arithmetic circuit 202, the output circuit 204, the second input circuit 221, the third input circuit 222, and the second arithmetic circuit 223 may be configured by a processing circuit including a processor such as a CPU or a DSP. The first memory 203 and the second memory 224 are realized by a storage device such as a semiconductor memory such as a ROM, a RAM, and a flash memory, a hard disk drive, or an SSD (Solid State Drive), for example. The first memory 203 and the second memory 224 may be combined into one memory. The processor executes a group of instructions described in the computer program expanded in the memory. Thereby, the processor can realize various functions.

  The first image acquisition unit 121 and the second image acquisition unit 122 of the learning device 12 acquire a first calculated captured image and a captured image for machine learning. An example of hardware of the first image acquisition unit 121 is a camera for capturing a calculated captured image, and specifically, a multi-pinhole camera, a coded aperture camera, a light field camera, a lensless camera, or the like. That is, the first image acquisition unit 121 is realized by, for example, the second input circuit 221 and a camera for capturing a calculated captured image. An example of hardware of the second image acquisition unit 122 is a camera for capturing a captured image, specifically, a digital camera or the like. That is, the second image acquisition unit 122 is realized by, for example, the third input circuit 222 and a camera for capturing a captured image.

  For example, a first calculated captured image captured by a camera for capturing a calculated captured image is stored in the second memory 224, and the second input circuit 221 acquires the first calculated captured image from the second memory 224. Thus, the first image acquisition unit 121 acquires the first calculated captured image. Note that the first image acquisition unit 121 may not include a camera for capturing a calculated captured image as hardware. In this case, the first image acquisition unit 121 (second input circuit 221) may acquire the first calculated captured image from the imaging unit 11 (specifically, the first image captured by the imaging unit 11). The calculated captured image is stored in the second memory 224, and the first calculated captured image may be acquired from the second memory 224), and the first calculation is performed from outside the identification system 1A via wired communication or wireless communication. A captured image may be acquired. The applied wired communication and wireless communication may be any of those exemplified above.

  Further, for example, a captured image captured by a camera for capturing a captured image is stored in the second memory 224, and the third input circuit 222 acquires the captured image from the second memory 224, thereby acquiring the second image. The unit 122 acquires a captured image. Note that the second image acquisition unit 122 may not include a camera for capturing a captured image as hardware. In this case, the second image acquisition unit 122 (third input circuit 222) may acquire a captured image from outside the identification system 1A via wired communication or wireless communication. The applied wired communication and wireless communication may be any of those exemplified above.

  The identification correct answer acquisition unit 123 acquires an identification correct answer for machine learning using the first calculated captured image acquired by the first image acquisition unit 121. The identification correct answer may be given together with the first calculated captured image from the outside of the identification system 1A, or may be given by the user manually inputting the identification correct answer. The identification correct answer includes category information to which the subject included in the first calculated captured image belongs, and position information of the subject. Examples of subject categories are people, cars, bicycles or signals. The position information includes a position on the image (specifically, a position on a plane or a position in the depth direction). The identification correct answer acquisition unit 123 stores the acquired identification correct answer in the second memory 224 in association with the first calculated captured image.

  However, as described above, the calculated captured image acquired by the acquisition unit 101 and the first image acquisition unit 121 is an image that cannot be visually recognized by a person in the same manner as the state of the real space. Therefore, it is difficult to input an identification correct answer to the first calculated captured image acquired by the first image acquisition unit 121. Therefore, the identification system 1A of the present embodiment includes the second image acquisition unit 122, which is acquired by the second image acquisition unit 122 instead of the first calculated captured image acquired by the first image acquisition unit 121. An identification correct answer is input to a captured image that can be visually recognized by a person in the same manner as in the real space. Details will be described later.

  The learning unit 124 uses the first calculated captured image acquired by the first image acquisition unit 121 and the identification correct answer for the captured image acquired by the second image acquisition unit 122 acquired by the identification correct answer acquisition unit 123. Learning of the classifier of the classifier 102 is performed. The learning unit 124 causes the discriminator stored in the second memory 224 to perform machine learning, and stores the latest discriminator after learning in the second memory 224. The identification unit 102 acquires the latest classifier stored in the second memory 224 and stores it in the first memory 203 while using it for the identification process. The machine learning is realized by, for example, an error back propagation (BP) in deep learning or the like. Specifically, the learning unit 124 inputs the first calculated captured image to the classifier, and acquires the identification result output by the classifier. Then, the learning unit 124 adjusts the discriminator so that the discrimination result becomes the discrimination correct answer. The learning unit 124 improves the discrimination accuracy of the discriminator by repeating such adjustment for a plurality of (for example, several thousand) first calculated captured images and discrimination correct answers corresponding thereto.

  Next, the operation of the learning device 12 will be described with reference to FIGS.

  FIG. 4 is a flowchart illustrating an example of the main processing flow of the learning device 12.

  First, in step S <b> 1, the learning unit 124 associates the first calculated captured image acquired by the first image acquisition unit 121 with the position (pixel) on the image of the captured image acquired by the second image acquisition unit 122. Get relationship. Specifically, the learning unit 124 acquires a correspondence relationship between the plurality of first pixels included in the first calculated captured image and the plurality of second pixels included in the captured image. This is realized by performing geometric calibration on the first calculated captured image and the captured image. Geometric calibration acquires in advance in the first calculated captured image and the captured image where a point with a known three-dimensional position is captured, and based on that information, the three-dimensional position of the subject and the first The calculated captured image and the relationship with the captured image are obtained. This can be realized, for example, by using a technique known as Tsai calibration. Normally, the three-dimensional position of the subject cannot be obtained from the captured image, but as described above, the three-dimensional position can be obtained from one image in the light field image that is the calculated captured image. Further, calibration is realized by acquiring corresponding points (pixels) on the image of the first calculated captured image acquired by the first image acquisition unit 121 and the captured image acquired by the second image acquisition unit 122. can do. For example, by acquiring the correspondence between the first calculated captured image and the captured image, the origin of the first calculated captured image and the captured image can be adjusted. If the positional relationship between the camera that captures the first calculated captured image and the camera that captures the captured image does not change, such calibration only needs to be performed once. In the following description, it is assumed that the calculated captured image is a light field image.

  The light field image has both information of pixel values and depth information. The light field image is acquired by a light field camera. A specific example of the light field camera is a camera using a multi-pinhole or a microlens. The imaging unit 11 may be a light field camera, and the first image acquisition unit 121 may acquire a light field image captured by the imaging unit 11. Alternatively, the first image acquisition unit 121 may acquire a light field image from outside the identification system 1A via wired communication or wireless communication.

  FIG. 5 is a diagram illustrating an example of a light field camera using a multi-pinhole.

  A light field camera 211 shown in FIG. 5 includes a multi-pinhole mask 211a and an image sensor 211b. The multi-pinhole mask 211a is arranged at a certain distance from the image sensor 211b. The multi-pinhole mask 211a has a plurality of pinholes 211aa arranged at random or at equal intervals. The plurality of pinholes 211aa is also referred to as a multipinhole. The image sensor 211b acquires an image of a subject through each pinhole 211aa. An image acquired through a pinhole is called a pinhole image. Since the pinhole image of the subject differs depending on the position and size of each pinhole 211aa, the image sensor 211b acquires a superimposed image of a plurality of pinhole images. The position of the pinhole 211aa affects the position of the subject projected on the image sensor 211b, and the size of the pinhole 211aa affects the blur of the pinhole image. By using the multi-pinhole mask 211a, it is possible to superimpose and acquire a plurality of pinhole images having different positions and degrees of blur. When the subject is away from the pinhole 211aa, the plurality of pinhole images are projected at substantially the same position. On the other hand, when the subject is close to the pinhole 211aa, a plurality of pinhole images are projected at distant positions. As described above, since the shift amount of the plurality of superimposed pinhole images corresponds to the distance between the subject and the multi-pinhole mask 211a, the superimposed image includes the depth information of the subject corresponding to the shift amount. include.

  For example, FIG. 6 and FIG. 7 each show an example of a normal captured image and an example of a light field image (calculated captured image) by a light field camera using a multi-pinhole.

  FIG. 6 is a schematic diagram showing an example of a subject image (captured image) that is normally captured, and FIG. 7 is an image of the subject (computed imaging) captured using a light field camera including a multi-pinhole mask. It is a schematic diagram which shows the example of an image.

  As shown in FIG. 6, in a normal captured image, a person A on a road and cars B and C are projected as subjects. When these subjects are imaged by, for example, a light field camera having four pinholes, the images of the person A, the cars B, and C are acquired as a plurality of superimposed images as shown in FIG. Specifically, the image of the person A is acquired as the persons A1, A2, and A3, the image of the car B is acquired as the cars B1, B2, B3, and B4, and the image of the car C is acquired by the cars C1, C2, Obtained as C3 and C4. 6 and FIG. 7, the road images on which the automobiles B and C travel are also acquired as a plurality of superimposed images as shown in FIG. 7. As described above, the calculated captured image is an image including parallax information in which a plurality of imaging objects (for example, person A, automobiles B, and C) and surrounding environments (for example, roads) of the imaging object are respectively superimposed.

  As shown in FIG. 4, in step S <b> 2, the first image acquisition unit 121 acquires a first calculated captured image including the imaging target and the surrounding environment of the imaging target from the second memory 224, and in step S <b> 3, The two-image acquisition unit 122 acquires a captured image including the imaging object and the surrounding environment from the second memory 224. Here, the first image acquisition unit 121 acquires a calculated captured image that is an image that cannot be visually recognized in the same manner as the state of the real space, but the second image acquisition unit 122 is visually in the same way as the state of the real space. A normal captured image that is a recognizable image is acquired.

  As shown in FIG. 4, in step S <b> 4, the identification correct answer acquiring unit 123 obtains the identification result (identification correct answer) of the imaging target and the surrounding environment of the imaging target included in the captured image acquired by the second image acquisition unit 122. get. The identification correct answer includes, for example, category information to which the imaging object and the surrounding environment of the imaging object (a person such as a person, a car, a bicycle, or a signal) belong, and the plane of the imaging object and the surrounding environment of the imaging object on the image. Position and region. The identification correct answer may include the imaging object on the image and the position in the depth direction of the surrounding environment of the imaging object. The identification correct answer is given from the outside of the identification system 1 </ b> A together with the first calculated captured image, or is given by the user to the captured image by the second image acquisition unit 122. The identification correct answer acquisition unit 123 identifies the subject based on the position of the subject in the captured image, and associates the identified subject with the category. As a result, the identification correct answer acquiring unit 123 acquires the subject area, the subject category, and the position information of the subject with respect to the captured image acquired by the second image acquiring unit 122 in association with each other. And

  The identification correct answer acquisition unit 123 uses an index when determining the plane position and area on the captured image of the subject. For example, the identification correct answer acquiring unit 123 uses a frame surrounding the subject as the index. Hereinafter, the frame surrounding the subject is also referred to as an identification area frame. The identification area frame can indicate the position and area of the subject. An example of the identification area frame is shown in FIGS. 8A and 8B.

  FIG. 8A is a schematic diagram illustrating a captured image in which an identification area frame is superimposed and displayed. FIG. 8B is a schematic diagram showing only the identification area frame.

  In the example shown in FIGS. 8A and 8B, the identification correct answer acquiring unit 123 sets a rectangular identification area frame that surrounds each subject from the outside and circumscribes each subject. Note that the shape of the identification area frame is not limited to the example of FIGS. 8A and 8B.

  8A and 8B, the identification correct answer acquiring unit 123 sets, for example, an identification area frame FA for the person A, an identification area frame FB for the automobile B, and an identification area frame FC for the automobile C. At this time, the identification correct answer acquiring unit 123 may calculate the alignment and coordinates of the entire identification area frame as information indicating the shape and position of the identification area frame, and calculate the coordinates of each vertex of the identification area frame. Alternatively, the coordinates of one vertex such as the upper left of the identification area frame and the length of each side may be calculated. For example, as described above, the coordinates are coordinates with respect to the origin when the origin is matched between the first calculated captured image and the captured image. As described above, the identification correct answer acquiring unit 123 outputs information including the plane position (coordinates) and the shape of the area of the identification area frame as the identification correct answer. Note that, as an identification correct answer, a captured image may be included in addition to the planar position and shape of the area of the identification area frame. Further, here, as an identification correct answer, an identification area frame is not set on the road, but an identification area frame may also be set for a surrounding environment such as a road.

  Further, the identification correct answer acquisition unit 123 may acquire the identification correct answer for each pixel instead of acquiring the identification area frame information as the identification correct answer. The identification correct answer for each pixel may be given as a mask on the image, for example, as shown by dot hatching in FIG.

  FIG. 9 is a schematic diagram illustrating an example of an identification correct answer given as a mask on an image.

  In the example of FIG. 9, the mask Aa is given to the person A and the masks Ba and Ca are given to the cars B and C, respectively, as correct identification answers. By doing in this way, the identification correct answer acquisition part 123 outputs an identification correct answer for every pixel. Here, as an identification correct answer, no mask is given to the road, but a mask may be given to the surrounding environment such as the road.

  As illustrated in FIG. 4, in step S5, the learning unit 124 refers to the correspondence relationship between the plurality of first pixels and the plurality of second pixels acquired in step S1, and based on the identification result of the captured image. Thus, an identification model (identifier) for identifying the first calculated captured image is generated. For example, each position in the captured image is referred to by referring to the correspondence relationship between the plurality of second pixels included in the captured image illustrated in FIG. 6 and the plurality of first pixels included in the first calculated captured image illustrated in FIG. It can be recognized which position (pixel) corresponds to each pixel in the first calculated captured image. Then, for example, the identification correct answer for the persons A1, A2, and A3 included in the first calculated captured image shown in FIG. 7 is the identification area frame FA as shown in FIG. 8B, which is the identification result of the captured image shown in FIG. Or the position of the mask Aa as shown in FIG. 9, and the machine learning is performed so that the category is human, and a discriminator is generated. In the same manner, machine learning is performed such that the identification correct answer for the automobiles B1, B2, B3, and B4 is the position of the identification area frame FB or the position of the mask Ba, and the category is the automobile, and the automobiles C1, C2 , C3 and C4 are machine-learned so that the correct answer for identification is the position of the identification area frame FC or the position of the mask Ca, and the category is a car, thereby generating a classifier. At this time, machine learning may also be performed on the imaging object and the position of the surrounding environment of the imaging object in the depth direction. Although details will be described later, a position in the depth direction can be easily acquired by using a multi-view stereo camera or the like as a camera for capturing a normal captured image, and machine learning is performed based on the acquired position in the depth direction. Can do.

  Many sets (for example, several thousand sets) of captured images as shown in FIG. 6 and first calculated captured images as shown in FIG. 7 are prepared. The learning unit 124 acquires the discriminator stored in the second memory 224, inputs the first calculated captured image to the discriminator, acquires the output result, and the output result is the first calculated captured image. The discriminator is adjusted so that the correct discrimination is input using the captured image corresponding to. Then, the learning unit 124 updates the discriminator in the second memory 224 by storing the adjusted discriminator in the second memory 224.

  In step S6, the learning unit 124 outputs an identification model (identifier) to the image identification device 10 that identifies the second calculated captured image. Thereby, the image identification device 10 uses the classifier generated by the learning device 12 and the imaging object and the imaging object included in the second calculated captured image that cannot be visually recognized by the person in the same manner as the state of the real space. The surrounding environment of the object can be identified. This will be described with reference to FIGS.

  FIG. 10 is a flowchart illustrating an example of an operation flow of the image identification device 10 according to the embodiment. In the following description, it is assumed that the imaging unit 11 is a light field camera.

  In step S <b> 101, the acquisition unit 101 acquires a second calculated captured image including the imaging object captured by the imaging unit 11 and the surrounding environment of the imaging object from the first memory 203 (see FIG. 3). Specifically, when the first input circuit 201 acquires the second calculated captured image from the first memory 203, the acquisition unit 101 acquires the second calculated captured image. For example, the imaging unit 11 captures (acquires) a light field image as the second calculated captured image for each first period that is a predetermined period, and the image is stored in the first memory 203. The acquisition unit 101 acquires the light field image captured by the imaging unit 11 and outputs it to the identification unit 102. The acquisition unit 101 may acquire a light field image from the outside of the identification system 1 (specifically, the light field image from the outside is stored in the first memory 203, and the acquisition unit 101 A light field image may be acquired from the memory 203).

  Next, in step S <b> 102, the identification unit 102 identifies the imaging target in the second calculated captured image using the classifier stored in the first memory 203. That is, the identification unit 102 detects an object to be identified in the light field image. The object to be identified may be set in the classifier in advance. For example, when the identification system 1 is mounted on a car, examples of objects to be identified are a person, a car, a bicycle, a signal, and the like. The identification unit 102 inputs the light field image to the classifier, and acquires the identification result of the object to be identified as an output result from the classifier. Details of the identification processing by the identification unit 102 will be described later. Note that the identification unit 102 may store the light field image subjected to the identification process in the first memory 203.

  Next, in step S <b> 103, the output unit 103 outputs a result (identification result) subjected to identification processing by the identification unit 102. For example, the output unit 103 may output image information including a light field image, or may output image information not including a light field image. At least the image information may include information on the object detected by the identification unit 102. The object information includes the position of the object (position on the plane or position in the depth direction), region, and the like. The output unit 103 may output image information to at least one of a display and an external device provided in the identification system 1.

  Further, the identification process in step S102 in FIG. 10 will be described. Image information and depth information can be simultaneously acquired from a light field image captured by the imaging unit 11 that is a light field camera. The identification unit 102 performs identification processing on the light field image using the classifier learned by the learning device 12. As described above, this learning is realized by machine learning using a neural network such as deep learning.

  The identification unit 102 may be configured to identify texture information and depth information, and collectively identify objects included in the image using the identified texture information and depth information. Such a configuration is shown in FIG.

  FIG. 11 is a schematic diagram illustrating an example of a functional configuration of the identification unit 102.

  As shown in FIG. 11, such an identification unit 102 includes a texture information identification unit 1021, a depth information identification unit 1022, and an integrated identification unit 1023. The texture information identification unit 1021 and the depth information identification unit 1022 are connected in parallel to the integrated identification unit 1023, for example.

  The texture information identification unit 1021 detects the subject using the texture information in the light field image. Specifically, the texture information identifying unit 1021 uses, for example, a neural network as described in Non-Patent Document 1 as a classifier, so that the subject area (position on the plane) and the subject in the light field image are detected. Identify the category. The input information to the texture information identification unit 1021 is a light field image, and the identification result of the texture information identification unit 1021 is the subject area and subject category on the light field image, as in the learning device 12. is there. In the case of a normal captured image, since the value of the direction of the incident light beam, that is, the depth value is integrated and included in the pixel value, the depth information is deleted. Compared to such a normal captured image, the light field image includes a lot of information about the subject in the image itself. For this reason, a light field image using a multi-pinhole or the like is used as input information of the discriminator, so that it is possible to perform discrimination with higher accuracy than when a normal captured image is used as input information.

  The depth information identification unit 1022 detects the depth information of the subject from the light field image. Specifically, the depth information identification unit 1022 learns in advance the depth information of the subject corresponding to the light field image in the learning device 12. As will be described later, the depth information of the subject may be calculated by acquiring a multi-view stereo image from the second image acquisition unit 122, or may be acquired from the identification correct acquisition unit 123.

  The integrated identification unit 1023 integrates the identification result of the texture information identification unit 1021 and the identification result of the depth information identification unit 1022 and outputs a final identification result. The discriminator used by the integrated discriminating unit 1023 receives the texture information of the texture information discriminating unit 1021 or its identification result and the depth information which is the discriminating result of the depth information discriminating unit 1022 and outputs the final discrimination result. It is. The final identification result includes the area of the object included in the light field image, the planar position on the image of the area, the depth position of the area, and the like.

  Note that a neural network for the texture information identification unit 1021 and a neural network for the depth information identification unit 1022 may be generated. That is, for the position and category in the plane, a neural network for identifying the position and category in the plane is used, and for the position in the depth direction, it is generated separately from the neural network for identifying the position and category in the plane. A neural network for identifying the position in the depth direction may be used. Further, the neural network for the texture information identification unit 1021 and the neural network for the depth information identification unit 1022 may be generated together. That is, for the position in the plane, the position in the depth direction, and the category, one neural network for collectively identifying the position in the plane, the position in the depth direction, and the category may be used.

  In the above description, the imaging unit 11 is a light field camera using a multi-pinhole or a microlens, but is not limited thereto. For example, the imaging unit 11 may be configured to capture a coded aperture image. This is also a kind of multi-pinhole camera.

  FIG. 12 is a schematic diagram of an example of a coded aperture mask that uses a random mask as a coded stop.

  As shown in FIG. 12, the coded aperture mask 311 has a light transmission region indicated by a non-colored region and a light shielding region indicated by a blacked region, and the light transmission region and the light shielding region are random. It can be seen that the Such a coded aperture mask 311 is produced by evaporating chromium on glass. When such a coded aperture mask 311 is placed on the optical path between the main lens and the image sensor, part of the light beam is blocked. As a result, it is possible to realize a camera that captures an encoded aperture image.

  Further, the second image acquisition unit 122 may acquire an image that can acquire depth information in addition to image information instead of a normal image. For example, the second image acquisition unit 122 may be configured with a multi-view stereo camera. The second image acquisition unit 122 can also acquire the three-dimensional information of the subject by acquiring the multi-view stereo image. Therefore, by calibrating the images acquired by the first image acquisition unit 121 and the second image acquisition unit 122 in advance, the images acquired by the first image acquisition unit 121 and the images acquired by the second image acquisition unit 122 Correspondence can be acquired. In this calibration, correspondence between the three-dimensional coordinates acquired by the second image acquisition unit 122 and the image coordinates acquired by the first image acquisition unit 121 is required. Thereby, the identification correct answer with respect to the captured image acquired by the second image acquisition unit 122 can be converted into the identification correct answer with respect to the first calculated captured image acquired by the first image acquisition unit 121. As described above, the captured image may be an image obtained by imaging the imaging object and the surrounding environment of the imaging object by the multi-view stereo camera.

  In the above description, as the identification correct answer, for example, category information to which a subject such as a person, a car, a bicycle, or a signal belongs, the planar position and area of the subject on the image, and the depth direction of the subject on the image Gave position. For example, the identification system 1 identifying the position (depth information) in the depth direction as the identification correct answer identifies the position (depth information) in the depth direction obtained from the multi-view stereo acquired by the second image acquisition unit 122. It can be realized by giving as.

  In addition, the identification unit 102 does not have a configuration in which the texture information identification unit 1021 and the depth information identification unit 1022 are in a parallel relationship, but is identified by the texture information identification unit 1021 after the depth information is extracted by the depth information identification unit 1022. May be configured.

  FIG. 13 is a schematic diagram illustrating another example of the functional configuration of the identification unit 102.

  As illustrated in FIG. 13, in the identification unit 102, the depth information identification unit 1022, the texture information identification unit 1021, and the integrated identification unit 1023 may be in a serial relationship. The depth information identification unit 1022 generates a depth image for the light field image. The texture information identification unit 1021 uses the depth image generated by the depth information identification unit 1022 as input information, for example, by using a neural network as described in Non-Patent Document 1, and thereby the subject position, region, and subject category. Identify The integrated identification unit 1023 outputs the identification result of the texture information identification unit 1021. As in the case where the texture information identifying unit 1021 and the integrated identifying unit 1023 are in a parallel relationship, the final identification result is the region of the object included in the light field image, the planar position on the image of the region, and the region. Including the depth position.

  In addition, the identification unit 102 may change the configuration of the neural network according to the imaging unit 11. When the imaging unit 11 is a light field camera, the depth image is generated using the position and size of the multi-pinhole of the imaging unit 11. For example, when the position and size of the multi-pinholes differ for each imaging unit 11 due to the type or manufacturing variation of the imaging unit 11, a neural network is configured for each imaging unit 11 (in other words, for each imaging unit 11. By performing machine learning individually, the identification accuracy of the identification unit 102 can be improved. Information on the position and size of the multi-pinhole can be acquired by performing camera calibration in advance.

  As described above, the identification unit 102 uses the light field image as input information, and performs identification processing from the texture information and depth information of the light field image. As a result, the identification unit 102 can identify how far away it is from the identification process based only on the texture image using the conventional normal captured image, for example, thus enabling a highly accurate identification process. .

  As described above, the identification system 1 according to the embodiment including the image identification device 10 including the identification unit 102, and the identification system 1A according to a modification of the embodiment including the image identification device 10 and the learning device 12 Was illustrated. However, for example, the identification unit 102 may include the learning device 12, and in this case, the identification system 1 includes the learning device 12. That is, in this case, the identification system 1 has a function equivalent to that of the identification system 1A.

  As described above, in the identification systems 1 and 1A according to the embodiment and the modification, the image identification device 10 identifies the subject in the image using the second calculated captured image such as the light field image. . Further, the image identification device 10 does not restore the second calculated captured image to the normal captured image in the course of a series of identification processes, and is included in the texture information included in the second calculated captured image and the calculated captured image. The subject in the second calculated captured image is identified based on the depth information. Therefore, the image identification device 10 can reduce the amount of subject identification processing. In particular, the image identification device 10 can significantly speed up the identification process as compared with a technique involving image restoration from the second calculated captured image to the normal captured image during the identification process. Further, since the depth information can be acquired without using a three-dimensional range finder or the like, the cost can be reduced.

  Also, the optical axis of the camera (for example, the first image acquisition unit 121) used for capturing the first calculated captured image and the optical axis of the camera (for example, the second image acquisition unit 122) used for capturing the captured image. It does not matter if they are approximately the same. FIG. 14A is a schematic diagram for explaining this.

  FIG. 14A is a schematic diagram showing that the optical axis of the second image acquisition unit 122 and the optical axis of the first image acquisition unit 121 are approximately the same.

  In this figure, as the first image acquisition unit 121 and the second image acquisition unit 122, cameras that are examples of the hardware are schematically shown. An optical axis 231 indicates the optical axis of the first image acquisition unit 121, and an optical axis 232 indicates the optical axis of the second image acquisition unit 122. In order to make the optical axes approximately coincide with each other, the first image acquisition unit 121 and the second image acquisition unit 122 may be brought close to each other and arranged so that the optical axes are substantially parallel to each other.

  Further, when the second image acquisition unit 122 is configured as a stereo camera, the optical axes of the two cameras configuring the second image acquisition unit 122 and the optical axes of the first image acquisition unit 121 are approximately the same. do it. FIG. 14B is a schematic diagram for explaining this.

  FIG. 14B is a schematic diagram showing that the optical axes of the stereo camera constituting the second image acquisition unit 122 and the optical axes of the first image acquisition unit 121 are approximately the same.

  In this figure, the same components as those in FIG. In this figure, optical axes 232 a and 232 b indicate the respective optical axes of the stereo camera constituting the second image acquisition unit 122. As described above, the identification system 1 or 1A according to the present embodiment identifies the correct identification for the captured image acquired by the second image acquisition unit 122 and the identification for the first calculated captured image acquired by the first image acquisition unit 121. Although it is converted to a correct answer, by making the optical axes approximately coincident, errors due to the conversion can be reduced, and more accurate identification can be realized.

  Further, in order to make the optical axes of the first image acquisition unit 121 and the second image acquisition unit 122 coincide with each other, a beam splitter, a prism, a half mirror, or the like may be used.

  FIG. 15 is a schematic diagram showing that a beam splitter is used to match the optical axis of the first image acquisition unit 121 and the optical axis of the second image acquisition unit 122.

  In this figure, the same components as those in FIG. Since the light beam from the subject can be separated into two by the beam splitter 240, one of the separated light beams is made to coincide with the optical axis 231 of the first image acquisition unit 121, and the other one of the second image acquisition unit 122. By matching with the optical axis 232, the optical axis of the first image acquisition unit 121 and the optical axis of the second image acquisition unit 122 can be matched. Thus, the optical axis of the camera (for example, the first image acquisition unit 121) used for capturing the first calculated captured image and the optical axis of the camera (for example, the second image acquisition unit 122) used for capturing the captured image. Is matched with a beam splitter, a prism, or a half mirror. As described above, the identification system 1 or 1A according to the present embodiment identifies the correct identification for the captured image acquired by the second image acquisition unit 122 and the identification for the first calculated captured image acquired by the first image acquisition unit 121. Although it is converted to a correct answer, by making each optical axis coincide, an error accompanying the conversion can be reduced, and more accurate identification can be realized.

  Although the learning device 12 of the present disclosure has been described based on the embodiments, the present disclosure is not limited to the above embodiments. Unless it deviates from the gist of the present disclosure, various modifications conceived by those skilled in the art have been made in the present embodiment, and forms constructed by combining components in different embodiments are also included in the scope of the present disclosure. It is.

  For example, in the above embodiment, the imaging object and the position in the plane of the surrounding environment of the imaging object, the position in the depth direction, and the category information included in the second calculated captured image are identified, but the present invention is not limited to this. For example, only one or two of the imaging object and the position in the plane of the surrounding environment of the imaging object, the position in the depth direction, and the category information may be identified. That is, only one or two of the imaging object and the position in the plane of the surrounding environment of the imaging object, the position in the depth direction, and the category information may be machine-learned to generate the identification model.

  Further, for example, in the above-described embodiment, the machine learning is performed on the position in the depth direction, but it may not be performed. For example, when the acquisition unit 101 acquires the second calculated captured image, an image of the subject is used by using an image in which a plurality of imaging objects and surrounding environments of the imaging target included in the second calculated captured image are respectively superimposed. The position in the depth direction may be calculated. That is, the position in the depth direction may be calculated directly from the second calculated captured image itself without using the identification model.

  Moreover, for example, the correct identification for the captured image acquired by the second image acquisition unit 122 is manually given by a person, for example, but is not limited thereto. For example, a learning model for giving an identification correct answer to the captured image acquired by the second image acquisition unit 122 may be prepared in advance, and the identification correct answer may be given using the learning model.

  Further, for example, the present disclosure can be realized not only as the learning device 12 but also as a learning method including steps (processes) performed by each component constituting the learning device 12.

  Specifically, as shown in FIG. 4, the learning method is a first calculated captured image including an imaging target object and a surrounding environment of the imaging target object, and includes a first pixel having a plurality of first pixels. A calculated captured image is acquired (step S2), and a captured image including a captured object and a surrounding environment of the captured object, the captured image having a plurality of second pixels is acquired (step S3). The identification result of the imaging target object and the surrounding environment of the imaging target object are acquired (step S4), and the correspondence relationship between the plurality of first pixels and the plurality of second pixels is referred to to obtain the identification result of the captured image. Based on this, an identification model for identifying the first calculated captured image is generated (step S5), and the identification model is output to the image identification device 10 for identifying the second calculated captured image (step S6).

  Further, for example, these steps may be executed by a computer (computer system). The present disclosure can be realized as a program for causing a computer to execute the steps included in these methods. Furthermore, the present disclosure can be realized as a non-transitory computer-readable recording medium such as a CD-ROM or the like on which the program is recorded.

  Further, in this disclosure, all or part of the system, device, member, or part, or all or part of the functional blocks in the block diagrams shown in the drawings may be a semiconductor device, a semiconductor integrated circuit (IC), or an LSI ( It may be performed by one or more electronic circuits including large scale integration).

  The LSI or IC may be integrated on a single chip, or may be configured by combining a plurality of chips. For example, the functional blocks other than the memory element may be integrated on one chip. Here, the term “LSI” or “IC” is used, but the term changes depending on the degree of integration, and it may be called system LSI, VLSI (very large scale integration), or ULSI (ultra large scale integration). A field programmable gate array (FPGA), which is programmed after the manufacture of the LSI, or a reconfigurable logic device capable of reconfiguring the junction relationship inside the LSI or setting up a circuit partition inside the LSI can be used for the same purpose.

  Furthermore, the functions or operations of all or part of the system, apparatus, member, or unit can be executed by software processing as described above. In this case, the software is recorded on a non-transitory recording medium such as at least one ROM, optical disk, or hard disk drive, and when the software is executed by a processor, the function specified by the software is processed. It is executed by a processor and peripheral devices.

  The system or apparatus may include one or more non-transitory recording media in which software is recorded, a processor, and a hardware device.

  Further, the numbers such as the ordinal numbers and the quantities used in the above are examples for specifically explaining the technology of the present disclosure, and the present disclosure is not limited to the illustrated numbers. In addition, the connection relationship between the constituent elements is exemplified for specifically explaining the technology of the present disclosure, and the connection relationship for realizing the functions of the present disclosure is not limited thereto.

  In addition, division of functional blocks in the block diagram is an example, and a plurality of functional blocks are realized as one functional block, one functional block is divided into a plurality of parts, or some functions are transferred to other functional blocks. May be. A single piece of hardware or software may process the functions of a plurality of functional blocks having similar functions in parallel or in time division.

  The identification system 1A according to an aspect of the present disclosure identifies an imaging unit 11 that captures a second calculated captured image including information on a surrounding environment of an imaging target, and a second calculated captured image captured by the imaging unit 11 This is an identification system including an image identification device 10 that detects a subject included in the image using a classifier and outputs a detection result, and a learning device 12 that generates a classifier. The learning device 12 obtains an identification correct answer related to the captured image acquired by the first image acquisition unit 121 that acquires the first calculated captured image, the second image acquisition unit 122 that acquires the captured image, and the second image acquisition unit 122. The identification correct answer acquisition unit 123 to be acquired and the learning unit 124 to acquire the classifier by performing machine learning on the first calculated captured image acquired by the first image acquisition unit 121 using the identification correct answer regarding the captured image. It is characterized by providing.

  The imaging unit 11 and the first image acquisition unit 121 include a multi-pinhole camera, a coded aperture camera, a light field camera, or a lensless camera.

  The imaging unit 11 and the first image acquisition unit 121 acquire an image that cannot be visually recognized even when viewed by a person as a calculated captured image.

  The learning device 12 uses the correspondence of the positional relationship on the image of the first calculated captured image acquired by the first image acquisition unit 121 and the captured image acquired by the second image acquisition unit 122, so that the second The identification correct answer regarding the captured image acquired by the image acquisition unit 122 is learned as the identification correct answer of the first image acquisition unit 121.

  The second image acquisition unit 122 acquires an image that can acquire depth information in addition to the image information.

  The second image acquisition unit 122 is a multi-view stereo camera.

  In the learning device 12, the optical axes of the first image acquisition unit 121 and the second image acquisition unit 122 are approximately the same.

  The learning device 12 further includes a beam splitter, and the optical axes are matched using the beam splitter.

  The learning device 12 according to an aspect of the present disclosure is acquired by the first image acquisition unit 121 that acquires the first calculated captured image, the second image acquisition unit 122 that acquires the captured image, and the second image acquisition unit 122. The identification correct answer acquisition unit 123 that acquires the identification correct answer related to the captured image, and the machine learning for the first calculated captured image acquired by the first image acquisition unit 121 using the identification correct answer related to the captured image identifies And a learning unit 124 for acquiring a vessel.

  In the learning method according to an aspect of the present disclosure, the first calculation captured image and the captured image are detected from the first calculated captured image by using the classifier to detect a subject included in the image and outputting a detection result. , The identification correct answer regarding the captured image is acquired, and the classifier is generated by performing machine learning on the first calculated captured image using the identification correct answer regarding the captured image.

  The technique of the present disclosure can be widely applied to a technique for recognizing an object in a calculated captured image. The technology of the present disclosure can be widely applied even when an imaging device that captures a calculated captured image is mounted on a moving body that requires a high identification processing speed. For example, an automatic driving technology for an automobile, a robot, It can be applied to a peripheral monitoring camera system or the like.

DESCRIPTION OF SYMBOLS 1,1A identification system 10 Image identification apparatus 11 Imaging part 12 Learning apparatus 101 Acquisition part 102 Identification part 103 Output part 121 First image acquisition part 122 Second image acquisition part 123 Identification correct answer acquisition part 124 Learning part 201 First input circuit 202 First arithmetic circuit 203 First memory 204 Output circuit 211 Light field camera 211a Multi-pinhole mask 211aa Pinhole 211b Image sensor 221 Second input circuit 222 Third input circuit 223 Second arithmetic circuit 224 Second memory 231, 232, 232a , 232b Optical axis 240 Beam splitter 311 Encoded aperture mask 1021 Texture information identification unit 1022 Depth information identification unit 1023 Integrated identification unit

Claims (11)

A learning device comprising a memory and a processing circuit,
The processing circuit is
(A) obtaining a first calculated captured image including an imaging target and a surrounding environment of the imaging target from the memory, wherein the first calculated captured image includes a plurality of first pixels;
(B) acquiring a captured image including the imaging object and a surrounding environment of the imaging object from the memory, the captured image having a plurality of second pixels;
(C) Obtaining the identification result of the imaging object and the surrounding environment of the imaging object included in the captured image;
(D) An identification model for identifying the first calculated captured image based on the identification result of the captured image with reference to the correspondence relationship between the plurality of first pixels and the plurality of second pixels. Produces
(E) outputting the identification model to an image identification device for identifying a second calculated captured image;
Learning device. The identification result includes a position in a plane of a surrounding environment of the imaging object and the imaging object,
The learning device according to claim 1. The identification result includes a position in a depth direction of the surrounding environment of the imaging object and the imaging object,
The learning device according to claim 1 or 2. The identification result includes category information to which the imaging object and the surrounding environment of the imaging object belong.
The learning device according to claim 1. The first calculated captured image and the second calculated captured image are images including parallax information in which a plurality of surroundings of the imaging object and the imaging object are respectively superimposed.
The learning device according to any one of claims 1 to 4. The first calculated captured image and the second calculated captured image are images of the imaging object and the surrounding environment of the imaging object by a multi-pinhole camera, a coded aperture camera, a light field camera, or a lensless camera. Is an image obtained by
The learning device according to claim 5. The captured image is an image obtained by imaging the imaging object and the surrounding environment of the imaging object with a multi-view stereo camera.
The learning device according to claim 1. 8. The learning according to claim 1, wherein an optical axis of a camera used for imaging the first calculated captured image and an optical axis of a camera used for capturing the captured image substantially coincide with each other. apparatus. The optical axis of the camera used for imaging the first calculated captured image and the optical axis of the camera used for capturing the captured image coincide with each other via a beam splitter, a prism, or a half mirror. The learning device described. (A) a first calculated captured image including an imaging object and a surrounding environment of the imaging object, the first calculated captured image having a plurality of first pixels;
(B) A captured image including the imaging object and a surrounding environment of the imaging object, the captured image having a plurality of second pixels,
(C) Obtaining the identification result of the imaging object and the surrounding environment of the imaging object included in the captured image;
(D) An identification model for identifying the first calculated captured image based on the identification result of the captured image with reference to the correspondence relationship between the plurality of first pixels and the plurality of second pixels. Produces
(E) outputting the identification model to an image identification device for identifying a second calculated captured image;
Learning method.   A program for causing a computer to execute the learning method according to claim 10.

Images