JP2019125116A - Information processing device, information processing method, and program - Google Patents

Abstract

To provide an information processing device capable of highly accurately performing image recognition, an information processing method, and a program.SOLUTION: An information processing device for outputting output results corresponding to a photographic image input on the basis of a learning model comprises: first acquisition means for acquiring photographing conditions under which the image used to generate the learning model was photographed; input means for inputting a photographic image of a processing object from an imaging device; second acquisition means for acquiring the photographing conditions under which the photographic image of the processing object was photographed; conversion means for converting the photographic image of the processing object on the basis of the photographing conditions acquired by the first and second acquisition means; and output means for outputting output results corresponding to the converted image on the basis of the learning model.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technology for performing image recognition.

  A technique of machine learning is known which learns patterns and feature points from an image obtained by photographing an object in advance, and recognizes the type, position, and shape of the object on the image using the learning result. In order to operate the image recognition by machine learning with high accuracy, the shooting conditions of the image to be learned and the shooting conditions of the image to be input to the recognizer learned so as to be able to recognize the above pattern and feature points match as much as possible. Is desirable. Here, the shooting conditions include the position and orientation of the imaging apparatus or the light source that affects the shooting locations when shooting an image for learning or shooting an image to be input to a recognizer. Indicates conditions such as brightness and tint.

  Patent Document 1 discloses a method of generating a learning image of an object for image recognition.

  Further, in Non-Patent Document 1, a method related to recognition processing constructed by machine learning based on an image given in advance recognizes a three-dimensional shape of an object in an image from a photographed image, and based on the shape A method is disclosed for estimating the position and orientation of the imaging device that has taken the above image.

JP, 2014-199584, A

K. Tateno, et. al. CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction, IEEE Computer Society Conference CVPR, 2017

  In Non-Patent Document 1, there is a premise that the shooting conditions of an image to be learned and the shooting conditions of an image to be input to a recognizer learned so as to be able to recognize a pattern or a feature point match. However, since it is practically difficult to match the shooting conditions of the environment in which learning is performed and the environment in which recognition is actually performed, recognition could not be performed with high accuracy.

  Patent Document 1 discloses a method of converting an image at the time of learning so as to correspond to a photographing condition for performing image recognition and then learning. However, with this method, if the imaging conditions of the environment in which image recognition is actually performed can not be grasped in advance, an image for performing necessary and sufficient learning can not be generated, and efficient learning can not be performed. In particular, when the environment in which image recognition is actually performed is unknown at the time of prior learning, the accuracy of recognition is degraded.

  The present invention has been made in view of the above problems, and improves the accuracy of image recognition even when shooting conditions do not match between an environment in which learning is performed in advance and an environment in which recognition is actually performed. The purpose is to provide technology.

  An information processing apparatus according to the present invention for achieving the above object is an information processing apparatus for outputting an output result corresponding to an input photographed image based on a learning model, and an image used when generating the learning model A first acquisition unit for acquiring a photographing condition in which a subject is photographed, an input unit for inputting a photographed image to be processed from an imaging device, and a second acquisition unit for acquiring a photographing condition in which the photographed image to be processed is photographed A conversion means for converting the photographed image to be processed based on the photographing conditions acquired by the first and second acquisition means, and an output result corresponding to the converted image based on the learning model And output means for

  According to the present invention, it is possible to provide a technique for improving the accuracy of image recognition even when shooting conditions do not match between an environment in which learning is performed in advance and an environment in which recognition is actually performed.

Block diagram showing a configuration example of an information processing apparatus Diagram showing the difference in the arrangement of imaging devices at recognition (a) and at learning (b) Block diagram showing an example of functional configuration of information processing apparatus Flow chart showing processing of information processing apparatus Block diagram showing an example of functional configuration of information processing apparatus Block diagram showing the configuration of an information processing system A diagram showing an example in which an information processing apparatus is mounted on a vehicle Flow chart showing processing of the entire information processing system

First Embodiment
In this embodiment, when performing image recognition using a learning model, attention is paid to changes (differences) between images captured under two conditions at the time of learning and at the time of image recognition processing. A method of improving the accuracy of image recognition by geometrically deforming an image used for recognition processing and bringing it close to the imaging conditions of the image used for learning will be described. In the image recognition in this embodiment, in this embodiment, the distance from the object shown in the input image to the imaging device is estimated. By using a learning model in which the input image and the correct value of the distance are learned, estimation of the distance becomes more accurate. The recognition process may be any method as long as it is a method of recognizing some information from an image by a method based on machine learning. Further, in the present embodiment, the learning model is a network structure based on a neural network that outputs distance information corresponding to an input image from an input image and parameters thereof.

  First, geometric “misalignment” that occurs in an image occurs because the position and orientation of the imaging device are different during learning and recognition processing. The "deviation" refers to a change (difference) between images captured under two conditions. For example, consider the case of mounting an imaging device on a vehicle as shown in FIG. Even if the same height or the same type of vehicle can be used in learning and recognition processing, there is a possibility that the positions and orientations of the imaging device can not all be the same. In addition, the shape and type may differ between the imaging device used for learning and the imaging device used when performing recognition processing. Therefore, it may be considered that the shooting conditions of the image used for learning do not match the shooting conditions of the image input to the recognizer learned so as to be able to recognize the pattern and feature points. With images having different imaging conditions, the effect of learning is less likely to be reflected, and the accuracy of image recognition is reduced.

  A configuration example of hardware of the present embodiment is shown using FIG.

  The central processing unit (CPU) 101 uses the RAM 103 as a work memory, reads and executes the OS and other programs stored in the ROM 102 and the storage device 104, controls various components connected to the system bus 106, Perform processing operations and logical decisions. The processing executed by the CPU 101 includes the image recognition processing of the embodiment.

  The storage device 104 is a hard disk drive, an external storage device, or the like, and stores programs and various data related to the image recognition processing of the embodiment.

  The input unit 105 is an imaging device such as a camera, an input device such as a button for inputting a user instruction, a keyboard, and a touch panel. The storage unit 104 is connected to the system bus 109 via an interface such as SATA, and the input unit 105 is connected to the system bus 109 via a serial bus such as USB. The communication unit 106 communicates with an external device by wireless communication. The display unit 107 is a display. The sensor 108 is an image sensor or a distance sensor.

  Note that the CPU can function as various means by executing a program. Note that a control circuit such as an ASIC operating in cooperation with the CPU may function as these means. Also, these means may be realized by cooperation between the CPU and a control circuit that controls the operation of the image processing apparatus. Also, the CPU need not be a single CPU, but may be plural. In this case, a plurality of CPUs can execute processing in a distributed manner. Also, the plurality of CPUs may be disposed in a single computer or may be disposed in physically different computers. Note that means realized by the CPU executing a program may be realized by a dedicated circuit.

  FIG. 2 is a diagram showing the difference between a vehicle 100 (a) used for image recognition processing and a vehicle 200 (b) used for learning. Further, (c) is a diagram showing an example of image conversion from the image 111 to the image 112 at the time of recognition processing. In the example of FIG. 2, the positions (heights) of the imaging devices mounted on the respective vehicles are different. Reference numerals 100 and 200 in FIG. 2 represent different vehicles, and reference numerals 110 and 210 represent imaging devices mounted on the respective vehicles. Although it is preferable that the two vehicles be of the same vehicle type, in this embodiment, they are different vehicle types. If the vehicle type is different, it is difficult to set the position and attitude of the imaging device exactly the same. When the position of the imaging device is different as shown in FIG. 2, as in the images shown by 111 and 211, geometric deviation occurs in the obtained image. The geometrical deviation in this case is specifically the deviation of the apparent position or angle of the road or building in the image. Therefore, for example, when learning is performed using the image 211 of the imaging device 210 in FIG. 2, the image 111 of the imaging device 110 with different imaging conditions is input to a learning model in which learning is performed to estimate the distance from the image. In some cases, sufficient image recognition accuracy may not be obtained.

So, in this embodiment, it uses for recognition processing based on the difference between the position and posture of the imaging device carried in the vehicle used for learning, and the position and posture of the imaging device carried in the vehicle used for recognition processing using a learning result The image is deformed geometrically to be close to the imaging condition of the image used for learning. As a result, the accuracy of image recognition is improved as much as possible even when the shooting conditions do not match between the environment in which learning is performed in advance and the environment in which recognition is actually performed. The functional configuration of the information processing apparatus of the embodiment will be described. Reference numeral 300 in FIG. 3 denotes an information processing apparatus according to this embodiment, and reference numeral 110 denotes an imaging apparatus for capturing an image. The information processing apparatus 300 includes an image acquisition unit 310, a photographing condition acquisition unit 320, an image conversion unit 330, a learning model holding unit 340, a learning model acquisition unit 350, and an image recognition unit 360.

  The imaging device 110 is a camera that captures an image of a two-dimensional process target. However, the camera may be a monochrome camera as well as a color camera. In this embodiment, this camera is a monocular color camera, and the captured image is a color image. Note that, as described with reference to FIG. 2, the imaging device 110 captures an image to be subjected to recognition processing using the learning result.

  The image acquisition unit 310 receives and acquires a color image to be processed from the imaging device 110.

  The imaging condition acquisition unit 320 acquires imaging conditions (first imaging conditions) of images used for learning and imaging conditions (second imaging conditions) of images used for image recognition. That is, the imaging conditions of the imaging device 110 and the imaging conditions of the imaging device 210 at the time of learning of FIG. 2 attached to different vehicles used for learning are acquired. In the present embodiment, the imaging conditions are the position and orientation of an imaging device mounted on a vehicle. (In this embodiment, the position and orientation will be described as including both the position and orientation.) The position and orientation are assumed to be the position and orientation in the coordinate system of the origin set in advance. In the present embodiment, in a state where the imaging device is mounted on a vehicle, the height based on the position of the ground is considered as information of the position, and the angle based on the horizontal plane is considered as the posture. The position and orientation may be any information that represents the position and orientation. For example, position and orientation parameters having a total of six degrees of freedom, that is, three degrees of freedom (X, Y, Z) and three degrees of freedom (Roll, Pitch, Yaw) on the world coordinate system of the environment may be used.

  The internal parameter matrix of the imaging device 110 and the imaging device 210 at the time of learning is known as K1 and K2, respectively. Here, the internal parameter matrix is a matrix including parameters related to the image center and the focus, and the values can be obtained by performing calibration in advance. Further, the position and orientation of the imaging device can be acquired as known information from design information related to the arrangement of the vehicle and the imaging device. The design information may be stored in the storage device 104 or may be received from the communication unit 106. Further, the photographing conditions are assumed to be stored in a storage area such as the storage device 104, and can be acquired by reading from the storage area.

  Besides the above, the imaging conditions may be converted by focusing attention on the imaging conditions for the zoom value of the image indicating the zoom magnification when imaging with the imaging device, the angle of view, and the hue. For example, when focusing on the angle of view as the difference in imaging conditions, the scale of the captured image is converted. The imaging conditions can be changed depending on the installation place and application of the imaging device. For example, the imaging device can be mounted on a surveillance camera or a work robot as well as a vehicle. In that case, the position and orientation of the image pickup apparatus may be considered to be the height based on the position of the ground as the position information, and the angle based on the horizontal surface as the attitude, or from the coordinate system of the reference separately designated by the user. You may consider the position and orientation of

  The image conversion unit 330 performs image conversion based on the imaging conditions of the image captured at the time of image recognition processing and the imaging conditions of the image used at the time of learning. That is, the image 111 acquired by the image acquisition unit 310 is geometrically converted to the image 112 based on the imaging condition of the imaging device 110 acquired by the imaging condition acquisition unit 320 and the imaging condition of the imaging device 210 at the time of learning.

  The learning model holding unit 340 holds a learning model learned using an image captured by the imaging device 210 at the time of learning and shooting conditions of the learning model. The imaging condition of the learning model refers to the imaging condition of a typical image used for learning. A representative image refers to the average image of the images used for learning. In the present embodiment, the learning model is a network structure based on a neural network that outputs a recognition result by machine learning from a learning image and its parameters. In the network learning, specifically, an image is set in the layer on the input side of the learning model, a correct value for the image is set in the layer of the output image, and an output calculated via the network is set as the correct value. It refers to the process of adjusting network parameters to get closer. In this embodiment, a two-dimensional image is used as an input to the learning model, and the learning model outputs a distance image in which the distance information is arranged corresponding to each position of the two-dimensional image. Do. This distance information is depth information from the imaging device, and represents the distance to a surrounding object at camera coordinates with the imaging device as the origin. In addition, since the technique which estimates a distance image from a two-dimensional image is known, detailed description is abbreviate | omitted. In Non-Patent Document 1, learning is performed by preparing a two-dimensional image and a distance image as its correct value. The distance image as the correct value is prepared by separately preparing measurement information obtained by measuring the surroundings with a distance sensor such as LiDER or ToF. The learning model is not limited to a model based on a neural network, but may be a learning model based on another machine learning method such as a random forest or a support vector machine. In addition, a predetermined object appearing in an image may be recognized. In that case, for example, learning image and its correct value (object shape, class, type) are given and learning so that the class and type of an object such as a person, a moving object, a vehicle and an obstacle are recognized from the image. I do.

  The learning model acquisition unit 350 acquires the learning model held by the learning model holding unit 340 and the imaging condition of the learning model. Alternatively, the learning model and the imaging condition of the learning model are acquired from the outside of the information processing apparatus 300. Specifically, the neural network held by the learning model holding unit 340 and the data of its parameters are read out.

  The image recognition unit 360 receives the image converted by the image conversion unit 330 and outputs distance information corresponding to the pixels or the area of the image captured by the imaging device 110 as an output result. In the present embodiment, the distance is estimated from the input image, and the distance information is output. Here, the recognition processing may estimate three-dimensional point group data other than the distance image as described in the present embodiment. In addition, a predetermined object appearing in an image may be recognized. For example, the position or type of a person or an object present in a scene may be recognized. When an on-vehicle imaging device is assumed, obstacles, marker recognition, a traveling region, and the position and posture of a vehicle may be recognized. As in the example of estimating a distance image, a learning model can be generated by performing learning with an image for learning and its correct value (shape of object, name). In either method, the accuracy of recognition can be improved by converting the image according to the shooting conditions of the image used for learning by the operation of the image conversion unit 330.

Next, the processing procedure of this embodiment will be described. FIG. 4 is a flowchart showing a processing procedure performed by the information processing apparatus. Hereinafter, the flowchart is realized by the CPU executing the control program. In the following description, S is added to the beginning of each process (step) and described, and the description of the process (step) is omitted.

  First, in S410, the learning model acquisition unit 350 acquires the learning model and the imaging conditions of the learning model from the learning model holding unit 340.

  In S420, the image acquisition unit 310 acquires an image captured by the imaging device 110.

  In S430, the imaging condition acquisition unit 320 acquires imaging conditions of the image captured by the imaging device 110 and imaging conditions of the image used for learning in the learning model. That is, the position and orientation of the imaging device 110 and the position and orientation of the imaging device 210 at the time of learning are acquired. Here, the difference between the relative positions and orientations of the two imaging devices is a three-dimensional vector t at a position and a rotation matrix R of three rows and three columns, with the imaging device 110 as an origin.

  In step S440, the image conversion unit 330 converts the image captured by the imaging apparatus 110 so that the imaging condition acquired by the imaging condition acquisition unit 320 approaches the imaging condition. That is, the image acquired by the image acquiring unit 310 is geometrically converted based on the imaging device imaging condition and the imaging condition of the imaging device 210 at the time of learning. Specifically, the geometric transformation described in the present embodiment is a homographic transformation (projective transformation). However, only some components such as position and rotation may be deformed, or affine transformation may be performed. For example, a part of the panoramic image may be cut, and a part of the image may be subjected to conversion such as scaling.

  First, the homography matrix H is calculated from the relative position and orientation of the imaging device 110 and the imaging device 210 at the time of learning. Assuming that the internal parameter matrix of the imaging device 110 and the imaging device 210 at the time of learning is K1 and K2, respectively, the homography matrix can be calculated by the following equation (1).

  Here, d is a numerical value representing the distance of the plane assumed with reference to the imaging device 110, and n is a normal vector of the plane. In the present embodiment, it is assumed that d is set in advance as a rough distance value of a scene and n is a direction facing the imaging device 110 in advance. T is a symbol representing a transposed matrix of a matrix.

  Finally, the homography matrix H is applied to the image acquired by the image acquisition unit 310 to deform the image. By this operation, it is possible to make the geometric imaging condition of the image close to the imaging condition at the time of learning.

  In S450, the image recognition unit 360 receives the image converted by the image conversion unit 330, and outputs distance information corresponding to the pixels of the image captured by the imaging device 110. Here, the learning model acquired by the learning model acquisition unit 350 is used. The recognition process in this embodiment may be any method as long as it is a method of recognizing some information from an image by a method based on machine learning. The position or type of a person or an object present in the scene may be recognized from the image, or the position and orientation of the vehicle or the control value may be recognized.

  In S460, the processing from S420 is repeatedly executed until there is a designation of system termination. If the system termination condition is satisfied, the process proceeds to YES to terminate the system. If the system termination condition is not satisfied, the process proceeds to NO and returns to S420. The condition of the system termination may be, for example, the user's input as a trigger for the system termination when considering automatic driving of a car. Here, the system ends at a predetermined time.

  As described above, in the present embodiment, recognition processing is performed by focusing on geometric “misalignment” that occurs in both photographed images due to differences between the photographing conditions of the image used for learning and the photographing conditions during recognition processing. Geometrically deform the image used for the image to bring it close to the imaging conditions of the image used for learning. This makes it possible to match the appearance of the image, thereby improving the recognition accuracy.

(Modification 1 of the first embodiment)
In the first embodiment, the position and orientation of the imaging device are considered as the imaging condition of the image, and the imaging condition acquisition unit 320 acquires the position and orientation of the imaging device based on design information and calibration results prepared in advance. However, when the imaging device is disposed in a vehicle, the position and posture of the imaging device in motion may be continuously fluctuated up and down in time series due to the unevenness of the road surface. Specifically, parameters of the position (height from the ground) of the imaging device and the posture (angle with respect to the horizontal plane) dynamically change with time. In order to cope with this dynamic change, the imaging condition acquisition unit 320 updates the imaging condition as needed from the result of the position and orientation estimation unit 510 estimating the position and orientation of the imaging device 110 at time s. Alternatively, the imaging conditions may be updated as needed from measurement information obtained by measuring the surroundings of the vehicle by a sensor separately attached to the vehicle. As a result, the imaging conditions can be updated at constant time intervals, and changes in imaging conditions that change with time can be reflected in image conversion, so that the accuracy of recognition can be improved. Here, processing of the position and orientation estimation unit 510 and the imaging condition acquisition unit 320, which are differences from the first embodiment, will be described.

  The position and orientation estimation unit 510 estimates imaging conditions (for example, the position of the imaging apparatus) of an image at a predetermined time. As a method by which the position and orientation estimation unit 510 estimates the position and orientation of the imaging device 110, a known SLAM (SIMULTANEOUS LOCALIZATION AND MAPPING) technique is used. Here, SLAM is a technology that accurately estimates its own position and orientation while recognizing the surrounding environment to be photographed by a camera. A specific method is disclosed in the aforementioned Non-Patent Document 1. Note that position and orientation estimation of the vehicle in SLAM may be performed based on the image acquired from the image acquisition unit 110 or may be performed based on measurement information of the sensor 108 which is a distance sensor. The distance sensor may be, for example, an active distance sensor such as LiDER or ToF, an infrared sensor, or a stereo camera. Further, the sensor 108 may be a position sensor such as a GPS or a gyro sensor or an attitude sensor. The first estimation method is a method in which the distance information measured by the imaging device 110 or the sensor 108 as described above is input to the SLAM. As a second estimation method, the position and orientation estimation unit 510 may estimate the position and orientation of the imaging apparatus using the distance image output from the image recognition unit 360 as an input.

  The imaging condition acquisition unit 320 updates the imaging conditions from the result of the position and orientation estimation unit 510 estimating the position and orientation of the imaging device 110. Alternatively, the imaging condition acquisition unit 320 may directly acquire and update measurement information on the position and orientation acquired by the sensor 108. In addition, the photographing conditions (for example, the brightness of the surroundings) that can be measured by the sensor may be updated.

  In the information processing apparatus having this functional configuration, for example, the image conversion unit 330 performs image conversion such that vibration during driving is removed from the image when there is a shooting condition without dynamic change such as vibration in the learning image. Do. As described above, by converting the image at the time of recognition in response to the dynamic change of the imaging device, it is possible to perform the recognition of the image with stable accuracy against the shaking of the vehicle and the like.

  Also, with regard to photographing of a learning image, the shaking may be removed from the image by performing the same operation. By this operation, the learning image can be set as an image under a constant imaging condition.

(Modification 2 of the first embodiment)
In the first embodiment, the case of one learning model has been described. Here, image degradation is suppressed and recognition is performed by holding a plurality of learning models and selecting a learning model that most closely matches the image shooting conditions used for learning in each learning model and the image shooting conditions used for recognition processing. The method of improving the performance of

  If the difference between the respective shooting conditions of the image at the time of learning and the image at the time of recognition is large, degradation of the image after conversion may be large. For example, when an image is geometrically transformed, unnatural distortion of the image may occur, or an area to be recognized may be out of the frame by shifting the position of the image. When converting the luminance of an image as described in the second embodiment, noise may be amplified. From the above reasons, it is desirable that the difference between the imaging condition of an image used for recognition processing and the imaging condition of an image used for learning be as small as possible. A plurality of learning models are created so that specific imaging conditions are different. For example, a learning model is prepared in which the brightness of the image used for learning is constant, and the posture of the imaging device is changed and learned using the image.

  So, in this embodiment, an image is image | photographed on several imaging conditions at the time of learning, and the several learning model which learned on each imaging condition is prepared. At the time of recognition processing, the difference between the imaging conditions is reduced by selecting and acquiring the learning model of the imaging conditions closest to the captured image. This improves the recognition performance. Here, processing of the learning model holding unit 340 and the learning model acquisition unit 350, which are differences from the first embodiment, will be described.

  The learning model holding unit 340 holds a plurality of learning models based on the images captured under a plurality of shooting conditions and the shooting conditions. In the first embodiment, one learning model was constructed under one imaging condition. On the other hand, in this modification, the imaging conditions are changed to construct and hold a plurality of learning models. In the present embodiment, the photographing condition is the position and orientation of the imaging apparatus as in the first embodiment. However, other imaging conditions such as luminance may be used. In addition, the learning model holding unit 340 may be located outside the information processing device 300. In that case, the information processing apparatus 300 communicates with the outside to acquire a plurality of learning models.

  The functional configuration and the hardware configuration are the same as in the first embodiment. Here, parts different from the first embodiment in the procedure will be described based on the flowchart of FIG.

  The learning model acquisition unit 350 selects the imaging condition of the learning model constructed based on the imaging condition that most closely matches the imaging condition of the imaging device 110 used for recognition acquired by the imaging condition acquisition unit 320 in S410 in FIG. , Acquired from the learning model holding unit 340. The learning model acquisition unit 350 may acquire a plurality of learning models, and the image conversion unit 330 may select the imaging condition that most closely matches the imaging condition of the imaging device used for recognition acquired by the imaging condition acquisition unit 320. . In the present embodiment, the degree of coincidence of the imaging conditions is the difference between the position and orientation of the imaging device. For example, a learning model is selected such that the difference between the parameters indicating the position and orientation of the imaging device at the time of learning and at the time of recognition is minimized. Specifically, the degree of coincidence may be calculated by looking at the difference in height of the imaging apparatus and the difference in posture. When the luminance of an image is used as a shooting condition, a learning model in which the luminance of the entire image or the feature portion is closest is selected. Alternatively, it is possible to select a learning model to which external factors such as shooting environment such as outdoors and indoors and time such as morning, day and night most apply. Note that S420 and subsequent steps are performed in the same procedure as the first embodiment.

  As described above, by holding a plurality of learning models and selecting a learning model that most closely matches the shooting conditions of the image used for recognition processing, it is possible to suppress image deterioration in image conversion, and as a result, image recognition Performance can be improved.

Second Embodiment
In the present embodiment, attention is paid to the difference between the imaging condition of an image used for learning and the brightness of the imaging condition of an image used for recognition processing using a learning result.

  First, the difference in the shooting condition of the brightness of the image arises from the difference in the brightness around the image pickup apparatus, the sensitivity of the sensor unique to the image pickup apparatus, and the like. In the case of imaging outdoors, the brightness of the image also differs depending on the time zone in which the image was taken and the weather. For example, when the imaging device is installed in a vehicle as in the first embodiment, the exposure may change in the daytime and in the night even at the same place, and the brightness may differ in a tunnel or a parking lot while traveling. . Moreover, even when imaging indoors, it is difficult to image under the same illumination conditions as at the time of learning. For example, a mobile object such as a working robot moving in a factory or a warehouse moves in and out of a space with different brightness, so that the brightness of the surroundings varies depending on the work place. Depending on the size of the moving object, it may pass under the shelf or desk, so the lighting conditions are different even in the same space. In order to correct such a difference, in the present embodiment, the brightness of the image of the image pickup device mounted on the vehicle used for learning and the brightness of the image of the image pickup device mounted on the vehicle used for recognition processing using the learning result Based on the difference, the brightness of the image used for the recognition process is changed. Then, it approaches the shooting conditions of the image used for learning. This improves the accuracy of recognition.

  First, the hardware configuration in the present embodiment is the same as that in the first embodiment. Next, the functional configuration of the information processing apparatus of the present embodiment will be described using FIG. The module configuration of the present embodiment is the same as the module configuration of the first embodiment. However, the processing of the photographing condition acquisition unit 320 and the processing of the image conversion unit 330 differ in the processing content, with the photographing condition as the brightness of the image. Hereinafter, differences from the first embodiment will be described.

  The imaging condition acquisition unit 320 acquires imaging conditions of an image used for image recognition and imaging conditions of an image used for learning. That is, information on the brightness of the imaging device 110 and information on the brightness of the imaging device 210 at the time of learning of FIG. 1 attached to a different vehicle used for learning are acquired as shooting conditions. In the present embodiment, the information on brightness, which is a shooting condition, is a degree of brightness of an image captured by an imaging device mounted on a vehicle. Furthermore, the degree of brightness is an average value of luminance values of the image. In addition, it is assumed that the average value of the luminance values of the image captured by the captured image used for learning is calculated in advance. Not only luminance information obtained from an image but also external information may be taken as a variable related to the degree of brightness. As a specific example, it deals quantitatively with information such as time, weather, intensity of illumination, number of illumination, color of illumination. Further, measurement information obtained by measuring the ambient brightness with a light sensor or the like may be used as the information on brightness.

  The image conversion unit 330 performs image conversion based on the imaging conditions of the image captured at the time of image recognition and the imaging conditions of the image used at the time of learning. That is, based on the imaging condition of the imaging device 110 acquired by the imaging condition acquisition unit 320 and the imaging condition of the imaging device 210 at the time of learning, conversion of luminance value is added to the image acquired by the image acquisition unit 310. In the present embodiment, the photographing condition is a degree of brightness of the image, and the conversion of the luminance value of the image is performed based on the ratio. Note that the imaging condition acquisition unit 220 and the image conversion unit 330 in the present embodiment acquire the degree of brightness of the entire image as the imaging condition, and convert the luminance value of the image based on the ratio. Here, the degree of brightness acquired by the shooting condition acquiring unit 220 may focus on the entire image or may divide the image into several parts and measure the brightness of each part. Similarly, in the conversion of the luminance value in the image conversion unit 330, the luminance value may be converted according to the degree of brightness of each part of the image. Further, the conversion may be performed together with the geometric conversion and other conversions performed in the first embodiment, not only the luminance conversion.

  Next, the processing procedure of this embodiment will be described. Although the order of the processing is the same as that of the first embodiment, the processing contents of the imaging condition acquisition unit 320 and the image conversion unit 330 in S430 and S440 are different. Here, the contents of the processing in S430 and S440 will be described.

  In S430, the imaging condition acquisition unit 320 acquires imaging conditions of the image captured by the imaging device 110 and imaging conditions of the image used for learning in the learning model. That is, the degree of brightness V1 of the image taken by the imaging device 210 at the time of learning in FIG. 1 attached to different vehicles that take pictures of the image used for learning, and the degree of brightness of the image taken by the imaging device 110 Get V2. First, the degree of brightness V2 of the image captured by the imaging device 110 is calculated by obtaining the average value of the luminance values of the image acquired by the image acquisition unit 310. Next, as the degree of brightness V1 at the time of learning, an average value of luminance values calculated in advance is acquired. Further, the ratio of the degree of brightness is defined as S = V1 / V2.

  In addition to this, the brightness may be calculated based on, for example, the season or time. Since the brightness of the environment is correlated with the time, for example, S = 1 for day, S = 2 for evening, S = 10 for night, etc., the ratio of the degree of brightness according to the expected brightness of the environment You may set it. Alternatively, brightness information measured by the sensor 108 may be used.

  In S440, the image conversion unit 330 uses the image acquisition unit 210 based on the degree of brightness of the image of the imaging device 110 acquired by the imaging condition acquisition unit 320 and the degree of brightness of the image of the imaging device 210 at the time of learning. Add luminance conversion to the image acquired in.

  Here, the luminance value of each pixel of the image acquired by the image acquisition unit 210 is converted based on the ratio S of the degree of brightness of the two imaging conditions. Specifically, the luminance value of each pixel is multiplied by S. By this operation, it is possible to make the photographing condition about the brightness of the image close to the photographing condition of the image at the time of learning. The image may be converted to correct a plurality of shooting conditions. For example, after conversion of shooting conditions regarding position and orientation, conversion of shooting conditions regarding brightness may be performed.

  As described above, in the present embodiment, the brightness value of the image used for the recognition process is focused on the difference between the image shooting condition of the image used for learning and the brightness of the image capturing condition for performing the recognition process using the learning result. To approximate the shooting conditions of the image used for learning. This makes it possible to match the appearance of the image, thereby improving the recognition accuracy.

Third Embodiment
In the present embodiment, an example in which the recognition results in the first and second embodiments are used for automatic driving of a vehicle will be described. Here, the term “automatic driving” as used herein refers to a technology in which an information processing apparatus or system controls the movement of a moving apparatus such as a vehicle or a moving object without basically requiring a human (driving control) operation. When the accuracy of image recognition is improved, it can be applied to distance measurement and marker recognition on a measured scale in automatic driving. Here, as a specific example, the driving control of the vehicle is performed by specifying the current position and orientation of the vehicle based on the recognition result of the captured image. In addition to the vehicle, the moving apparatus can also apply the present embodiment to automatic operation control of a mobile robot such as a drone, a work robot, or a transport robot. The present invention may be applied to a vehicle or robot walking on or traveling on land, a device moving in the air, a device moving on the water, a moving object diving in the water and moving.

  First, the module configuration of the present embodiment will be described. The module configuration of the present embodiment has a position and orientation estimation unit 610, an operation control unit 620, and an actuator unit 630 as shown in FIG. 6 in addition to the module configuration of FIG. 3 in the first and second embodiments. In addition, these modules are mounted on the vehicle 100 as shown in FIG. 7 and perform processing based on the image captured by the imaging device 110. Here, in the present embodiment, as in the first or second embodiment, the image recognition unit 360 outputs a distance image as an output result. Note that the image recognition unit 360 may output obstacle detection, sign recognition, and travel area recognition as an output result as necessary.

  The position and orientation estimation unit 610 estimates the position and orientation of the vehicle 100 (or the imaging device 110) in the coordinate system of the map information that is the coordinates of the destination of the vehicle 100. In the present embodiment, the position of the moving device is estimated based on distance information obtained by converting the distance information recognized by the image recognition unit 360 into coordinates in the map information. As a method of estimating the position or posture of a vehicle or an imaging device from an image or a distance image, known SLAM can be mentioned. In the present embodiment, it is assumed that the position or posture of the vehicle 100 is estimated based on the distance image output by the image recognition unit 360. Here, assuming that the relative position-posture relationship between the vehicle 100 and the imaging device 110 is known, the position and orientation of the vehicle 100 can be calculated from the position and orientation of the imaging device 110. That is, the position and orientation estimation unit 610 can estimate the position and orientation of the vehicle 100. A position sensor or attitude sensor such as a GPS or a gyro sensor may be separately mounted, and the position / posture of the vehicle 100 in world coordinates may be acquired based on the measurement information. Alternatively, the position or posture of the vehicle 100 may be estimated from an image captured by the imaging device 110 or distance information measured by the sensor 108. The position and orientation estimation unit 610 may be included in the information processing device 300 or may be incorporated in the moving device body or the operation control device of a vehicle. It is assumed that the vehicle 100 stores map information in the storage unit in advance, or that the communication unit can acquire map information of the surroundings.

  The operation control unit 620 calculates a control value for moving the moving device to the destination. That is, based on at least one of the estimation result of the position and orientation of the vehicle 100 in the position and orientation estimation unit 610 and the recognition result using the learning model of the image recognition unit 360, a control value such as a direction to move the vehicle and acceleration is calculated. . Alternatively, a control parameter corresponding to an operation by which a person depresses the accelerator or an operation to rotate the steering wheel is calculated. Here, as a specific method of calculating a control value from the position and orientation of a vehicle, an example of automatically driving the vehicle to a destination set on a map will be described. The actuator unit 630 performs driving by moving each mechanism (for example, the torque or the direction of the wheel) of the vehicle based on the control value output from the operation control unit 620.

  FIG. 8 is a flowchart showing the processing procedure of the information processing system. Hereinafter, differences from the first embodiment will be mainly described. As initial setting, GPS information or the like is obtained by using GPS or the like to indicate coordinates of the current position of the vehicle and a destination. The processing of the position and orientation estimation unit 610, which will be described later, is assumed to estimate the position and orientation based on the coordinates. In S810, the imaging device 110 captures an image of the area around the vehicle 100. In S820, the image acquisition unit 310 of the information processing device 300 receives the processing target image captured in S810, and acquires the processing target image. In S830, the learning model acquisition unit 350 acquires the learning model and the imaging conditions of the image at the time of learning. The learning model acquisition unit 350 may acquire a plurality of learning models, and the image conversion unit 330 may select the imaging condition that most closely matches the imaging condition of the imaging device used for recognition acquired by the imaging condition acquisition unit 320. . At this time, a suitable learning model may be acquired from the plurality of learning models held by the learning model holding unit based on the information of the imaging conditions acquired in S840. In S840, the imaging condition acquisition unit 320 acquires imaging conditions of the image captured by the imaging device 110 and imaging conditions of the image used for learning in the learning model. In S850, the image conversion unit 330 converts the image captured by the imaging device 110 so that the imaging condition of the processing target image approaches the imaging condition of the image used for learning. In S860, the image recognition unit 360 receives, as an input, the image converted by the image conversion unit 330, and outputs distance information corresponding to the pixels or the area of the image captured by the imaging device 110 as an output result. This distance information is depth information from the imaging device, and represents the distance to a surrounding object at camera coordinates with the imaging device as the origin.

  In S870, position and orientation estimation unit 610 performs self-position estimation of vehicle 100 based on the output result obtained in S860. The distance information which is the output result of S850 is converted from camera coordinates to map coordinates or world coordinates, and the position of the vehicle 100 in the map information is estimated. As a method of self-position estimation, SLAM or another method may be used.

  In S880, the operation control unit 620 in the mobile device calculates a route from the start point to the destination based on the position of the vehicle 100 estimated by the position and orientation estimation unit 610 and the map information. In the vehicle 100, it is assumed that map information indicating coordinates of a destination is stored in the storage unit in advance, or that the communication unit can acquire maps of the surroundings. In addition, the route may be searched while creating a map of the surrounding area at any time using the technique of SLAM (SIMULTANEOUS LOCALIZATION AND MAPPING). Next, a control value for following the route is calculated, and automatic driving of the vehicle is performed via the actuator unit 630. The position and orientation estimation unit 610 always estimates the position and orientation of the vehicle that changes during automatic driving, and updates the control value to realize highly accurate automatic driving. Further, the position and orientation estimation unit 610 detects the presence of bumps and obstacles present on the road based on the distance image output by the information processing device 300, and the operation control unit 620 performs control to stop the speed of the vehicle or stop. Control can be selected automatically. Specifically, the acceleration and the traveling direction are calculated as control values. The actuator unit 630 performs driving by moving each mechanism (for example, the torque or the direction of the wheel) of the vehicle based on the control value output from the operation control unit 620.

  In S890, the operation control unit 620 ends the system when the vehicle 100 arrives at the destination. When the distance between the destination and the vehicle 100 is equal to or less than a predetermined value in S870, the position and orientation estimation unit 610 determines that the driving control unit 620 has arrived at the destination. If it has not arrived at the destination, the process returns to S810. Other than this, a signal of system termination may be instructed by a human operation.

  As described above, in the present embodiment, the current position and orientation of the vehicle can be identified with high accuracy based on the recognition result of the photographed image. Thereby, the accuracy can be improved also in the control of the vehicle based on the recognition result.

(Other embodiments)
In the first and second embodiments, a method has been described in which the image conversion is performed by focusing on the position and orientation and the brightness of the imaging device as the imaging conditions. As the shooting conditions, conversion may be performed by paying attention to the zoom value of the image, the angle of view, and the shooting conditions for the color tone other than this.

  For example, when focusing on the zoom value, the imaging conditions can be matched by enlarging or reducing the image used for the recognition processing in accordance with the zoom value of the image used for learning. In addition, in the case of focusing on the color, the photographing condition can be matched by acquiring the ratio of RGB values as the color of the image used for learning and changing the ratio of the RGB values of the image used for recognition processing. In addition, when there is a difference between channels as a sensor of an image used for learning and an image used for recognition processing, a conversion for correcting the difference (such as a color image and a monochrome gray-scale image, an infrared image) may be added. The angle of view of the imaging device 110 used for recognition processing may be set wider than the angle of view of the imaging device 210 used for learning. For example, when focusing on the angle of view as the difference in imaging conditions, the scale of the captured image is converted. By widening the angle of view, by shifting the position of the image generated when converting the position or angle of the image in the image conversion unit 330, it is possible to suppress a missing portion in which the area to be recognized is out of the frame. it can.

  The present invention is also realized by performing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network for data communication or various storage media. Then, the computer (or CPU or MPU or the like) of the system or apparatus reads out and executes the program. Alternatively, the program may be provided by being recorded on a computer readable recording medium.

100 Vehicle 110 at the time of recognition execution 200 Vehicle at the time of learning 210 Imager 300 Information processor 310 Image acquisition unit 320 Imaging condition acquisition unit 330 Image conversion unit 340 Learning model holding unit 350 Learning model acquisition unit 360 Image recognition unit

Claims (17)

An information processing apparatus that outputs an output result corresponding to an input captured image based on a learning model,
A first acquisition unit configured to acquire imaging conditions under which an image used when generating a learning model is acquired;
An input unit for inputting a photographed image to be processed from the imaging apparatus;
A second acquisition unit configured to acquire a photographing condition under which the photographed image to be processed is photographed;
Conversion means for converting the photographed image to be processed based on the photographing conditions acquired by the first and second acquisition means;
An information processing apparatus comprising: output means for outputting an output result corresponding to the converted image based on the learning model.   The information processing apparatus according to claim 1, wherein the second acquisition unit updates the imaging condition based on measurement information of a sensor that measures the periphery of the imaging apparatus. The apparatus further comprises a first estimation unit that estimates the position of the imaging device based on the distance information measured by the sensor.
The second acquisition unit updates the imaging condition acquired by the second acquisition unit based on the estimation result of the first estimation unit.
The information processing apparatus according to claim 2, wherein the conversion unit converts the photographed image to be processed based on the updated photographing condition and the photographing condition acquired by the first acquisition unit. A second estimation unit configured to estimate the position of the imaging device based on distance information among the output results;
The information processing apparatus according to claim 1, wherein the second acquisition unit updates the estimation result of the second estimation unit as the photographing condition acquired by the second acquisition unit. The first estimation means estimates the position of the imaging device based on the image captured by the imaging device,
4. The information processing apparatus according to claim 3, wherein the second acquisition unit updates the estimation result of the first estimation unit as the imaging condition acquired by the second acquisition unit. The first estimation means estimates the position of the imaging device based on the measurement information;
4. The information processing apparatus according to claim 3, wherein the second acquisition unit updates the estimation result of the first estimation unit as the imaging condition acquired by the second acquisition unit.   When the conversion means has a plurality of learning models and is learned using images taken under a plurality of different imaging conditions, the imaging condition selected from the plurality of imaging conditions and the second acquisition means The information processing apparatus according to any one of claims 1 to 6, wherein the photographed image of the processing target is converted based on the photographing condition acquired in (4). Learning is performed to output an object included in the input captured image from the captured image input when generating the learning model,
The said output means recognizes and outputs the said object contained in the said converted image based on the said converted image and the said learning model, It is characterized by the above-mentioned. Information processing equipment. The learning is performed to output distance information corresponding to the pixel or the region of the input photographed image from the photographed image input when generating the learning model,
The output means outputs distance information corresponding to a pixel or a region of the converted image based on the converted image and the learning model. Information processor as described. An image acquisition unit that acquires an image captured by an imaging device;
A first imaging condition of an image used for learning in a learning model learned to output at least distance information corresponding to a pixel or a region of the input image from the input image, a position or a posture of the imaging device, A photographing condition acquisition unit that acquires, as a photographing condition, at least one of information regarding brightness when photographing an image, and a second photographing condition in the image;
Conversion means for converting an image captured by the imaging device based on the first imaging condition and the second imaging condition;
An information processing apparatus, comprising: recognition means for outputting at least distance information corresponding to a pixel or a region of the converted image based on the image converted by the conversion means and the learning model. An image acquisition unit that acquires an image captured by an imaging device;
A photographing condition for acquiring a first photographing condition of an image used for learning in a learning model learned to output an output result corresponding to an input image and a second photographing condition of an image photographed by the imaging device Acquisition means,
Conversion means for converting an image captured by the imaging device based on the first imaging condition and the second imaging condition;
An information processing apparatus, comprising: recognition means for outputting an output result corresponding to the image converted by the conversion means based on the learning model. An information processing system comprising: a moving device that moves based on map information indicating a destination, an imaging device mounted on the moving device, and an information processing device,
An information processing apparatus that outputs an output result corresponding to an input captured image based on a learning model is:
A first acquisition unit configured to acquire imaging conditions under which an image used when generating a learning model is acquired;
An input unit for inputting a photographed image to be processed from the imaging apparatus;
A second acquisition unit configured to acquire a photographing condition under which the photographed image to be processed is photographed;
Conversion means for converting the photographed image to be processed based on the photographing conditions acquired by the first and second acquisition means;
And output means for outputting an output result corresponding to the converted image based on the learning model.
The moving device is
Position estimation means for estimating at least the position of the mobile device based on distance information obtained by converting distance information in the output result into coordinates in the map information;
An information processing system comprising: control means for controlling the movement device to a destination based on the position of the movement device and the map information. An information processing system comprising: a moving device that moves based on map information indicating a destination, an imaging device mounted on the moving device, and an information processing device,
The information processing apparatus is
An image acquisition unit that acquires an image captured by the imaging device;
A first imaging condition of an image used for learning in a learning model learned to output at least distance information corresponding to a pixel or a region of the input image from the input image, a position or a posture of the imaging device, A photographing condition acquisition unit that acquires, as a photographing condition, at least one of information regarding brightness when photographing an image, and a second photographing condition in the image;
Conversion means for converting an image captured by the imaging device based on the first imaging condition and the second imaging condition;
Recognition means for outputting distance information which is depth information from the imaging device corresponding to a pixel or a region of the image converted based on the image converted by the image conversion means and the learning model;
The moving device is
Position estimation means for estimating at least the position of the mobile device based on distance information obtained by converting the distance information into coordinates in the map information;
An information processing system comprising: control means for controlling the movement device to a destination based on the position of the movement device and the map information. An information processing method for outputting an output result corresponding to an input photographed image based on a learning model,
A first acquisition step of acquiring imaging conditions under which an image used to generate a learning model is captured;
An input step of inputting a photographed image to be processed from the imaging device;
A second acquisition step of acquiring a photographing condition under which the photographed image to be processed is photographed;
A conversion step of converting the photographed image to be processed based on the photographing conditions acquired by the first and second acquisition means;
An output step of outputting an output result corresponding to the converted image based on the learning model. An image acquisition step of acquiring an image captured by the imaging device;
A photographing condition for acquiring a first photographing condition of an image used for learning in a learning model learned to output an output result corresponding to an input image and a second photographing condition of an image photographed by the imaging device Acquisition process,
A conversion step of converting an image captured by the imaging device based on the first imaging condition and the second imaging condition;
A recognition step of outputting an output result corresponding to the image converted by the conversion means based on the learning model.   The program for functioning a computer as each means with which the information processing apparatus in any one of Claims 1-11 is equipped.   A computer readable storage medium storing the program according to claim 16.

Images