JP2020038605A - Information processing method and information processing system - Google Patents

Abstract

To provide an information processing method capable of improving training efficiency pf each of the models trained by machine learning.SOLUTION: An information processing method includes the steps of: acquiring sending data (S10); determining a synthetic portion which synthesizes recognition target data on the sensing data (S20); generating synthetic data in the synthetic portion by synthesizing recognition target data having characteristics identical to or similar to the characteristics perceived by a human receptor in the sensing data (S30); inputting the synthetic data to the model trained using machine learning so as to recognize the recognition target and acquiring recognition result data (S40); determining whether to perform second determination for determining whether to perform first determination for determining training data of the model on the basis of the synthetic data using at least correct answer data containing the synthetic portion and recognition result data (S50); and performing the first determination when the second determination determines that the first determination is performed (S60).SELECTED DRAWING: Figure 2

Description

  The present disclosure relates to an information processing method and an information processing system.

  In constructing a data set for machine learning (hereinafter also referred to as a training data set or a learning data set) or the like, a sufficient amount of data (hereinafter also referred to as training data or learning data) is prepared. As a method, a method of generating a composite image has been proposed. For example, Patent Literature 1 discloses a simulation system in which a plurality of different types of sensors are used together, and a CG (Computer Graphics) image is generated based on information obtained from these sensors to increase the number of learning samples. Is disclosed. Also, for example, Patent Document 2 discloses an image processing apparatus or the like that uses a difference image between an image obtained by photographing an object having a portion having at least one of color and brightness similar to a background image and a background image as learning data. Have been.

International Publication No. WO2018 / 066351 WO 2017/154630

  In the above related art, the generated training data is not always useful for each model trained by machine learning (hereinafter, also referred to as a training model or a learning model). Therefore, it is difficult to improve the training efficiency of each model with the above-mentioned conventional technology.

  Thus, the present disclosure provides an information processing method and an information processing system that can improve the efficiency of individual training of a model trained by machine learning.

  An information processing method according to a non-limiting, exemplary aspect of the present disclosure obtains sensing data, determines a synthesis part for synthesizing recognition target data on the sensing data, and sets the sensing data to the synthesis part. Generating synthetic data by synthesizing recognition target data having the same or similar characteristics as characteristics perceived by the human sensory organs, and training the synthesized data using machine learning to recognize the recognition target. Inputting the obtained model to obtain recognition result data, and determining whether or not to make a first determination of determining training data of the model based on the composite data. The decision is made using the correct answer data including at least the synthesized part and the recognition result data, and when the first decision is made in the second decision, the first decision is made. Performs constant.

  Note that the above comprehensive or specific aspects may be realized by a recording medium such as a system, an apparatus, a method, an integrated circuit, a computer program or a computer-readable recording disk, and the system, the apparatus, the method, and the integrated circuit. , A computer program and a recording medium. The computer-readable recording medium includes a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). Additional benefits and advantages of one aspect of the disclosure will be apparent from the description and drawings. This benefit and / or advantage can be provided individually by the various aspects and features disclosed herein and in the drawings, and not all are required to achieve one or more of the above.

  According to the information processing method and the like according to the present disclosure, it is possible to improve the efficiency of individual training of a model trained by machine learning.

FIG. 1 is a block diagram illustrating an example of a configuration of the information processing system according to the embodiment. FIG. 2 is a flowchart illustrating an example of the flow of the information processing method according to the embodiment. FIG. 3 is a diagram illustrating a captured image acquired by the image acquisition unit. FIG. 4 is a diagram illustrating a captured image in which an object combining position on an image is determined by the combining position determining unit. FIG. 5 is a diagram illustrating a synthesized image generated by synthesizing an object at an object synthesis position in the synthesized image generation unit. FIG. 6 is a block diagram illustrating an example of a configuration of an information processing system according to the first modification. FIG. 7 is a flowchart illustrating an example of the flow of the information processing method according to the first modification. FIG. 8 is a block diagram illustrating an example of a configuration of an information processing system according to a second modification. FIG. 9 is a flowchart illustrating an example of a flow of an information processing method according to the second modification. FIG. 10 is a block diagram illustrating an example of a configuration of an information processing system according to Modification 3. FIG. 11 is a flowchart illustrating an example of the flow of the information processing method according to the third modification.

  The outline of one embodiment of the present disclosure is as follows.

  An information processing method according to an aspect of the present disclosure obtains sensing data, determines a synthesis portion that synthesizes recognition target data on the sensing data, and, in the synthesis portion, by a human sensory organ that the sensing data has. Generating synthesized data by synthesizing recognition target data having the same or similar characteristics as the perceived characteristics, and inputting the synthesized data to a model trained using machine learning to recognize the recognition target Obtaining recognition result data and determining whether or not to make a first decision to determine training data for the model based on the combined data, at least a second decision to determine whether to make the It is performed using the correct answer data and the recognition result data, and if the first determination is made in the second determination, the first determination is performed. At this time, the feature of the sensing data may be a statistical feature of an element of the sensing data. Further, the feature of the sensing data may be a qualitative feature of the sensing data.

  According to the above aspect, for example, when the sensing data is an image, the coordinates of the combining portion (for example, the object combining position) and the object are combined to combine the recognition target (for example, object) data with the desired combining portion on the image. It is not necessary to newly add an annotation indicating the type or the like to the composite data (here, the composite image). Therefore, the time required for a series of information processing for creating the correct answer data can be reduced. According to the above aspect, in order to synthesize object data at a desired position and size on an image, information such as coordinates of an object synthesis position is input to a learning image (hereinafter, also referred to as a recognition model) of the synthesized image. It can be used as correct answer data in the case. Therefore, by comparing the output data obtained by inputting the synthesized image in which the object is synthesized at the object synthesis position into the learning model with the correct answer data, it is possible to specify the synthesized image with low recognition accuracy in the learning model. . This makes it possible to use the synthesized image or an image similar to the synthesized image as training data of the learning model based on the synthesized image with low recognition accuracy in the learning model. Therefore, the efficiency of individual training of the learning model can be improved. In other words, if data that is not useful for machine learning is mixed, it is difficult for the processing of machine learning to converge, but according to the above aspect, data that is useful for machine learning is specified and used as training data. Machine learning processing is likely to converge. Thereby, the efficiency of individual training of the learning model is improved. For example, the recognition accuracy of the learning model can be improved in a shorter time than when all generated synthetic images are used as training data for the learning model. According to the above aspect, an object having the same or similar feature (that is, visual feature) as a feature (for example, visual feature) perceived by a human sensory organ included in the image is synthesized at the object synthesis position. Therefore, when the image is an image actually captured by a camera or the like (hereinafter, a captured image), a natural composite image close to the captured image can be obtained. Therefore, a learning model trained by machine learning using the synthetic image as training data can obtain recognition accuracy closer to the recognition accuracy when the captured image is used as training data. Note that an object having the same or similar visual feature as an image has, for example, the same or similar color or edge of an image, which is a statistical feature of an element (for example, an image parameter) of the image. It may be an object having the same or similar weather conditions, such as rain or snow, which are qualitative features of the image, the state of the road surface due to the weather conditions, and occlusion. Since the object has the above characteristics, it is easy to adapt to the image. Therefore, a synthesized image generated by synthesizing the object at the object synthesis position is a natural image.

  For example, in the information processing method according to an aspect of the present disclosure, in the first determination, the composite data may be determined as training data of the model.

  According to the above aspect, it is possible to use a synthesized image determined to have low recognition accuracy in the learning model as training data. Therefore, accumulation of data with high recognition accuracy in the learning model, that is, data unnecessary as training data, as training data is suppressed. Therefore, the cost for storing data is reduced. In other words, since images of scenes with low recognition accuracy in the learning model can be accumulated as training data, efficient learning can be performed on scenes with low recognition accuracy. Therefore, the recognition accuracy of the learning model is further improved.

  For example, in the information processing method according to an aspect of the present disclosure, in the first determination, correspondence data having the same or similar feature as the feature of the composite data is determined as training data of the model. Is also good. At this time, the characteristic of the composite data may be a statistical characteristic of an element of the composite data. Further, the characteristic of the composite data may be a qualitative characteristic of the composite data.

  According to the above aspect, since the corresponding image having the same or similar visual feature as the visual feature of the composite image is determined as the training data, an image of a scene with low recognition accuracy in the learning model and a similar image are used. The image of the scene to be used can be used as training data. Therefore, the number and variations of training data for a scene with low recognition accuracy can be efficiently increased. When the corresponding image is a photographed image, the learning effect can be improved as compared with the case where the synthetic image is used as training data. When the visual feature is a statistical feature of an element (for example, an image parameter) of the composite image, the number and variations of the training data can be efficiently increased from a statistical viewpoint. When the visual features are qualitative features of the composite image, the number and variations of training data having features that are difficult to quantify can be efficiently increased.

  For example, in the information processing method according to an aspect of the present disclosure, the sensing data is an image, the recognition target is an object, and the combining portion is an object combining position for combining object data on the image. The synthesized data is a synthesized image generated by synthesizing object data having the same or similar visual characteristics as the visual characteristics of the image at the object synthesis position, and the recognition result data is The first image is object recognition result data obtained by inputting the composite image to the model, and the first determination is to determine training data of the model based on the composite image. The determination may be performed using correct data including at least the object synthesis position and the object recognition result data. For example, the first determination is to determine a corresponding image having the same or similar visual feature as the visual feature of the composite image as training data of the model, and the visual determination of the composite image is The feature may be an aspect of the object in the composite image, and the visual feature of the corresponding image may be an aspect of a corresponding object having the same or similar attribute as the attribute of the object. In this case, the aspect may be a position of the object on the composite image. Further, the aspect may be a posture of the object.

  According to the above aspect, since the visual feature of the composite image is the aspect of the object in the composite image, for example, the object in the learning model depends on the aspect of the object such as the position of the object on the composite image or the posture of the object. The training data is determined based on the synthesized image determined to have low recognition accuracy. Thus, an image of a scene with low recognition accuracy in the learning model and an image of a scene similar to the image can be used as training data. Therefore, the number and variations of training data for a scene with low recognition accuracy can be efficiently increased. A recognition model constructed using such training data has improved accuracy in recognizing an object from an image.

  For example, in the information processing method according to an aspect of the present disclosure, the combining portion may further include the size of the object data combined on the image.

  According to the above aspect, it is possible to obtain combined data with less discomfort for the image.

  For example, in the information processing method according to an aspect of the present disclosure, data having the same or similar feature as the feature of the combined data is selected or generated as the corresponding data from sensing data different from the combined data. May be.

  According to the above aspect, the captured image can be used as training data. Therefore, a higher learning effect can be obtained as compared with the case where the synthesized image is used as training data. Note that the selection of the captured image may be to determine whether or not to record the image based on a predetermined condition each time the image is acquired, and to select an image based on a predetermined condition from the acquired images. May be sampled, or a captured image satisfying a predetermined condition may be searched for and extracted from a captured image stored in a memory or a database. According to the above aspect, the corresponding image can be generated from the captured image. Specifically, an image of a scene with low recognition accuracy in the recognition model and an image similar to the scene can be generated from the captured image. Accordingly, the corresponding image can be generated even when the captured image cannot be used as the corresponding image as it is, so that the number and variations of the training data can be easily increased.

  For example, in the information processing method according to an aspect of the present disclosure, the recognition target data may be combined with the combining portion using a GAN (Generative Adversary Network) model.

  According to the above aspect, it is possible to obtain a more natural synthesized image close to a captured image while synthesizing a desired object at a desired position. By using such a synthesized image as training data, it is possible to improve the recognition accuracy of the object in the learning model.

  For example, in the information processing method according to an aspect of the present disclosure, when the first determination is made and the second determination is made, the user of the learning model may be notified. At this time, the notification may be a notification regarding a request for training of the model using the determined training data. In the information processing method according to an aspect of the present disclosure, the model may be further trained using the determined training data, and the notification may be a notification regarding completion of the training.

  According to the above aspect, when the training data of the learning model is determined based on the composite image, the user of the learning model is notified, so that the user grasps that there is a scene in which the learning model has difficulty in recognizing the object. can do. In addition, when the notification is a notification regarding a request for training of a learning model, the user can determine a timing for performing training of the learning model. In addition, when the notification is a notification regarding the completion of the training, the user can know that the learning model has been updated by the training.

  In addition, the information processing system according to an aspect of the present disclosure includes a first acquisition unit that acquires sensing data, a first determination unit that determines a combining unit that combines recognition target data on the sensing data, and the combining unit. A generating unit that generates synthetic data by synthesizing recognition target data having the same or similar characteristics as characteristics perceived by a human sensory organ included in the sensing data, and recognizing the recognition target with the synthetic data. A second acquisition unit that acquires recognition result data by inputting to a model trained using machine learning, and a first determination that is to determine training data of the model based on the combined data. A second decision to determine whether or not to perform is performed using the correct answer data including at least the synthesized part and the recognition result data, and the first determination is performed when the first determination is performed. If it is determined in the determination of 2, and a second determination unit to perform the first determination.

  According to the above aspect, for example, when the sensing data is an image, the coordinates of the combining portion (for example, the object combining position) and the object are combined to combine the recognition target (for example, object) data with the desired combining portion on the image. It is not necessary to newly add an annotation indicating the type or the like to the composite data (here, the composite image). Therefore, the time required for a series of information processing for creating the correct answer data can be reduced. According to the above aspect, in order to synthesize object data at a desired position and size on an image, information such as coordinates of an object synthesis position is input to a learning image (hereinafter, also referred to as a recognition model) of the synthesized image. It can be used as correct answer data in the case. Therefore, by comparing the output data obtained by inputting the synthesized image in which the object is synthesized at the object synthesis position into the learning model with the correct answer data, it is possible to specify the synthesized image with low recognition accuracy in the learning model. . This makes it possible to use the synthesized image or an image similar to the synthesized image as training data of the learning model based on the synthesized image with low recognition accuracy in the learning model. Therefore, the efficiency of individual training of the learning model can be improved. In other words, if data that is not useful for machine learning is mixed, it is difficult for the processing of machine learning to converge, but according to the present embodiment, data that is useful for machine learning is specified and used as training data. Therefore, the machine learning process is easily converged. Thereby, the efficiency of individual training of the learning model is improved. For example, the recognition accuracy of the learning model can be improved more efficiently in a shorter time than when all the generated synthesized images are used as training data for the learning model. According to the above aspect, in order to combine an object having the same or similar visual feature as the visual feature of the image at the object combining position, if the image is a captured image actually captured by a camera or the like, Thus, it is possible to obtain a natural synthesized image close to the captured image. Therefore, a learning model trained by machine learning using the synthetic image as training data can obtain recognition accuracy closer to the recognition accuracy when the captured image is used as training data.

  Note that the above comprehensive or specific aspects may be realized by a recording medium such as a system, an apparatus, a method, an integrated circuit, a computer program or a computer-readable recording disk, and the system, the apparatus, the method, and the integrated circuit. , A computer program and a recording medium. The computer-readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM.

  Hereinafter, an information processing method and an information processing system according to an embodiment of the present disclosure will be specifically described with reference to the drawings. Each of the embodiments described below shows a comprehensive or specific example. Numerical values, shapes, components, arrangement positions and connection forms of components, steps (processes), order of steps, and the like shown in the following embodiments are examples, and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components not described in the independent claims indicating the highest concept are described as arbitrary components. In the following description of the embodiments, expressions with “substantially” such as substantially parallel or substantially orthogonal may be used. For example, substantially parallel means not only completely parallel, but also substantially parallel, that is, including, for example, a difference of about several percent. The same applies to expressions with other “abbreviations”. In the following description of the embodiments, the similarity of two elements means that, for example, a half or more part or a main part is the same between two elements, or that two elements are common. It means having properties. Each drawing is a schematic diagram and is not necessarily strictly illustrated. Further, in each of the drawings, substantially the same components are denoted by the same reference numerals, and redundant description may be omitted or simplified in some cases.

(Embodiment)
[Configuration and Operation of Information Processing System According to Embodiment]
The configuration and operation of the information processing system according to the embodiment will be described with reference to FIGS. FIG. 1 is a block diagram illustrating an example of a configuration of an information processing system 100 according to the present embodiment. FIG. 2 is a flowchart illustrating a flow of the information processing method according to the present embodiment.

  As shown in FIG. 1, the information processing system 100 includes an image acquisition unit 10, a combination position determination unit 20 that determines an object combination position on an image, a combination image generation unit 30, and a learning model (hereinafter, referred to as a learning model). An output data obtaining unit 40 obtains output data obtained by inputting to the recognition model, and a determination unit 50 that determines learning data of the recognition model.

  The image acquisition unit 10 is an example of a first acquisition unit that acquires sensing data. For example, the sensing data is an image. The combining position determining unit 20 is an example of a first determining unit that determines a combining unit (here, an object combining position) that combines recognition target data on sensing data. For example, the recognition target is an object. The synthetic image generating unit 30 is an example of a generating unit that generates synthetic data by synthesizing recognition target data having the same or similar characteristics as characteristics perceived by a human sensory organ included in sensing data in a synthetic part. is there. For example, the combining portion is an object combining position at which the object data on the image is combined, and the combining data is a combined image. Further, the output data acquisition unit 40 inputs the synthesized data to a model trained using machine learning to recognize the recognition target (hereinafter, also referred to as a recognition model or a learning model) and obtains recognition result data. It is an example of a 2nd acquisition part. For example, the recognition result data is object recognition result data obtained by inputting a composite image to a recognition model.

  As shown in FIGS. 1 and 2, the image acquisition unit 10 acquires a captured image (S10 in FIG. 2). Acquiring may be, for example, acquiring an image photographed by an imaging device (hereinafter, also referred to as a photographed image), or acquiring a photographed image by photographing the image. In the former case, the image acquiring unit 10 is, for example, a receiving unit, and receives an image captured by the imaging device through communication. In the latter case, the image acquisition unit 10 is, for example, an imaging unit such as a camera, and captures an image.

  The combining position determining unit 20 determines an object combining position on the image acquired by the image acquiring unit 10 at which the object data is combined (S20 in FIG. 2). Objects to be synthesized with the image include, for example, movable objects such as people, animals and vehicles, and immovable objects such as plants and road accessories. The position on the image where the object data is combined is arbitrarily determined. The object combining position may further include the size of the object data combined on the image.

  The synthetic image generating unit 30 generates a synthetic image by synthesizing object data having the same or similar visual characteristics as the visual characteristics of the captured image at the object synthesizing position (S30 in FIG. 2). The visual characteristics of the captured image are statistical characteristics of elements of the captured image (hereinafter, also referred to as image parameters). The statistical characteristics of the image parameters include, for example, the color tone, brightness, and edge of the image. Further, the visual features of the captured image are qualitative features of the captured image. The qualitative feature of an image is a feature of an image that is difficult to quantify. No.

  The synthetic image generation unit 30 synthesizes an object at an object synthesis position using, for example, a GAN (Generative Adversary Network) model. Thereby, in the obtained composite image, the color tone and brightness of the composite object and the background, that is, the statistical characteristics of the image parameters of the composite object and the captured image on the captured image are the same or similar. . Therefore, a more natural synthesized image close to the captured image can be obtained while synthesizing a desired object at a desired position. Note that a method of generating a composite image using GAN will be described later. GAN is an example, and the method of generating a composite image is not particularly limited. The method of generating the composite image may be any method that can obtain a composite image closer to a more natural captured image.

  The output data acquisition unit 40 acquires object recognition result data (that is, output data of the recognition model) obtained by inputting the composite image obtained by the composite image generation unit 30 to the recognition model (S40 in FIG. 2). .

  The decision unit 50 makes a second decision, which is to decide whether or not to make the first decision, using the correct answer data and the output data. More specifically, the determination unit 50 determines at least a second determination to determine whether to make a first determination to determine training data of the recognition model based on the composite image, This is performed using the correct data including the object combining position and the output data (S50 in FIG. 2). The correct answer data includes, for example, information indicating the coordinates of the area of the object combining position, the type of the object, the posture of the object, and the like. The determination unit 50 determines the accuracy of recognizing the object synthesized at the object synthesis position (hereinafter, object recognition accuracy) based on the magnitude of the difference between the correct answer data and the output data. The deciding unit 50 makes the first decision when the first decision is made in the second decision (S60 in FIG. 2). More specifically, when the recognition accuracy of the object in the recognition model is lower than a predetermined threshold, the determination unit 50 trains the synthesized image input to the recognition model and an image identical or similar to the synthesized image to the recognition model. Determined as data for use.

  As described above, the information processing system 100 constructs a recognition model in which the recognition accuracy of the object is improved by causing the recognition model to learn using the training data. In the present embodiment, the recognition model is a machine learning model using a neural network such as Deep Learning (deep learning), but may be another learning model. For example, another learning model may be a machine learning model using Support Vector Machine, Boosting, Random Forest, or Genetic Programming.

  Hereinafter, with reference to FIG. 3 to FIG. 5, an example of a procedure from acquisition of a captured image to generation of a composite image in the information processing system 100 according to the present embodiment will be described. FIG. 3 is a diagram illustrating a captured image obtained by the image obtaining unit 10. FIG. 4 is a diagram illustrating a captured image in which the object combining position on the image is determined by the combining position determination unit 20. FIG. 5 is a diagram illustrating a combined image generated by combining the object at the object combining position in the combined image generation unit 30. Here, an example in which an object is synthesized at an object synthesis position using a GAN model will be described.

  The image acquisition unit 10 acquires the captured image shown in FIG. This captured image is, for example, an image captured by a vehicle-mounted camera.

  Next, as shown in FIG. 4, the combining position determination unit 20 determines an object combining position A and an object combining position B at which an object is combined on the captured image. The object combining position A and the object combining position B are positions arbitrarily determined. In the GAN model, noise is generated at the object combining position A and the object combining position B on the captured image.

  Next, as illustrated in FIG. 5, the combined image generation unit 30 combines images having different generators at each of the object combination position A and the object combination position B where noise has occurred. The images having different generators include, for example, images of people having different genders, ages, clothes, and postures, images of people driving vehicles such as bicycles, and images of people standing near a car or the like. Is mentioned. A predetermined object is synthesized at a predetermined object synthesis position based on these generators. For example, as shown in FIG. 5, a pedestrian A1 is synthesized at the object synthesis position A, and a person B1 driving a bicycle is synthesized at the object synthesis position B.

  In the GAN model, it is determined whether or not the object synthesized at the object synthesis position can be recognized as a person, and whether the object synthesized at the object synthesis position is familiar with the background. For example, even if it is determined that the synthesized object is recognized as a person, if it is determined that the object is not familiar with the background, the generator is adjusted and the object is synthesized again at the object synthesis position. As a result, a natural image close to the original captured image can be obtained as the entire composite image.

  As described above, in the present embodiment, since an object is synthesized at a desired object synthesis position on a captured image, it is not necessary to add an annotation indicating the coordinates of the object synthesis position and the type of the object to the synthesized image. Therefore, the time required for a series of information processing for creating the correct answer data can be reduced. Further, in the present embodiment, in order to synthesize an object at a desired object synthesis position on a captured image, information such as coordinates of the object synthesis position is input to a learning model (also referred to as a recognition model). It can be used as correct answer data. Therefore, by comparing the output data obtained by inputting the synthesized image in which the object is synthesized at the object synthesis position into the learning model with the correct answer data, it is possible to specify the synthesized image with low recognition accuracy in the learning model. . This makes it possible to use the synthesized image or an image similar to the synthesized image as training data of the learning model based on the synthesized image with low recognition accuracy in the learning model. Therefore, the training efficiency of each of the learning models can be improved as compared with the case where all the generated synthetic images are used as training data of the learning model. In other words, if data that is not useful for machine learning is mixed, it is difficult for the processing of machine learning to converge, but according to the present embodiment, data that is useful for machine learning is specified and used as training data. Therefore, the machine learning process is easily converged. Thereby, the efficiency of individual training of the learning model is improved. For example, the recognition accuracy of the learning model can be improved in a shorter time.

  Further, in the present embodiment, an object having the same or similar feature (that is, visual feature) as a feature (here, visual feature) perceived by a human sensory organ of an image is synthesized at the object synthesis position. Therefore, when the image is a photographed image actually photographed by a camera or the like, a natural composite image close to the photographed image can be obtained. Therefore, the learning model that has performed learning using the composite image as training data can obtain recognition accuracy closer to the recognition accuracy when the captured image is used as training data.

  As described above, the information processing system 100 includes the image acquisition unit 10 that acquires an image, the composition position determination unit 20 that determines the object composition position on the image, and the same visual features of the image at the object composition position. Alternatively, a synthesized image generating unit 30 that generates a synthesized image by synthesizing objects having similar visual characteristics, and an output data obtaining unit that obtains output data of a learning model obtained by inputting the synthesized image to the learning model 40. Correct answer data including at least an object combining position and output a second decision to determine whether or not to make a first decision to determine training data of a learning model based on the combined image. A determination unit that performs the first determination when the determination is made in the second determination that the first determination is performed using the data. At this time, the visual characteristics of the image are statistical characteristics of image parameters of the image. The visual features of the image are qualitative features of the image.

  Further, the information processing method according to the present embodiment obtains an image (S10), determines an object combining position on the image (S20), and sets the object combining position to be the same as or similar to the visual feature of the image. A synthetic image is generated by synthesizing an object having a visual feature (S30), output data of a learning model obtained by inputting the synthetic image to the learning model is obtained (S40), and learning is performed based on the synthetic image. A second decision to decide whether to make a first decision to decide training data of the model is made by using the correct answer data including at least the object synthesis position and the output data (S50). If the first decision is made in the second decision, the first decision is made (S60).

  Accordingly, the object is synthesized at the desired object synthesis position on the image, and thus information such as the coordinates of the object synthesis position can be used as correct data of the learning model. Therefore, by comparing the output data obtained by inputting the synthesized image in which the object is synthesized at the object synthesis position into the learning model with the correct answer data, it is possible to specify the synthesized image with low recognition accuracy in the learning model. . This makes it possible to use the synthesized image or an image similar to the synthesized image as training data of the learning model based on the synthesized image with low recognition accuracy in the learning model. Therefore, the efficiency of individual training of the learning model can be improved. In other words, if data that is not useful for machine learning is mixed, it is difficult for the processing of machine learning to converge, but according to the present embodiment, data that is useful for machine learning is specified and used as training data. Therefore, the machine learning process is easily converged. Thereby, the efficiency of individual training of the learning model is improved. For example, the recognition accuracy of the learning model can be improved more efficiently in a shorter time than when all the synthesized images are used as training data for the learning model. Further, in the present embodiment, in order to synthesize an object having the same or similar visual characteristics as the visual characteristics of the image at the object synthesis position, if the image is an image actually captured by a camera or the like, A natural synthesized image close to the captured image can be obtained. Therefore, the learning model that has performed learning using the composite image as training data can obtain recognition accuracy closer to the recognition accuracy when the captured image is used as training data.

  Note that an object having the same or similar visual feature as an image has, for example, an object having the same or similar color or edge of an image, which is a statistical feature of image parameters of the image. It may be an object having the same or similar weather conditions, such as rain or snow, which are qualitative features of the image, the state of the road surface due to the weather conditions, and occlusion. Since the object has the above characteristics, it is easy to adapt to the image. Therefore, a synthesized image generated by synthesizing the object at the object synthesis position is a natural image.

(Modification 1)
[Configuration of Information Processing System According to Modification 1]
An information processing system according to a first modification of the embodiment will be described with reference to FIG. FIG. 6 is a block diagram illustrating an example of a configuration of an information processing system 100 according to the first modification.

  In the information processing system 100 according to the embodiment, an example has been described in which the image acquisition unit 10 may be either a reception unit that acquires an image or an imaging unit that captures an image. An example in which the image acquiring unit 10 is a receiving unit that receives a captured image will be described.

  The information processing system 100 according to the first modification includes a recognition processing unit 200 including an imaging unit 210 and a recognition unit 220, and a recognition model updating unit 300.

  The information processing system 100 according to Modification 1 synthesizes an image having the same or similar visual feature as an image captured by the imaging unit 210 (hereinafter, also referred to as a captured image) by synthesizing the same. A synthetic image is generated, an annotation is given to an object synthesized in the synthetic image, and training data for constructing a recognition model is determined. Further, the information processing system 100 constructs a recognition model for detecting an object from the image using the training data determined based on the composite image. To construct the recognition model, a learning model described below is applied. The training data is data used by the recognition model for learning (in other words, for the recognition model to be trained using machine learning). The training data includes a synthesized image, information such as the type and motion of the object in the synthesized image, and information such as the position and area of the object.

  The recognition processing unit 200 receives the recognition model constructed by the training by the recognition model updating unit 300 by the recognition model receiving unit 3 of the recognition unit 220. The recognition model received by the recognition model receiving unit 3 is input to the recognition model updating unit 4 and updated. When the recognition model is updated, the update information presenting unit 5 presents a notification that the recognition model has been updated. The presentation of the notification may be sound or may be displayed as an image on a screen. Further, the recognition unit 220 recognizes an object included in the image captured by the imaging unit 210. Note that the recognition processing unit 200 may output the recognition result of the object as a voice or an image to notify the user.

  Hereinafter, a description will be given on the assumption that the recognition processing unit 200 is mounted on a moving object, specifically, a car, and the recognition model updating unit 300 is mounted on a server located at a position away from the car. The recognition processing unit 200 of the vehicle and the server are connected via wireless communication, and wirelessly communicate with each other via a communication network such as the Internet, for example. The recognition processing unit 200 and the recognition model updating unit 300 transmit and receive information via wireless communication. For the wireless communication, a wireless LAN (Local Area Network) such as Wi-Fi (registered trademark) (Wireless Fidelity) may be applied, or other wireless communication may be applied. The server may be an information processing device such as a computer. The server may include one or more servers, and may constitute a cloud system.

  The information processing system 100 may include a wireless communication device such as a communication circuit, or may use a wireless communication device included in a server. The recognition processing unit 200 may include a wireless communication device such as a communication circuit, or may use a wireless communication device included in an automobile. Note that the recognition processing unit 200 and the recognition model updating unit 300 may be connected via wired communication without being connected via wireless communication, and exchange information with each other via a recording medium such as a nonvolatile memory. May be.

  If the computer mounted on the vehicle can process, the recognition model updating unit 300 may be mounted on the vehicle. In this case, the recognition model updating unit 300 and the recognition processing unit 200 may be integrated. Then, the recognition model updating unit 300 may exchange information with the outside of the vehicle via wireless communication, wired communication, or a recording medium.

  Further, a detailed configuration of the recognition processing unit 200 and the recognition model updating unit 300 of the information processing system 100 according to the first modification will be described with reference to FIG. Hereinafter, an example will be described in which the recognition model updating unit 300 uses the GAN model to generate a composite image.

  In the information processing system 100 according to the first modification, the recognition processing unit 200 includes an imaging unit 210 and a recognition unit 220.

  The imaging unit 210 is, for example, a camera, and includes an image imaging unit 1 and an image transmission unit 2. The image captured by the imaging unit 210 is transmitted to the image acquisition unit 110 of the recognition model updating unit 300 via the image transmission unit 2.

  The recognition unit 220 recognizes, for example, an object such as a person included in an image captured by the imaging unit 210. The recognition unit 220 includes a recognition model receiving unit 3, a recognition model updating unit 4, and an update information presenting unit 5. The recognition model receiving unit 3 receives the recognition model updated by the recognition model updating unit 300 and outputs the received recognition model to the recognition model updating unit 4. The recognition model updating unit 4 updates the recognition model by storing the recognition model output from the recognition model receiving unit 3. The update information presenting unit 5 may be configured by a display and / or a speaker, and notifies the user of the recognition model when the first determination is made and the second determination is made. For example, when a predetermined amount of training data is stored in the training data holding unit 160, the update information presentation unit 5 notifies a request for training of a recognition model using the determined training data. In addition, for example, when the training of the recognition model using the determined training data is performed in the training unit 170, the update information presentation unit 5 notifies the completion of the training. In addition, the update information presenting unit 5 may present to the user that the recognition model stored in the recognition unit 220 has been updated to a trained recognition model. Further, the update information presenting unit 5 may present the user with update information such as a difference between the updated recognition model and the pre-update recognition model, and an effect obtained by being updated. Note that the display may be configured by a display panel such as a liquid crystal panel, an organic or inorganic EL (Electro Luminescence).

  The recognition model update unit 300 includes an image acquisition unit 110, a sampling unit 112, a synthesis position setting unit 120, an image synthesis unit 130, a detection processing unit 140, a data use determination unit 150, a training unit 170, a recognition model transmission unit 180, and a training model. A data storage unit 160 and a recognition model storage unit 142 are provided.

  The image obtaining unit 110 obtains an image transmitted from the imaging unit 210. The image obtaining unit 110 outputs the obtained image to the sampling unit 112.

  The sampling unit 112 receives the image output from the image acquisition unit 110, periodically samples the image from the received image, for example, and outputs the sampled image to the combining position setting unit 120.

  The synthesis position setting unit 120 is an example of the synthesis position determination unit 20 (see FIG. 1) in the embodiment, and arbitrarily sets the object synthesis position on the image sampled by the sampling unit 112.

  The image combining unit 130 is an example of the combined image generating unit 30 (see FIG. 1) in the embodiment, and combines an object at the object combining position set by the combining position setting unit 120. At this time, a GAN model is used as a method of combining the objects. Since the GAN model has been described in the embodiment, the description thereof is omitted here.

  The detection processing unit 140 is an example of the output data acquisition unit 40 (see FIG. 1) in the embodiment, and outputs a recognition model obtained by outputting the combined image combined by the image combining unit 130 to the recognition model holding unit 142. Get output data. More specifically, the detection processing unit 140 acquires output data of a recognition model obtained by inputting a composite image to the recognition model held in the recognition model holding unit 142. The detection processing unit 140 outputs the obtained output data to the data use determination unit 150.

  The data use determination unit 150 is an example of the determination unit 50 (see FIG. 1) in the embodiment, and uses a correct answer data including at least an object combining position and output data to train a recognition model based on a combined image. A second decision is made as to whether to make a first decision to determine the data. When the recognition accuracy of the recognition model determined based on the difference between the correct answer data and the output data is higher than a predetermined threshold, the data use determination unit 150 makes a second determination not to make the first determination. More specifically, in this case, the data use determination unit 150 determines that the composite image input to the recognition model is an image that is recognized by the recognition model, and performs training for the recognition model based on the composite image. A second decision not to determine the data is made. The data use determination unit 150 does not determine the training data of the recognition model based on the composite image according to the second determination.

  On the other hand, when the recognition accuracy of the recognition model is lower than the predetermined threshold, the data use determination unit 150 makes a second determination to make the first determination. More specifically, the data use determination unit 150 determines that the composite image input to the recognition model is an image in which an object is difficult to be recognized by the recognition model, and based on the composite image, training data of the recognition model. Is determined, a second determination is made. The data use determination unit 150 determines the composite image as training data according to the second determination. Further, the data use determination unit 150 determines, as training data, a corresponding image having the same or similar visual feature as the visual feature of the composite image. The corresponding image may be selected from an image stored in the training data holding unit 160 or may be generated. The data use determination unit 150 stores the image determined as the training data in the training data holding unit 160 as new training data.

  Here, the visual characteristics of the composite image are statistical characteristics of the image parameters of the composite image. Further, the visual characteristics of the composite image are qualitative characteristics of the composite image. Note that the statistical characteristics and qualitative characteristics of the image parameters are the same as those described in the embodiment, and thus description thereof will be omitted.

  The visual feature of the composite image is the aspect of the object in the composite image, and the visual characteristic of the corresponding image is the aspect of the corresponding object having the same or similar attribute as the attribute of the object. For example, the aspect is a position of the object on the composite image. More specifically, the position of the object on the composite image is the coordinates of the area occupied by the object. Also, for example, the aspect is the posture of the object.

  Note that the attribute of an object is a property of the object, such as the type, shape, color, and material of the object. More specifically, when the type of the object is a person, gender, physique, age, skin color, clothing, belongings, posture, age, expression, and the like may be included in the attributes of the object. When the type of the object is an automobile, the attribute of the object may include a vehicle type, a shape, a body color, a window glass color, and the like.

  The training data holding unit 160 stores new training data, a pre-hold DB (Data Base) including various images held in advance as training data, and the like. The training data holding unit 160 can store background information, object information, environment information such as weather, and the like in addition to the above-described data, and can retrieve the stored information. The training data holding unit 160 is realized by, for example, a storage device such as a RAM (Random Access Memory), a semiconductor memory such as a flash memory, a hard disk drive, or an SSD (Solid State Drive).

  The training data holding unit 160 stores the information of the image combining position on the image and the combined image in association with the combined image determined by the data use determination unit 150 as the training data of the learning model. Further, the training data holding unit 160 compares the corresponding image determined as training data based on the composite image by the data use determination unit 150 with the corresponding image of the corresponding object corresponding to the object synthesized on the composite image. And the corresponding image are stored in association with each other.

  The training data holding unit 160 holds the training data holding unit in response to receiving from the data usage determining unit 150 a command requesting an image that is the same as or similar to the synthesized image determined by the data use determining unit 150 as training data. A desired image is output from the new training data stored in the unit 160 and the pre-holding DB to the data use determining unit 150. In addition, in response to receiving a command for requesting training data from the training unit 170, the training data holding unit 160 reads the new training data stored in the training data holding unit 160 and the desired data from the pre-holding DB. Is output to the training unit 170.

  The recognition model holding unit 142 stores the same recognition model as the recognition model of the recognition unit 220. The recognition model holding unit 142 outputs, to the detection processing unit 140, output data obtained by inputting the synthesized image generated by the image synthesis unit 130 to the recognition model. The recognition model holding unit 142 acquires the recognition model trained by the training unit 170, and updates the obtained recognition model by storing it in the recognition model holding unit 142.

  The training unit 170 executes the training of the recognition model using the training data determined by the data use determining unit 150. For example, when a predetermined amount of new training data is stored in the training data holding unit 160, the training unit 170 reads the training data from the training data holding unit 160 and sends the training data to the training unit 170. The training of the recognition model is executed by inputting to the stored recognition model. The training unit 170 outputs the recognition model trained using machine learning to the recognition model holding unit 142 and the recognition model transmitting unit 180.

  The recognition model transmission unit 180 transmits the recognition model trained by the training unit 170 to the recognition unit 220 of the recognition processing unit 200. Upon receiving the trained recognition model, the recognition model receiving unit 3 of the recognition unit 220 outputs the recognition model to the recognition model updating unit 4.

[Operation of Information Processing System According to Modification 1]
The operation of the information processing system 100 according to the first modification will be described with reference to FIG. FIG. 7 is a flowchart illustrating a flow of the information processing method according to the first modification.

  As shown in FIG. 7, in the information processing system 100 according to the first modification, in step S10, the image obtaining unit 110 obtains an image captured by the imaging unit 210. The image obtaining unit 110 outputs the obtained image to the sampling unit 112.

  Next, in step S101, the sampling unit 112 receives the image output from the image acquisition unit 110, and, for example, periodically samples the image from the received image. The sampling unit 112 outputs the sampled image to the combining position setting unit 120.

  Next, in step S20, the combining position setting unit 120 receives the image output from the sampling unit 112, and arbitrarily determines the object combining position on the received image. The combining position setting unit 120 outputs the image in which the object combining position is determined to the image combining unit 130.

  Next, in step S30, the image combining unit 130 combines the object at the object combining position to generate a combined image. The image synthesis unit 130 synthesizes an object at an object synthesis position on an image using, for example, a GAN model. The image synthesis unit 130 outputs the generated synthesized image to the detection processing unit 140.

  Next, in step S40, the detection processing unit 140 receives the synthesized image output from the image synthesis unit 130, inputs the synthesized image to the recognition model stored in the recognition model holding unit 142, and obtains output data. I do. The detection processing unit 140 outputs the obtained output data to the data use determination unit 150.

  Next, in step S50, the data use determination unit 150 determines whether to make the first determination using the correct data of the composite image and the output data obtained by inputting the composite image to the recognition model. Make a second decision to do so. The first determination is to determine the training data based on the composite image. If it is determined in the second determination that the first determination is not made (NO in step S501), the flow of the information processing method for determining training data based on the composite image ends. On the other hand, when the first decision is made in the second decision (YES in step S501), in step S60, the data use determination unit 150 makes the first decision. At this time, the data use determination unit 150 determines training data based on the composite image. The data use determination unit 150 determines the synthesized image as training data for the recognition model. Further, the data use determination unit 150 determines a corresponding image having the same or similar visual feature as the visual feature of the composite image as the training data of the recognition model. Next, in step S601, the data use determination unit 150 stores the image determined as training data in the training data holding unit 160 as training data.

  When a predetermined amount of training data is stored in the training data holding unit 160, in step S70, the training unit 170 executes training of the recognition model using the determined training data.

  Next, in step S80, the training unit 170 outputs the recognition model trained using machine learning to the recognition model holding unit 142 and the recognition model transmitting unit 180. The recognition model holding unit 142 updates by storing the trained recognition model output from the training unit 170. The recognition model transmitting unit 180 transmits the trained recognition model output from the training unit 170 to the recognition unit 220 of the recognition processing unit 200.

  Note that the trained recognition model transmitted from the recognition model transmission unit 180 is received by the recognition model receiving unit 3 of the recognition unit 220, and is output to the recognition model updating unit 4. The recognition model updating unit 4 updates the recognition model by storing the trained recognition model received from the recognition model receiving unit 3. Further, when the recognition model receiving unit 3 receives the trained recognition model, the update information presentation unit 5 notifies the user of the completion of the training.

[Effects of Modification Example 1]
According to the information processing system 100 and the information processing method according to the first modification as described above, the following effects are obtained in addition to the effects described in the embodiment.

  In the information processing method according to the first modification, in the first determination, the synthesized image is determined as the training data of the recognition model.

  Thus, a synthesized image determined to have low recognition accuracy in the learning model can be used as training data. Therefore, accumulation of data with high recognition accuracy in the learning model, that is, data unnecessary as training data, as training data is suppressed. Therefore, the cost for storing data is reduced. In other words, since images of scenes with low recognition accuracy in the learning model can be accumulated as training data, efficient learning can be performed on scenes with low recognition accuracy. Therefore, the recognition accuracy of the learning model is further improved.

  In the information processing method according to the first modification, in the first determination, a corresponding image having the same or similar visual feature as the composite image has is determined as the training data of the recognition model. At this time, the visual characteristics of the composite image are statistical characteristics of the image parameters of the composite image. Further, the visual characteristics of the composite image are qualitative characteristics of the composite image.

  As described above, since the corresponding image having the same or similar visual feature as that of the composite image is determined as training data, an image of a scene with low recognition accuracy in the learning model and a scene similar to the image are used. Can be used as training data. Therefore, the number and variations of training data for a scene with low recognition accuracy can be efficiently increased. When the corresponding image is a photographed image, the learning effect can be improved as compared with the case where the synthetic image is used as training data. When the visual feature is a statistical feature of the image parameters of the composite image, the number and variations of the training data can be efficiently increased from a statistical viewpoint. When the visual features are qualitative features of the composite image, the number and variations of training data having features that are difficult to quantify can be efficiently increased.

  In the information processing method according to Modification 1, the visual feature of the composite image is an aspect of the object in the composite image, and the visual characteristic of the corresponding image has the same or similar attribute as the attribute of the object. FIG. In this case, the aspect is the position of the object on the composite image. Further, the aspect is a posture of the object.

  Thereby, for example, when it is determined that the recognition accuracy of the object in the learning model is low due to a difference in the state of the object such as the position of the object or the posture of the object on the synthetic image, the training data is determined based on the synthetic image. Is done. Therefore, an image of a scene with low recognition accuracy in the learning model and an image of a scene similar to the image can be used as training data. This makes it possible to efficiently increase the number and variations of training data for a scene with low recognition accuracy. A recognition model constructed using such training data has improved accuracy in recognizing an object from an image.

  Further, in the information processing method according to the first modification, when the first determination is made and the second determination is made, the user of the recognition model is notified. At this time, the notification is, for example, a notification regarding a request for training of a recognition model using the determined training data. Further, in the information processing method according to the first modification, the recognition model is further trained using the determined training data, and the notification is a notification regarding the completion of the training.

  With this, when the training data of the learning model is determined based on the synthetic image, the user of the learning model is notified, so that the user can grasp that there is a scene where it is difficult to recognize the object in the learning model. it can. In addition, when the notification is a notification regarding a request for training of a learning model, the user can determine a timing for performing training of the learning model. In addition, when the notification is a notification regarding the completion of the training, the user can know that the learning model has been updated by the training.

(Modification 2)
[Configuration of Information Processing System According to Modification 2]
An information processing system according to a second modification of the embodiment will be described with reference to FIG. FIG. 8 is a block diagram illustrating an example of a configuration of an information processing system 100 according to the second modification.

  In the information processing system 100 according to the first modification of the embodiment, the sampling unit 112 periodically samples an image from the images acquired by the image acquisition unit 110 and outputs the sampled image to the combination position setting unit 120. Examples have been described. In the second modification, the sampling unit 112 further samples an image that satisfies a predetermined condition from among the images acquired by the image acquisition unit 110 and stores the sampled image in the training data holding unit 160 as training data. explain. Hereinafter, the information processing system 100 according to Modification 2 will be described focusing on differences from the information processing system 100 according to Modification 1.

  In the information processing system 100 according to the second modification, an image having the same or similar visual feature as the composite image may be selected as a corresponding image from a captured image different from the composite image. The selection of the captured image may be performed by sampling the image based on a predetermined condition from the images acquired by the image acquisition unit 110, and may be performed based on the captured image stored in the training data holding unit 160. A desired captured image may be searched and extracted. The sampling unit 112 periodically samples an image from the images acquired by the image acquisition unit 110 and outputs the image to the synthesis position setting unit 120. Further, the sampling unit 112 samples an image that satisfies a predetermined condition from the images acquired by the image acquisition unit 110 and stores the sampled image in the training data holding unit 160 as training data. Here, the predetermined condition is a condition based on a scene in which the recognition accuracy of the object in the recognition model is low. For example, the type of the object, the position of the object, the state of the object, the lighting state, the weather condition, the climate, the building Consists of layout, road conditions, etc. For example, when the data use determination unit 150 determines the training data based on the composite image, that is, when the data use determination unit 150 makes the first determination and the second determination, It is configured to include the feature amount of the image that is the same or similar to the feature. Thereby, the sampling unit 112 updates the predetermined condition so that the same or similar image as the image with low recognition accuracy of the object in the recognition model is sampled as training data.

  When making the second decision to make the first decision, the data use determining unit 150 sets a predetermined condition so as to include a feature amount of an image that is the same as or similar to the visual feature of the composite image. , And outputs the conditions to the sampling unit 112.

  The information processing system 100 according to the second modification is different from the information processing system 100 according to the above embodiment and the first modification in that a similar scene search unit 190 is provided. For example, when the data use determination unit 150 makes the second determination that the first determination is made, the similar scene search unit 190 determines whether the image having the same or similar visual feature as the composite image has Is selected from the captured images stored in the training data holding unit 160. The visual features that are the same as or similar to the visual features of the synthesized image include, for example, the synthesized position of the object on the image, the background of the image, the form of the synthesized object such as the posture of the person, the color and edge of the image, Statistical characteristics of image parameters, and qualitative characteristics such as weather conditions, road surface wetting, and occlusion. The captured image may be a captured image selected by the sampling unit 112 and stored as new training data, or may be a captured image included in the pre-hold DB. The pre-hold DB has images of various scenes held as defaults by the information processing system. The images of various scenes include, for example, images taken in regions with different climates, different images such as weather, road surface conditions, landscapes, etc. For example, different images. Further, for example, when a storage unit for temporarily storing an image acquired by the image acquisition unit 110 is provided, the captured image may be selected from images temporarily stored in the storage unit.

[Operation of Information Processing System According to Modification 2]
The operation of the information processing system 100 according to Modification 2 will be described with reference to FIG. FIG. 9 is a flowchart illustrating an example of a flow of an information processing method according to the second modification.

  In the information processing system 100 according to the first modification of the embodiment, the sampling unit 112 periodically samples an image from the image acquired by the image acquisition unit 110 and outputs the image to the synthesis position setting unit 120. In the information processing system 100 according to Modification 2 of the embodiment, in addition to the above operation, the sampling unit 112 samples an image that meets predetermined conditions from among the images acquired by the image acquisition unit 110, and trains the image. The training data is stored in the training data holding unit 160 as training data. The similar scene search unit 190 obtains the same or the same visual feature as the composite image from the captured image stored as the new training data in the training data holding unit 160 and the captured image stored in the pre-holding DB. Select images with similar visual characteristics. In the information processing system 100 according to the second modification, a recognition model is constructed using an image group including these captured images as training data. Hereinafter, Modification 2 will be described focusing on differences from the embodiment and Modification 1.

  Specifically, in step S101, the sampling unit 112 samples an image that meets predetermined conditions from the image acquired in step S10, in addition to the same operation as step S101 in the first modification, Is stored in the training data holding unit 160. Next, the information processing system 100 performs the operations of steps S20 to S60 in the same manner as in the embodiment and the first modification. Although not shown, in the second modification, when the data use determination unit 150 makes the second determination to perform the first determination, the data use determination unit 150 determines the characteristics of the image that are the same as or similar to the visual characteristics of the composite image. A predetermined condition is set so as to include the quantity, and the condition is output to the sampling unit 112. When the sampling unit 112 receives the condition, the sampling unit 112 updates the predetermined condition by storing the predetermined condition in the sampling unit 112.

  Next, in step S602, the similar scene search unit 190 sets, as a corresponding image, an image that is the same as or similar to the synthesized image when the second determination for performing the first determination is performed, as the corresponding image, A search is performed from the captured images stored in 160, and a desired captured image is selected as training data. Next, the information processing system 100 performs the operations of steps S70 and S80 in the same manner as in the first modification.

[Effects of Modification Example 2]
According to the information processing system 100 and the information processing method according to Modification 2 as described above, the following effects are provided in addition to the effects described in the embodiment and Modification 1.

  In the information processing method according to the second modification, an image having the same or similar visual feature as the composite image is selected as a corresponding image from a captured image different from the composite image. The selection of the photographed image may be performed by sampling the image based on predetermined conditions from the images acquired by the image acquiring unit 110, and may be performed by storing the image in a memory and a database of the training data holding unit 160 and the like. A desired captured image may be searched for and extracted from the stored captured images.

  Thus, the captured image can be used as training data. Therefore, a higher learning effect can be obtained as compared with the case where the synthesized image is used as training data. Note that the selection of the captured image may be to determine whether or not to record the image based on a predetermined condition each time the image is acquired, and to select an image based on a predetermined condition from the acquired images. May be sampled, or a captured image satisfying a predetermined condition may be searched for and extracted from a captured image stored in a memory or a database.

(Modification 3)
[Configuration of Information Processing System According to Modification 3]
An information processing system according to Modification 3 of the embodiment will be described with reference to FIG. FIG. 10 is a block diagram illustrating an example of a configuration of an information processing system 100 according to Modification 3.

  The information processing system 100 according to Modification 2 of the embodiment includes a similar scene search unit 190. When the data use determination unit 150 makes the second determination that the first determination is made, the similar scene search unit 190 stores an image that is the same as or similar to the synthesized image in the training data holding unit 160. A search is performed from the captured images, and a desired captured image is selected as training data. The information processing system 100 according to the third modification does not include the similar scene search unit 190 but includes a similar scene processing unit 192. Hereinafter, the information processing system according to Modification 3 will be described focusing on differences from the information processing system 100 according to Modification 2.

  In the information processing system 100 according to Modification 3, an image having the same or similar visual feature as the composite image may be generated as a corresponding image from a captured image different from the composite image.

  For example, when the data use determination unit 150 makes the second determination that the first determination is made, the similar scene processing unit 192 determines that the image having the same or similar visual feature as the composite image has Is generated from the captured image stored in the training data holding unit 160.

[Operation of Information Processing System According to Modification 3]
The operation of the information processing system 100 according to Modification 3 will be described with reference to FIG. FIG. 11 is a flowchart illustrating an example of the flow of the information processing method according to the third modification.

  In the information processing system according to the second modification of the embodiment, when the similar scene search unit 190 makes a second determination that the data use determination unit 150 makes a first determination, the similar scene search unit 190 performs An image having the same or similar visual feature as the feature, that is, an image similar to the composite image is selected from the captured images stored in the training data holding unit 160. In the information processing system according to the third modification of the embodiment, when the similar scene processing unit 192 makes the second determination that the data use determination unit 150 makes the first determination, the similar scene processing unit 192 performs the visual determination of the composite image. An image having the same or similar visual feature as the feature, that is, an image similar to the composite image is generated from the captured image stored in the training data holding unit 160. Hereinafter, Modification 3 will be described focusing on differences from the embodiment and Modifications 1 and 2.

  Specifically, in step S603, the captured image of the similar scene, that is, the image of the scene similar to the composite image in the case where the second determination of performing the first determination is executed is set as the corresponding image. The captured image stored in the training data holding unit 160, which is different from the composite image, is processed and generated.

[Effects of Modification 3 etc.]
According to the information processing system 100 and the information processing method according to the third modification described above, the following effects are provided in addition to the effects described in the embodiment and the first modification.

  In the information processing method according to Modification 3, an image having the same or similar visual feature as the composite image is generated as a corresponding image from a captured image different from the composite image.

  Thereby, the corresponding image can be generated from the captured image. Specifically, an image of a scene with low recognition accuracy of an object in the recognition model and an image similar to the scene can be generated from the captured image. Accordingly, the corresponding image can be generated even when the captured image cannot be used as the corresponding image as it is, so that the number and variations of the training data can be easily increased.

[Other Modifications]
As described above, the embodiments and the modifications have been described as examples of the technology disclosed in the present application. However, the technology according to the present disclosure is not limited to these, and can be applied to a modification of the embodiment in which change, replacement, addition, omission, and the like are appropriately made or other embodiments. Further, it is also possible to combine the components described in the embodiment and the modified example to form a new embodiment or a modified example.

  The information processing system 100 according to the embodiment and the modified example is applied to an automobile. The information processing system may be applied to any system as long as the system recognizes a recognition target from sensing data. For example, the information processing system may be applied to a system that observes the behavior or state of a person in a building such as a residence or an office. In this case, the recognition processing unit 200 may be mounted on a sensor module such as a camera, and the recognition model updating unit 300 may be mounted on a sensor module or may be mounted on a device separate from the sensor module such as a server.

  Further, in the above-described embodiment, an example has been described in which the processing target is an image, but the processing target may be sensing data other than an image. For example, audio data output from a microphone, point cloud data output from a radar such as LiDAR, pressure data output from a pressure sensor, temperature data or humidity data output from a temperature sensor or humidity sensor, output from a scent sensor If it is sensing data from which correct answer data such as scent data to be obtained can be obtained, the sensing data may be set as a processing target. For example, when the sensing data is audio data, the elements of the audio data are frequencies and amplitudes, and the statistical characteristics of the elements of the audio data are frequency bands and sound pressures. Are noise and background sound.

  It should be noted that the general or specific aspects of the present disclosure may be realized by a recording medium such as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording disk. The present invention may be realized by an arbitrary combination of a circuit, a computer program, and a recording medium. The computer-readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM.

  For example, each component included in the information processing system 100 according to the embodiment and the modification is typically realized as an LSI (Large Scale Integration) which is an integrated circuit. These may be individually formed into one chip, or may be formed into one chip so as to include some or all of them. Further, the integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  In the embodiments and the modifications, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

  In addition, a part or all of the above components may be configured by a removable IC (Integrated Circuit) card or a single module. The IC card or module is a computer system including a microprocessor, a ROM, a RAM, and the like. The IC card or module may include the above-described LSI or system LSI. The IC card or module achieves its functions by the microprocessor operating according to the computer program. These IC cards and modules may have tamper resistance.

  The above method may be realized by an MPU, a CPU, a processor, a circuit such as an LSI, an IC card, a single module, or the like.

  Further, the technology of the present disclosure may be realized by a software program or a digital signal including the software program, or may be a non-temporary computer-readable recording medium on which the program is recorded.

  Note that the program and a digital signal including the program are stored in a computer-readable recording medium such as a flexible disk, a hard disk, an SSD, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, and a BD (Blu-ray). (Registered trademark) Disc), or may be recorded on a semiconductor memory or the like. The program and the digital signal composed of the program may be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, or the like. Further, the program and a digital signal composed of the program may be implemented by another independent computer system by being recorded on a recording medium and transferred, or transferred via a network or the like. .

  Further, the numbers such as ordinal numbers and quantities used above are merely examples for specifically explaining the technology of the present disclosure, and the present disclosure is not limited to the illustrated numbers. In addition, the connection relation between the components is illustrated for specifically describing the technology of the present disclosure, and the connection relation that realizes the function of the present disclosure is not limited thereto.

  The division of functional blocks in the block diagram is merely an example, and a plurality of functional blocks can be implemented as one functional block, one functional block can be divided into a plurality of functional blocks, and some functions can be transferred to other functional blocks. You may. Also, the functions of a plurality of functional blocks having similar functions may be processed by a single piece of hardware or software in parallel or time division.

  INDUSTRIAL APPLICABILITY The present disclosure can improve the efficiency of individual training of a learning model, and thus can be used for technologies such as an automatic driving system, a traffic management system, a security system, and a manufacturing management system.

DESCRIPTION OF SYMBOLS 1 Image imaging part 2 Image transmission part 3 Recognition model reception part 4 Recognition model update part 5 Update information presentation part 10 Image acquisition part 20 Synthesis position determination part 30 Synthetic image generation part 40 Output data acquisition part 50 Determination part 100 Information processing system 110 Image acquisition unit 112 Sampling unit 120 Synthesis position setting unit 130 Image synthesis unit 140 Detection processing unit 142 Recognition model storage unit 150 Data use determination unit 160 Training data storage unit 170 Training unit 180 Recognition model transmission unit 190 Similar scene search unit 192 Similar Scene processing unit 200 Recognition processing unit 210 Imaging unit 220 Recognition unit 300 Recognition model update unit

Claims (18)

Using a computer,
Get sensing data,
Determine a combining portion for combining the recognition target data on the sensing data,
In the synthesis portion, to generate recognition data by synthesizing recognition target data having the same or similar characteristics as the characteristics perceived by the human sensory organ of the sensing data,
The synthesized data is input to a model trained using machine learning to recognize a recognition target to obtain recognition result data,
Determining the training data of the model based on the combined data, determining whether to make a first decision or not, a second decision to determine whether or not to make a first decision; Using data and
If the first decision is made and the second decision is made, the first decision is made;
Information processing method. The feature that the sensing data has is a statistical feature of an element of the sensing data,
The information processing method according to claim 1. The feature of the sensing data is a qualitative feature of the sensing data,
The information processing method according to claim 1. In the first determination, the synthesized data is determined as training data for the model.
The information processing method according to claim 1. In the first determination, corresponding data having the same or similar feature as the feature of the composite data is determined as training data of the model,
The information processing method according to claim 1. The feature of the combined data is a statistical feature of an element of the combined data,
The information processing method according to claim 5. The feature of the combined data is a qualitative feature of the combined data,
The information processing method according to claim 5. The sensing data is an image,
The recognition target is an object,
The combining part is an object combining position for combining the object data on the image,
The synthesized data is a synthesized image generated by synthesizing the object data having the same or similar visual feature as the visual feature of the image at the object synthesis position,
The recognition result data is object recognition result data obtained by inputting the composite image to the model,
The first determination is to determine training data of the model based on the composite image,
The second determination is performed using correct data including at least the object synthesis position and the object recognition result data,
The information processing method according to claim 1. The first determination is to determine a corresponding image having the same or similar visual feature as the visual feature of the composite image as training data of the model,
The visual feature of the composite image is an aspect of the object in the composite image,
The visual feature of the corresponding image is an aspect of the corresponding object having the same or similar attribute as the attribute of the object,
An information processing method according to claim 8. The aspect is a position of the object on the composite image,
The information processing method according to claim 9. The aspect is a posture of the object,
The information processing method according to claim 9. The combining portion further includes a size of object data to be combined on the image,
The information processing method according to claim 8. Data having the same or similar feature as the feature of the combined data is selected or generated as the corresponding data from sensing data different from the combined data,
The information processing method according to claim 5. Synthesizing the recognition target data with the synthesizing part using a GAN (Generative Adversary Network) model;
The information processing method according to claim 1. Further, when the first decision is made and the decision is made in the second decision, the user of the model is notified,
The information processing method according to claim 1. Further, performing the training of the model using the determined training data,
The notification is a notification regarding completion of the training,
The information processing method according to claim 15. The notification is a notification regarding a request for training the model using the determined training data,
The information processing method according to claim 15. A first acquisition unit for acquiring sensing data;
A first determination unit that determines a combining part that combines recognition target data on the sensing data;
A generation unit that generates synthesis data by synthesizing recognition target data having the same or similar characteristics as characteristics perceived by a human sensory organ of the sensing data, in the synthesis unit,
A second acquisition unit that inputs the synthesized data to a model trained using machine learning to recognize a recognition target and obtains recognition result data;
Determining the training data of the model based on the combined data, determining whether to make a first decision or not, a second decision to determine whether or not to make a first decision; And a second determining unit that performs the first determination when the first determination is performed when the first determination is performed using the data.
Comprising,
Information processing system.

Images