JP2019211969A - Learning management device, learning management server, and learning management method - Google Patents

Abstract

To provide a learning management device, a learning management server, and a learning management method in which whether or not an inference model is appropriate is confirmed even when a condition, etc., has been changed.SOLUTION: An inference model is set in an inference engine (S81), image data is inputted and an output for displaying a guide is obtained with use of the inference model (S55), a guide is displayed on the basis of the output from the inference engine (S57), a determination regarding an update of the inference model is made, and a request to update the inference model or a request to determine whether or not an update of the inference model is necessary is sent to an external learning device on the basis of the determination result (S63, S69).SELECTED DRAWING: Figure 6

Description

  The present invention relates to a learning management apparatus, a learning management server, and a learning management method for confirming whether an inference model is appropriate in an inference apparatus that performs inference using an inference model generated in a learning apparatus.

  In recent years, the use of deep learning has advanced, and it has been proposed to analyze images using an inference model for deep learning (see Patent Document 1). In other words, various proposals have been made to use image data as a population (teacher data), generate an inference model using deep learning, analyze the input image using this inference model, and perform a guide or the like. . In addition, it has been proposed to change the image data depending on the situation (see Patent Document 2).

JP 2017-091525 A JP 2017-173307 A

  As described above, various proposals have been made to input an image, perform inference using an inference model, and perform a guide or the like. However, after the inference model is generated, the situation or the like changes, and there are cases where appropriate guidance or the like cannot be performed. However, conventionally, the situation change after generation of the inference model has not been considered.

  The present invention has been made in view of such circumstances, and a learning management apparatus and a learning management server that confirm whether or not an inference model is appropriate even when the situation or the like changes. An object is to provide a learning management method.

  In order to achieve the above object, a learning management device according to a first aspect of the present invention inputs an inference model, inputs image data, and uses the inference model to obtain an output for performing guide display to the user. Check the inference unit, the display unit that displays the guide based on the output of the inference unit, and the update of the inference model, and request the external learning device to update the inference model based on the result of the check Or an update confirmation unit that requests to determine whether or not the inference model needs to be updated.

In the learning management apparatus according to the second invention, in the first invention, the update confirmation unit acquires the latest teacher data, and uses the latest teacher data to trust the inference model currently used. If the reliability is low, the external learning device is requested to update the inference model.
The learning management device according to a third invention is the learning management device according to the first invention, wherein the update check unit requests external learning when a predetermined time elapses after requesting the external learning device to update the inference model. The apparatus is requested to update the inference model or to determine whether the inference model needs to be updated.

In the learning management device according to a fourth aspect of the present invention, in the first aspect, the update confirmation unit monitors a change in the current state, and updates the inference model to the external learning device when the current state changes. Ask.
In the learning management device according to a fifth aspect of the present invention, in the fourth aspect of the invention, the update confirmation unit determines that the current state has changed when the user does not adopt the guide display and does not reflect it in the operation. .

In the learning management device according to a sixth aspect of the present invention, in the first aspect, the update confirmation unit always confirms whether or not the inference model needs to be updated.
According to a seventh aspect of the present invention, in the learning management device according to the first aspect, the external device confirms whether or not the inference model needs to be updated, and the update confirmation unit confirms the confirmation result from the external device. Based on the determination.

The learning management server according to the eighth invention includes a determination unit that determines whether or not a request for confirmation of an inference model has been input, or whether or not a predetermined time has elapsed since the last update of the inference model; Input a request for confirmation of the inference model, or if it is determined that a predetermined time has elapsed since the last update of the inference model, a population creation unit that sets a learning population from the database, and And an update confirmation unit that determines the necessity of updating based on the reliability of the inference model that has been requested using the learning population created by the population creation unit.
The learning management server according to a ninth invention is the learning management server according to the eighth invention, wherein the population generation unit sets a learning population from the database based on a condition used when a previous inference model is generated. To do.

  When the learning management server according to the tenth invention generates an inference model by learning, whether a predetermined time has elapsed since the database for storing usable data and the learning for generating the inference model have been performed If the time determination unit and the time determination unit determine that the predetermined time has elapsed, the database is searched based on the conditions used when the previous inference model was generated. A determination unit that determines whether or not the upper data is fluctuating, and a recommendation unit that recommends updating of the inference model when the determination unit determines that the upper data is fluctuating. Have.

  In the learning management server according to the eleventh aspect of the present invention, when the inference model is generated by learning, a database that stores data including a third party evaluation that can be used and the third party evaluation is changed. A determination unit that determines whether or not there is a recommendation unit, and a recommendation unit that recommends updating of the inference model when the evaluation by the third party is changed by the determination unit.

  A learning management method according to a twelfth aspect of the present invention sets an inference model in an inference unit, inputs image data, obtains an output for performing guide display to a user using the inference model, The guide display is performed based on the output, and the update of the inference model is determined. Based on the result of the determination, the external learning device is requested to update the inference model, or whether the inference model needs to be updated. Ask for a decision.

A learning management method according to a thirteenth aspect of the present invention is the learning management method according to the twelfth aspect of the present invention, wherein the latest teacher data is acquired, the reliability of the inference model currently used is detected using the latest teacher data, If the reliability is low, the external learning device is requested to update the inference model.
The learning management method according to a fourteenth aspect of the present invention is the learning management method according to the twelfth aspect of the present invention, when a predetermined time elapses after the external learning apparatus is requested to update the inference model, the external learning apparatus Requests an update or requests to determine whether the inference model needs to be updated.

According to a fifteenth aspect of the present invention, in the twelfth aspect, a learning management method according to the fifteenth aspect of the present invention monitors the current state change and requests the external learning device to update the inference model when the current state changes.
In the learning management method according to the sixteenth aspect of the present invention, in the fifteenth aspect of the present invention, when the user does not adopt the guide display and does not reflect it in the operation, it is determined that the current state has changed.
In the learning management method according to the seventeenth aspect of the present invention, in the twelfth aspect of the present invention, it is always confirmed whether or not the inference model needs to be updated.

  A computer embedding method according to an eighteenth aspect of the present invention is a computer embedding method capable of cooperating with a social network system, wherein the social network system can post content items at any time, and the posted content The quality of the item can be evaluated from the user of the social network system at any time, and the computer-embedded method is a specific content item with the quality evaluation information attached among the listed content items. An inference model that infers the quality evaluation from the features of the content item using the content item as teacher data depends on the change of the content item added by the above time transition and the time change of the evaluation trend of the content item. Initial aim Determining that the inference result can no longer be output.

According to a nineteenth aspect of the present invention, there is provided a computer incorporation method according to the eighteenth aspect, further comprising the step of recording, as a history, conditions when machine learning is performed using the content item as teacher data. It is recommended to generate new teacher data according to
According to a twentieth aspect of the invention, there is provided a computer incorporation method according to the eighteenth aspect of the invention, further comprising the step of recording, as a history, a condition when a specific user performs machine learning using the content item as teacher data.

  According to the present invention, it is possible to provide a learning management device, a learning management server, and a learning management method that confirm whether or not an inference model is appropriate even when the situation or the like changes. .

It is a block diagram which shows the whole structure and learning management method of the learning management system which concern on 1st Embodiment of this invention. It is a block diagram which shows the structure for inference model formation in the learning system which concerns on 1st Embodiment of this invention. It is a figure which shows the image change at the time of inserting an industrial endoscope into a test object, when the learning management system which concerns on 1st Embodiment of this invention is applied to an industrial endoscope. 5 is a flowchart illustrating a control operation of the image acquisition device in the learning management system according to the first embodiment of the present invention. 5 is a flowchart showing an inference model acquisition operation in the learning management system according to the first embodiment of the present invention. It is a block diagram which shows mainly an electrical structure of the learning system which concerns on 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the camera in the learning system which concerns on 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the external apparatus in the learning system which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the operation | movement of the image management in the learning system which concerns on 2nd Embodiment of this invention. It is a block diagram which shows mainly an electrical structure of the learning system which concerns on 3rd Embodiment of this invention. It is a block diagram which mainly shows the electrical structure of the camera in the learning system which concerns on 3rd Embodiment of this invention. It is a flowchart which shows operation | movement of the camera in the learning system which concerns on 3rd Embodiment of this invention. It is a flowchart which shows operation | movement of the camera in the learning system which concerns on 3rd Embodiment of this invention. It is a flowchart which shows operation | movement of the image management server in the learning system which concerns on 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the image management server in the learning system which concerns on 3rd Embodiment of this invention.

  Hereinafter, as a first embodiment of the present invention, an example applied to a learning management system will be described with reference to FIGS. 1A to 4.

  FIG. 1A shows a configuration for generating an inference model in a learning apparatus. For example, this learning device is linked with a social network system, and a third party “likes” a content item (such as a user-posted photo or blog) posted on a social network. It assumes an environment where quality evaluation (not necessarily limited to subjective evaluation) can be written. In such an environment, among the posted content items, a specific content item accompanied by quality evaluation information is a teacher for inferring what kind of content item gets what kind of evaluation. For use as data, it is possible to obtain content items and their evaluation results from social networks. Therefore, machine learning for inferring the relationship between the characteristics of the content item and the quality evaluation using the acquired teacher data can be performed by a computer or the like. By using the inference engine obtained from the learning result, it is possible to predict what kind of evaluation the content item to be posted will get.

  In the above-mentioned social network system, content items can be posted and posted at any time, and the posted content items are evaluated at any time (including preferences) by users of the social network system. Is possible. For this reason, the trend of followers may change due to the appearance of new innovative content items, the addition of new evaluators, and the addition of opinion leaders. As the situation changes with time, the inference engine becomes obsolete. For example, even when the situation has changed, learning may be performed with old data. In this case, the teacher data is already an outdated data set (old data set).

  In FIG. 1A, an inference engine 203A (network design (number of layers)) performs learning using an old data set 201A composed of data sets A and B as teacher data, and generates an inference model. That is, the old data set 201A is teacher data composed of a combination of input data (image data or the like) and an output result. The learning start unit 204A causes the inference engine 203A to perform learning so that it matches the output result of the teacher data when input data (image data or the like) of the teacher data is input to the inference engine 203A. The inference model is a result of executing learning, and is the strength and weight of neuron connection in the inference engine 203A. The configuration of the inference engine 203A will be described later with reference to FIG. 1B.

  After the inference model is generated, the inference engine 203A inputs input data such as image data from the input unit 205A, and outputs an inference result from the output unit 206A. The learning management control unit 207A determines the reliability of the inference result. As shown in the lower right graph in FIG. 1A, the reliability of this inference result decreases with time. As obsolete begins, the rate of decline may increase unexpectedly.

  For example, it is assumed that an inference model is generated using image data evaluated as “like” as the old data set 201A. In this case, when the image data is input to the input unit 205A, the inference engine 203A can perform a guide so that an image that is evaluated as “like” can be taken. However, after the inference model is generated, if the evaluator's preference of the image uploaded to the Internet changes, the inference model generated by the old data set 201A may capture an image to be evaluated. It may not be possible. In this case, even if the latest “like” image A ′ is input, it is not evaluated as “like”, and the reliability B ′ is lowered.

  When the reliability decreases, the learning management unit 207A sets the learning data as a new data set for the learning device 200B (in FIG. 1, 200B and 200A may be the same, but are indicated by 200B for simplicity). Change to 201B. The inference engine 203B performs learning using the data set 201B as teacher data, and regenerates an inference model. The regenerated inference model is output from the external output unit 205B and input to the inference engine 203 in the image acquisition apparatus 210 or the like outside. By using the regenerated inference model, the reliability of the inference result can be maintained at a certain level.

  Next, the configuration of the inference engines 203A, 203B, and 203 that performs the deep learning described above will be described with reference to FIG. 1B. In this inference engine 203, an intermediate layer (neuron) is arranged between the input layer I and the output layer O. As this intermediate layer, several layers of neurons N11 to N1n, N21 to N2n,... Are arranged. The number of neuron layers is appropriately determined by design, and the number of neurons in each layer is also appropriately determined by design. Note that the inference engines 203A, 203B, and 203 may be chips formed by hardware circuits or may be realized by software.

  When performing deep learning, input data such as image data in the teacher data is input to the input layer I, and a correct answer in the teacher data is given to the output layer O. The strength (weighting) of each neuron and the connection between each neuron is determined by learning so that the correct answer is output from the output layer O when image data is input to the input layer I. As the inference model, the strength (weight) of each neuron obtained by learning and the connection between each neuron is output. The image data input to the input layer I is narrowed down to a group of image data according to a predetermined condition, and the correct data of the output layer O is given a high evaluation (“like” evaluation) from a third party. It may be.

  Next, with reference to FIG. 2, an operation when an image is acquired with an industrial endoscope as an image acquisition device will be described. Previously, we described an example of an environment for rating social network systems. However, since the invention of the present application is not limited to this system, an example in which the invention of the present application can be practiced without being in that environment is given here. Over time, the reasoning engine becomes obsolete is that it is a third-party evaluation, and the design and material of the object to be inspected may change depending on the era. This is because the machines that carry out the inspection also have model changes, minor changes, available accessories that change, and the habits of users that change depending on the generation.

  FIG. 2 shows an example of a teacher image when the present embodiment is applied to an industrial endoscope 301 that observes the inside of the tube 311. 2A shows a state where the industrial endoscope 301 is inserted into the tube 311 and collides with the tube wall, and FIG. 2B shows the state of the industrial endoscope 301 inside the tube 311. Is inserted and shows the state of going straight.

  The insertion portion 302 of the industrial endoscope 301 is flexible and tubular. The insertion portion 302 includes a distal end portion 302a, a bending portion 302b, and a flexible tube portion 302c in order from the distal end side. The bending portion 302b has, for example, a plurality of bending pieces (not shown). The bending portion 302b is freely bendable and can be bent in an arbitrary direction by a user's hand operation. The distal end portion 302a is provided with an imaging element 303 having an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor (not shown) and an illumination part (not shown).

  The pipe 311 includes two pipes, a pipe 311 a and a pipe 311 b, and the pipe 311 a and the pipe 311 b are joined at a joint 312. In the example shown in FIG. 12, ring-shaped rust 313 is generated inside the joint 312, and is raised inside.

  FIGS. 2A and 2B show how the insertion portion 302 of the industrial endoscope 301 is inserted into the tube 311 in order to observe the state of the rust 313. FIG. 2A shows a case where the distal end portion 302a of the insertion portion 302 cannot be successfully moved to the observation position of the rust 313 and collides with the tube wall. On the other hand, FIG.2 (b) shows the case where the front-end | tip part 302a of the insertion part 302 was made to advance straight in a pipe | tube, and was able to be moved successfully.

  Images P <b> 1 to P <b> 8 shown in the lower part of FIG. 2A indicate images acquired by the imaging unit 303 of the industrial endoscope 301. Images P <b> 1 to P <b> 3 are images near the entrance of the tube 311, and the far side of the tube 311 is dark (see black circles). The image P4 is an image acquired at time T = 0, and the rust 313 is visible outside the black circle indicating the distance.

  At time T1, when the distal end portion 302a of the insertion portion 302 is moved downward and proceeds as it is (time T1 to T3, images P5 to P7), the distal end portion collides with the rust 313 (time). T4, image P8). In this case, the rust 313 cannot be observed from above.

  In FIG. 2B, images acquired before time T0 are the same as those in FIG. At time T1, the tip 302a goes straight through the tube 311 and passes near the rust 313, so that the center of the image is dark, and the rust 313 appears circumferentially outside. Further, at time T2, the tip 302a passes over the rust 313, so that the rust 313 is not visible, and only the center of the image is dark.

  As described above, when the industrial endoscope 301 is inserted into the tube 311 and the inside is observed, unless it is appropriately operated, as shown in FIG. Can not be observed. However, as described with reference to FIG. 1, guide display can be performed by generating an inference model of the inference engine using the teacher image data.

  However, the inside of the tube 311 changes, and the imaging unit and the insertion unit of the industrial endoscope 301 may be updated, and the reliability of the guide display using the inference model may be reduced. Therefore, in the present embodiment, the learning management control unit 207A requests the learning device to update the inference model using new teacher data when the reliability is lowered.

  Next, the operation of the learning management system according to the first embodiment will be described using the flowcharts shown in FIGS. 3 and 4. The flowchart shown in FIG. 3 shows the control operation of the image acquisition device 210. The image acquisition apparatus 210 shown in this flowchart is an example applied to an industrial endoscope 301 as shown in FIG.

  When the flow shown in FIG. 3 starts, first, an inference model is acquired as necessary and a history is recorded (S1). Here, an inference model used by the inference engine of the image acquisition device 210 is acquired as necessary (see S47 in FIG. 4). If it has already been acquired, or if it has already been acquired and no update is necessary, the acquisition process is not performed. In addition, when an inference model is acquired, the history of acquisition date, type of device used, inference model version, inference model usage, inference model target, conditions under which the inference model is valid, etc. are recorded. To do.

  Subsequently, an image is input, displayed, and inferred (S3). Here, image data is input by the imaging unit 303, and display is performed on the display unit based on the image data. The user can observe the inside of the tube 311 by this display. The inference engine 203 performs inference for guide display.

  Next, an image is temporarily recorded (S5). Here, the image acquired in step S3 is temporarily recorded in the memory in the industrial endoscope 301. In steps S7 and S9, which will be described later, inference is performed based on the change in the temporarily recorded image.

  Once the image has been temporarily recorded, the images are then compared to determine whether a change has been detected (S7). Here, the most recently temporarily recorded image is compared with the image temporarily recorded before that. Then, based on the comparison result, it is determined whether or not the image has changed. For example, there is no change between the images P1 to P3 in FIG. 2A, while there is a change between the images P3 and P4, and similarly between P4 and P5, P5 and P6,. But the image has changed.

  If a change is detected as a result of the determination in step S7, an inference is performed using image changes before and after the change (S9). Here, the change in the image before and after the change is input to the inference engine 203 to obtain an inference result. For example, as shown in FIG. 2A, when the distal end portion 302a of the insertion portion 302 is moving, if the image changes like P4 to P5 and P5 to P6, it collides with the tube wall. Can be inferred. On the other hand, as shown in FIG. 2B, when the distal end portion 302a of the insertion portion 302 is moving, if the image changes like P14 to P15 and P15 to P16, it collides with the tube wall. You can infer that you will go straight ahead without doing. When inference is performed based on the image change, a guide based on the inference result is displayed.

  Subsequently, it is determined whether or not the change is smooth (S11). In this step, based on the inference result in step S9, it is determined whether or not it is changing smoothly. For example, in the case as shown in the images P4 to P8 in FIG. 2A, it is inferred that it collides with the tube wall, and in such a case, it is not judged to be smooth. On the other hand, as shown in the images P11 to P16 in FIG. 2B, when it is inferred that the vehicle travels straight in the pipe, it is determined to be smooth.

  If the result of determination in step S11 is not smoothly changing, a warning is displayed (S13). Here, since it is not changing smoothly, a warning is displayed so as to make the operation cautious to the user. . Once the warning is displayed, it is next determined whether or not the warning operation is continued (S15). In this step, although the warning is displayed, the operation performed by the user is continued as it is, and it is determined whether or not there is any trouble (for example, a collision or the like).

  If the result of determination in step S15 is that the operation has been continued, shooting is performed and the fact that the warning is ignored is recorded (S17). In spite of the warning display, the user or the object may be damaged when the user continues to operate forcibly. Therefore, in this step, in order to record the location of the responsibility as evidence, a picture is taken and the fact that the warning is ignored is recorded.

  On the other hand, if the result of determination in step S15 is that the operation has not been continued, re-learning is requested (S19). If the user changes behavior without ignoring the warning, the inference model may no longer match the current situation. For example, there may be various circumstances such as when the imaging unit 303 of the industrial endoscope is replaced, when the material of the insertion unit 302 is different, or when the internal state of the tube 311 has changed significantly. In such a case, in order to update the inference model, the learning device 200 is requested to perform re-learning (see 31 in FIG. 4).

  If it is recorded in step S17 that the warning is ignored, or if re-learning is requested in step S19, or if the result of determination in step S7 is that the image has not changed, or if the image has changed smoothly Next, it is determined whether or not a shooting operation has been performed (S21). Here, it is determined whether or not an operation for acquiring a still image (or moving image) has been performed by the imaging unit 303. If the result of this determination is that no shooting operation has been performed, processing returns to step S3.

  If the result of determination in step S21 is that a shooting operation has been performed, shooting is performed and recorded (S23). Here, image data is acquired by the imaging unit 303, and image processing is performed on the image data, and then the image data is recorded on a recording medium or the like. When the image data is recorded, the process returns to step S3.

  Next, the inference model acquisition operation will be described using the flowchart shown in FIG. This inference model acquisition is realized by a processor such as a CPU (Central Processing Unit) in the learning management control unit 207A of the learning device 200 controlling each unit in the learning device according to a program stored in the storage unit. .

  When the inference model acquisition flow shown in FIG. 4 starts, it is first determined whether or not it is a re-learning request (S31). As described above, the image acquisition device 210 requests the learning device 200 to re-learn and create a new inference model through the communication unit when the reliability of the inference model decreases. come. In this step, it is determined whether or not a re-learning request has been received from the image acquisition apparatus.

  If the result of determination in step S31 is that there is no request for re-learning, the image change upon insertion is set as a population (S33). Here, the image acquisition device 210 (industrial endoscope 301) extracts the image change at the time of insertion into the tube 311 from the acquired image data in step S5 or step S19, that is, difference data of the image. The difference data is used as a population.

  On the other hand, when there is a re-learning request, the mother set is updated according to the learning history (S35). In this step, the learning history recorded in step S1 is referred to renew the mother set. That is, after the inference model is generated last time, a new set is regenerated, for example, by setting an image after the time when the reliability of the inference result is lowered as a set. In renewing this mother set, a mother set is created based on whether it is similar to the currently observed image based on the learning history. In other words, it is possible to generate a highly reliable inference model by using, as a population, images that are images after the previous learning and that are similar to the current image. In addition, when looking at the history information and not being separated from the current situation, or when there is not enough teacher data, it is only necessary to replace a part of the mother set.

  In step S33 or S35, when a population is generated, next, an image change at the time of insertion is converted into a teacher image (S37). In this step, as shown in FIG. 2 (b), when the industrial endoscope 301 is inserted into the tube 311 from the population image, the image changes when it can be moved smoothly. Is extracted, and the image and the insertion OK are associated with each other as teacher data.

  Next, a teacher image is formed by distinguishing image changes at the time of insertion NG (S39). In this step, as shown in FIG. 2A, when the industrial endoscope 301 is inserted into the tube 311 from the image of the population, the image of the image when it cannot be moved smoothly. The change is extracted, and this image is associated with the inserted NG to be used as teacher data.

  Subsequently, an inference model is generated (S41). Here, the image extracted in steps S37 and S39 is used as teacher data, and the learning apparatus 200 generates an inference model. That is, the inference engine 203 sets the strength (weight) of the connection of each neuron in the intermediate layer so that the result matches each input image when the result is OK or NG. Calculate and output the calculation result as an inference model.

  When the inference model is generated, if there is request data, inference is performed (S43). Here, if there is request data other than the teacher image in steps S37 and S39, the request data is used for inference. That is, if there is request data recorded in an external database or the like and the result is known, the data is input, and inference is performed using the inference model generated in step S41. Request data is not limited to data recorded in external data, but may be data that has been newly acquired and whose results are known, and data that has been obtained as a result of trial and error under request conditions. Can also be used as teacher data. Since the inference model generated in step S41 must be correctly inferred using these request data, it is preferable to use image group data (request data) that is actually adapted. By performing this inference, the reliability of the inference model generated in step S41 can be predicted.

  Subsequently, it is determined whether or not the inference reliability is equal to or higher than a predetermined value (S45). Here, the reliability of the inference model generated in step S41 is determined. For example, the reliability is determined by calculating a LOSS value. The LOSS value is based on the inference result of the inference model generated by deep learning and the previously known answer when deep learning is performed with a practice question whose answer (for example, OK or NG at the time of insertion) is known. Is the difference. As a practice question for which the answer is known in advance, the request data in step S43 may be used, or a recorded image at insertion (an image other than the teacher image) may be used. In addition to this, if verification is performed using the previous request data, OK is displayed for data that goes well, and NG is displayed for data that does not go well. It can also be determined that the property is good.

  If the result of determination in step S45 is that the reliability is not greater than or equal to a predetermined value, the learning population is reset (S49). The reason why the reliability is low is that there is a possibility that the population of image data used when the inference model is generated in step S41 is not appropriate. So, in this step, try changing the population. In this case, processing such as changing to image data at an appropriate time may be performed based on the learning history.

  On the other hand, as a result of the determination in step S45, if the reliability of the inference model is high, the inference model is transmitted (S47). In the image acquisition device 210 (industrial endoscope 301), an inference model is installed in the initial stage. In this step, when a new inference model is generated, the new inference model is transmitted to the requesting image acquisition apparatus 210 through the communication unit. When the inference model is transmitted, this flow is finished.

  As described above, in the first embodiment of the present invention, when the image change is not smooth and the warning is displayed in the image acquisition device 210 and the state where the user ignores the warning continues, the learning device 200 is Ask for re-learning. Relearning makes it possible to always maintain a highly reliable inference model.

  In the present embodiment, re-learning is requested based on the user's behavior when a warning is displayed, troubles that occur thereafter, and the like. However, the timing for requesting relearning is not limited to this. For example, if the target model has changed from the previous one after the inference model has been set for the device, or if the worker has changed, If it is considered that the environment has changed from the assumed environment, re-learning may be requested. In addition, even if an inference model is used, re-learning may be requested even when work cannot be performed well, such as when it takes too much time. In such a case, teacher data may be newly collected (the device itself may be created or may be obtained from the outside), and re-learning may be requested. Further, for example, the LOSS value may be calculated at a predetermined timing in the image acquisition apparatus using the teacher data (may be request data), and the re-learning request may be performed based on the result.

  In the case of an industrial endoscope as shown in FIG. 2, for example, when an image group that precedes a collision with a tube wall is input to the inference engine, a warning is displayed at the timing before the collision. Is required. On the other hand, when a predetermined observation can be made without collision, when the image group at the time of the operation is input to the inference engine, it is required that an OK display is displayed together with the display of the image group. This is a reliability determination as to whether or not a correct learning model has been made based on the correct request data (determination in S45).

  Next, a second embodiment of the present invention will be described with reference to FIGS. The present embodiment is an example in which a camera is employed as an image acquisition device constituting the learning system. This camera is linked to social network systems, and content items (user-posted photos, blogs, etc.) posted on social networks are evaluated by third parties for quality evaluations such as “Like!” (It is not necessary to be limited to subjective evaluation). In such an environment, among the posted content items, a specific content item accompanied by quality evaluation information is a teacher for inferring what kind of content item gets what kind of evaluation. For use as data, it is possible to obtain content items and their evaluation results from social networks. Therefore, machine learning for inferring the relationship between the characteristics of the content item and the quality evaluation using the acquired teacher data can be performed by a computer or the like. By using the inference engine obtained from the learning result, it is possible to predict what kind of evaluation the content item to be posted will get.

  A camera equipped with the inference engine described above displays a guide so that an image recorded by an external database is evaluated as “like” by a third party. This guide is performed using an inference model inferred by the learning unit using images and evaluations recorded in an external database.

  FIG. 5A shows a block diagram mainly showing an electrical configuration of the learning system according to the second embodiment. This learning system includes a camera 10 and an external device 20. The camera 10 corresponds to the image acquisition device 207 in FIG. 1, and the external device 20 corresponds to the learning device 200.

  The camera 10 includes an image input unit 11, an inference engine 12, a communication unit 14, a display unit 15, a setting unit 16, and a control unit 17. The camera 10 functions as an imaging device that receives an inference model. The image input unit 11 includes a photographing lens, an imaging element, an imaging control circuit, an image data processing circuit, and the like, and generates image data of a subject.

  The inference engine 12 inputs an inference model generated by an input / output modeling unit 28 in the external device 20 described later, and stores this in the storage unit 13 as a learning product 13a. The inference engine 13 uses the inference model stored in the storage unit 13 to infer whether the image data input by the image input unit 11 is an image that matches the request set by the setting unit 16. . As in FIG. 1B, the inference engine 13 has an input layer, an intermediate layer, and an output layer, and the strength and weight of the neuron connection between them is in accordance with the inference model (learning product) stored in the storage unit 13. Is set. The inference engine 12 functions as an inference unit (inference engine) that receives image data and obtains an output for performing guide display to the user using an inference model.

  The communication unit 14 includes a communication circuit, and performs wireless communication or wired communication with the communication unit 26 of the external device 20 through the network. The communication unit 14 transmits the request set by the setting unit 16 to the learning unit 25 and inputs the inference model generated by the input / output modeling unit 28. The communication unit 14 functions as an input unit (input interface) for inputting an inference model.

  When the camera 10 cannot directly access the external device 20, the camera 10 may communicate with the external device 20 through a portable communication device such as a smartphone. In addition, when the camera 10 itself is built in a portable communication device such as a smartphone, it is only necessary to communicate with the external device 20 through the portable communication device.

  The display unit 15 includes a display panel and the like, and performs display based on the image data acquired by the image input unit 11. In addition, the inference result in the inference engine 12 is displayed. In addition, items set in the setting unit 16 are displayed as shown in FIG. The display unit 15 displays the previously learned date 15b, and also displays an update icon 15c for the user to manually request re-learning. The re-learning instruction is not limited to the update icon 15c, and may be performed by operating an operation member such as a switch or a dial. The display unit 15 functions as a display unit that performs guide display based on the output of the inference unit.

  The setting unit 16 includes an interface circuit, and sets a request (request) when a user requests an inference model. For example, it is possible to set a photographing guide that is evaluated by a third party (e.g., evaluated as “like” by a third party) and can capture a photograph desired by the user. In the example shown in FIG. 5B, the accompanying information such as the nationality, gender, theme, etc. of the evaluator (target follower) can be set so as to match the user's request. However, it should be noted that content items can be posted and posted at any time in a network environment represented by such a social network system. There is a possibility that quality (including preferences) is evaluated by users such as the system. This can lead to drastic changes in such circumstances, especially with the introduction of new and innovative content items, the addition of new evaluators, and the addition of opinion leaders. The trend may change. There was a risk that the inference engine would become obsolete for such changes in the situation over time.

  The control unit 17 is a processor having a CPU (Central Processing Unit), peripheral circuits, a memory, and the like, and the CPU controls each unit in the camera 10 according to a program stored in the memory.

  The control unit 17 includes a learning management unit 17b. As described with reference to FIG. 1A (the learning management control unit 207A in FIG. 1A), the learning management unit 17b determines whether to check the reliability of the inference using the inference model, and the reliability decreases. If it is, the learning device is requested to re-learn. Even when the user instructs re-learning by touching the update icon 15c or the like, the learning device is requested to re-learn (see S63 to S73 and S81 in FIG. 6).

  The learning management unit 17b determines whether to update the inference model, and based on the determination result, requests an external learning device to update the inference model, or requests to determine whether the inference model needs to be updated. It functions as a confirmation unit (see, for example, S15 and S19 in FIG. 3, S63 in FIG. 6, S183 and S185 in FIG. 11A). The update confirmation unit acquires the latest teacher data, detects the reliability of the currently used inference model using the latest teacher data, and if this reliability is low, An update of the inference model is requested (for example, see S65, S67, and S69 in FIG. 6).

  In addition, the update confirmation unit requests whether or not to update the inference model from the external learning device when a predetermined time elapses after requesting the external learning device to update the inference model, or whether the inference model needs to be updated. (For example, refer to S65 in FIG. 6). The update confirmation unit monitors changes in the current state, and when the current state changes, requests an external learning device to update the inference model (for example, S7 to S19 in FIG. 3, S63 in FIG. 6, S271 in FIG. 12B). Etc.). The update confirmation unit determines that the current state has changed when the user does not adopt the guide display and reflects it in the operation (see, for example, S15 and S19 in FIG. 3 and S183 and S185 in FIG. 11A).

  The update confirmation unit always confirms whether or not the inference model needs to be updated (for example, see S63 in FIG. 6). In addition, the external device confirms whether or not the inference model needs to be updated, and the update confirmation unit makes a determination based on the confirmation result from the external device (see S63 in FIG. 6, S157 and S165 in FIG. 8). .

  The external device 20 includes an external image database (DB) 21 and a learning unit 25. The external image DB 21 and the learning unit 25 do not need to be arranged in the same device, and it is only necessary that both can be linked by communication. A plurality of external image DBs 21 may be provided, as long as the learning unit 25 can access the image data to the plurality of external image DBs through the communication unit 25 and the communication unit 26 and acquire information about the image data. .

  The external image DB 21 includes a communication unit 22, an image classification unit 23, and a control unit 24. The external image DB 21 can be accessed from a third party. For example, if the third party likes it, it is indicated that the third party's preference is met, such as “like”. This intention expression is recorded in the external image database in association with the evaluated image via the net, and is announced via the net. The external image DB 21 functions as a database that accumulates usable data when generating an inference model by learning. The external DB 21 functions as a database that can be used when generating an inference model by learning and accumulates data including third-party evaluations.

  The communication unit 22 includes a communication circuit, and transmits / receives data to / from the communication unit 26 of the learning unit 25. The image classification unit 23 performs image classification of image data stored in the external image DB 21 or stored in the external image DB 21 and an external image DB. The image classification unit 23 stores a follower profile 23a and a theme 23b. In general, image data is recorded in the image DB together with a third party's evaluation of the image, for example, the number evaluated by the third party as “like”. The follower profile 23a records information (follower profile, for example, a region where the follower lives, sex, age, etc.) regarding a person (follower) who is evaluated as “like”. The theme 23b records information on the theme of an image, for example, information on a subject to be photographed (for example, landscape, person, cooking, etc.) in association with the image.

  The control unit 24 is a processor having a CPU (Central Processing Unit), peripheral circuits, memory, and the like, and the CPU controls each unit in the external image DB 21 according to a program stored in the memory. The control unit 24 includes a DB management unit 24b.

  The DB management unit 24b manages image data stored in the external image DB. The DB management unit 24b functions as a time determination unit that determines whether or not a predetermined time has elapsed since learning for generating an inference model (see, for example, S151 in FIG. 8). When the time determination unit determines that the predetermined time has elapsed, the DB management unit 24b searches the database based on the conditions used when the previous inference model was generated, and the upper data changes. It functions as a determination unit that determines whether or not (see, for example, S153 and S155 in FIG. 8). The DB management unit 24b functions as a recommendation unit that recommends updating of the inference model when the determination unit determines that the upper data is fluctuating (see, for example, S157 in FIG. 8).

  The DB management unit 24b functions as a determination unit that determines whether or not a third party's evaluation has changed (see, for example, S161 in FIG. 8). Further, the DB management unit 24b functions as a recommendation unit that recommends updating of the inference model when there is a change in the evaluation of a third party by the determination unit (see, for example, S165 in FIG. 8).

  The learning unit 25 includes a communication unit 26, a population generation unit 27, an input / output modeling unit 28, and a control unit 29. The learning unit 25 generates an inference model that can be photographed based on a request of the user of the camera 10 from the image data stored in the external image DB 21. The request is a determination criterion when determining the quality and quantity ratio for an image before shooting. For example, the learning unit 25 extracts, from the image database, an image that matches a request transmitted from the camera 10 and is evaluated by a third party and exceeds a predetermined value, and uses the extracted image to Learning is performed and an inference model is output (for example, see S91 to S99 in FIG. 7). It is possible to determine whether or not the image has a desired quality level by inputting the image to the inference engine in which the inference model is set before shooting or at the time of shooting.

  In addition, when a reliable inference model is generated in the range of image data for which deep learning is determined, the learning unit 25 outputs the model as a learning intermediate model (for example, S101 and S105 in FIG. 9). (See S109).

  The communication unit 26 includes a communication circuit, and transmits / receives data to / from the communication unit 22 of the external image DB 21. The communication unit 26 also transmits / receives data to / from the communication unit 14 of the camera 10.

  The population creation unit 27 creates an optimal population based on a request set by the camera setting unit 16 to be described later. As the optimal population, all the image data may be used as they are. However, if the population is too large, processing takes time. If the population is too large, over-learning may occur and reliability may be reduced. Therefore, a mother set of an appropriate size is extracted from all the image data using information such as nationality and gender transmitted as accompanying information.

  The population creation unit 27 inputs a request for confirmation of the inference model by the determination unit, or sets a learning population from the database when it is determined that a predetermined time has elapsed since the last update of the inference model. It functions as a population generation unit (see, for example, S115 in FIG. 7). This population creation unit sets a learning population from the database based on the conditions used when the previous inference model was generated (see, for example, S115 in FIG. 7).

  The input / output modeling unit 28 performs deep learning (machine learning) using so-called artificial intelligence (AI). Specifically, the input / output modeling unit 28 generates an inference model of an image satisfying the request set by the setting unit 16 by deep learning using the image data population created by the population creation unit 27. To do. The output modeling unit 28 has a configuration as shown in FIG. 1B. The image data created in the population creation unit 27 is input to the input layer I. In addition, an image evaluation result, for example, teacher data (correct answer) “a woman in France in the 20s gave a high evaluation (like and evaluate)” is given to the output layer O. An inference model is generated by calculating the strength (weighting) of the connection between the neurons N11 to N2n so that the input and the output match.

  If the image data population is large, it takes a considerable amount of time to complete the inference model by deep learning (depending on the size of the population and the request (condition) set by the setting unit 16) ). Therefore, even if inference does not end for all of the image data of the population, if a certain level of reliability can be expected, the inference model at this time may be output to the camera as a learning intermediate model. Good.

  The control unit 29 is a processor having a CPU (Central Processing Unit), peripheral circuits, a memory, and the like. The CPU controls each unit in the learning unit 25 according to a program stored in the memory.

  The control unit 29 has a learning management unit 29b. Upon receiving a re-learning request from the camera 10, the learning management unit 29b causes the population creation unit 27 to create a population, and re-learns to the input / output modeling unit 28 using this population to infer the inference model. Is generated. Further, the learning management unit 29b may substitute for the function of confirming the inference model performed by the learning management unit 17b of the camera 10. The processing in this case will be described later with reference to FIG.

  The learning management unit 29b functions as a determination unit that determines whether a request for confirmation of the inference model has been input or whether a predetermined time has elapsed since the last update of the inference model (for example, FIG. 7). (See S111). The learning management unit 29b functions as an update confirmation unit that determines the necessity of updating based on the reliability of the requested inference model using the learning population created by the population creation unit (FIG. 7). (See S121).

  Note that all or part of the CPU peripheral circuits in the control units 17, 24, and 29 may be realized by a CPU (Central Processing Unit) and a program code, or a program code such as a DSP (Digital Signal Processor). It may be realized by a circuit to be executed, or may be a hardware configuration such as a gate circuit generated based on a programming language described by Verilog, or may be executed by a hardware circuit. Of course it doesn't matter. Further, a part of the CPU function may be realized by a circuit executed by a program code such as a DSP, or may be a hardware configuration such as a gate circuit generated based on a program language described by Verilog. Also, it may be realized by a hardware circuit.

  Next, operations in the present embodiment will be described with reference to flowcharts shown in FIGS.

  FIG. 6 shows the operation in the camera 10, and the control unit 17 executes the flowchart shown in FIG. 6 according to the program stored in the memory. When the camera control flow starts, it is first determined whether or not the camera is in shooting mode (S51). In this flow, the camera 10 describes only two types of shooting mode and learning mode. However, in addition to these modes, other modes such as a playback mode may be executed. If the learning mode is not set by an operation member (not shown), it is determined as the shooting mode. The learning mode is a mode in which the user requests the learning unit 25 to generate an inference model by the setting unit 16.

  If the result of determination in step S51 is shooting mode, image input is performed (S53). In this step, image data is acquired by the image input unit 11. A through image is displayed on the display unit 15 based on the acquired image data.

  Once the image has been input, inference is performed (S55). Here, the inference engine 12 performs inference on the input image data using the inference model generated by the input / output modeling unit 28 (obtained in S81 described later). The inference engine 15 is configured as shown in FIG. 1B. The image data input by the image input unit 11 is input to the input layer I of this inference engine, and the inference model stored in the storage unit 13 is set as the strength (weight) of the connection between the neurons N11 to N2n. . The output from the output layer O of the inference engine 15 becomes an inference result.

  Once the inference is performed, the inference result is displayed (S57). Here, the inference result performed in step S5 is displayed. For example, if the request for the learning unit 25 includes accompanying information according to “a photograph that is evaluated by a woman in 20s in France”, whether or not the image input in step S53 matches this request, that is, Whether or not a certain number of third-party evaluations (“like”) can be obtained is displayed on the display unit 15.

  The input / output modeling unit 28 does not end the deep learning for the image data initially set as the learning set, but generates and transmits a learning intermediate model for the reset image data set, and the inference engine. When 12 uses the mid-learning model to infer, the display is performed so that the mid-learning model is used. Further, when the inference model is not suitable for an image displayed on the current through image, a warning display indicating that the inference cannot be performed may be performed. Moreover, when evaluation from a third party cannot be obtained as a result of inference, an advice display for taking a photograph so as to obtain the evaluation may be displayed.

  Once the inference result is displayed, it is next determined whether or not to shoot (S59). Here, the determination is made based on whether or not the release button of the camera 10 is fully pressed. The user can refer to the display of the inference result in step S57 depending on whether or not to perform shooting. Further, photographing may be performed with reference to the advice display. If the result of determination in this step is not shooting, processing returns to step S51 and the above-described operation is executed.

  If the result of determination in step S59 is shooting, shooting is executed (S61). Here, exposure control is performed so as to achieve proper exposure, image data input by the image input unit 11 is subjected to image processing for recording, and the image processed image data is recorded on a recording medium.

  Once shooting has been carried out, it is next confirmed whether or not the inference model is updated (re-learning) (S63). As described with reference to FIG. 1A, even when an optimal inference model is generated, if the situation changes thereafter, the reliability of the inference model decreases. That is, it is possible to confirm whether or not the photographing result in step S61 is inferred with correct judgment. In other words, it is possible to confirm whether or not the quality determination and quality level determination according to the user's request are valid even in the current situation. For this reason, when there is anxiety about whether or not to update (relearn) the inference model, a branch is provided so that the user can confirm this. For example, even when a shooting guide capable of shooting “Like” as described above is displayed, it may not be evaluated as “Like” due to a change in the viewer's preference or the like. Therefore, in this step, it is determined whether or not re-learning is necessary so that the optimal inference model can be updated. As a result of the determination in this step, when the necessity of re-learning cannot be confirmed, the process returns to step S51 and the above-described operation is executed.

  In step S63, whether or not to confirm the update is determined not only by the user's manual operation but also by the camera based on specific conditions, or the external device is requested to reflect the result. You may make it do. For example, in order to perform online learning all the time, you may always make it confirm. In step S69, the update may be automatically confirmed when a predetermined time elapses from the time when the re-learning is requested. In this case, steps S65 and S67 described later may be omitted, and re-learning may be requested in S69. Further, when the external database side monitors an image or the like and determines that the inference model needs to be updated, the camera side may request confirmation (for example, see S157 and S165 in FIG. 8). In this case, steps S65 and S67 described later may be executed, but may be omitted and re-learning may be requested in S69. Further, as described above, the image may be requested for evaluation using the photographed image. Alternatively, the latest popular image or the like may be acquired and input to the current inference model set in the camera 10 to determine whether or not a high evaluation is obtained.

  Further, as an update confirmation method in step S63, the determination may be made based on an external evaluation, such as uploading an image captured by the camera 10 to an external database and evaluating “like”. The inference model of the camera 10 provides a shooting guide so that it is evaluated as “like”, and when it is uploaded to an external database, it is considered that the reasoning model is inappropriate because it is not evaluated as “like”. If the user uses the camera 10 and feels that re-learning is necessary, the user operates the update icon 15c. This operation is determined in this step.

  If the result of determination in step S63 is to confirm whether or not to relearn, then the latest sample is acquired and the inference result is confirmed (S65). Inference by the inference engine is an inference result (output) when input data is input. The latest sample is the input data and the result of evaluation such as part-of-speech pressure and pass / fail for the input data. By requesting the learning unit 25, an image satisfying the condition can be acquired (S117, S119 in FIG. 8). reference). In addition to obtaining the latest sample from the learning unit 25, an image obtained by a third party by uploading an image obtained by the camera 10 or the like may be used. The latest sample may be obtained by regularly collecting third party evaluations. You can upload it to an external database and use it as the latest sample along with the evaluation results. Here, an example is shown in which the camera is the main request, but the camera may determine the necessity of acquiring the latest sample based on information transmitted / received from an external device. The information may be the sample itself. Note that the teacher data is a large number of data for generating the inference model, whereas the sample data only needs to have a number of data that can confirm the reliability of the inference model.

  Subsequently, it is determined whether or not the reliability is lowered (S67). In this step, the plurality of latest samples acquired in step S65 are used as teacher data and inference is performed in the inference engine 12, and it is determined whether or not the reliability of the inference result is lowered. That is, if the ratio of cases where the results for the input data do not match is greater than a predetermined value, it is determined that the reliability has decreased. The confirmation of the decrease in reliability is not necessarily performed at this timing, and may be referred to when the inference model is acquired in step S71.

  If the result of determination in step S67 is that the reliability has decreased, a request is made for learning with new teacher data (S69). Here, the camera 10 requests re-learning to the learning management 29b through the communication units 14 and 26. Upon receiving a request for re-learning, the learning management unit 29b causes the population generation unit 27 to generate new teacher data by referring to the learning history and the like, and the input / output modeling unit 28 uses this teacher data to generate an inference model. Is generated (see FIG. 7). If the learning is renewed in step S69, or if the result of determination in step S67 is that reliability has not decreased, the process returns to step S51 to execute the above-described operation.

  Returning to step S51, if the result of determination in this step is not shooting mode, it is determined whether or not an inference model or the like is acquired (S71). When the user wants to obtain a shooting guide (either manual determination or automatic determination), the user requests use of the shooting guide and transmits it to the learning unit 25. For example, when a third party wants to take a photograph or the like that gives a high evaluation (“like”), the user transmits a request set by the setting unit 16 to the learning unit 25 of the external device 20. The request sets, for example, accompanying information such as the nationality, gender, theme, etc. of the evaluator (target follower) as in “a photograph that is evaluated by a woman in her 20s in France” (see step S75 for details). Then, the inference model is returned from the learning unit 25, and the camera 10 performs shooting using the inference model. If the inference model has already been acquired at this timing, it is also possible to perform some correction or version upgrade. Therefore, the reliability determination as in step S67 may be performed here.

  In step S71, it is determined whether or not the user wants to acquire this inference model. Since the user sets the learning mode using the touch panel, the operation member, or the like, the determination is made based on these operation states. If the result of this determination is that the inference model or the like is not acquired, processing returns to step S51 and the above-described operation is performed.

  Next, it is determined whether or not it is received (S73). When a request is transmitted to the learning unit 25 in step S79 described later, or when re-learning is requested to the learning unit 25 in step S69, an inference model is transmitted from the learning unit 25 (see S103 in FIG. 8). ). In this step, it is determined whether or not the inference model transmitted from the learning unit 25 is received.

  If the result of determination in step S73 is reception, acquisition or renewal (update) of an inference model for a specific application is performed (S81). Here, the inference model is received, and when the inference model has already been received, it is renewed (replaced) with the newly received inference model. It is assumed that the renewal (update) of this inference model is in response to the request, and if this request has already been made, there is no need to perform it again, so a new version of the inference model is acquired. . If there is an inference model that can be used for general purposes without request information, the general model may be acquired. When the processing in this step is performed, the process returns to step S51.

  If the result of determination in step S73 is not reception, a request is set (S75). When the user requests the learning unit 25 to perform deep learning, the setting unit 16 sets a request in order to perform shooting according to the user's intention. In this step, these requests are set. The request includes, for example, an item (accompanying item) that is set to receive high evaluation from a third party. This accompanying item is the theme of the photo of the evaluator's profile (also referred to as “follower profile”) such as nationality, gender, age group, etc. of the third party to be evaluated, landscape, person, cooking, etc. (Also called “target profile”). Note that the request is not limited to the follower profile and the theme, and may be included as appropriate as long as the request represents a photograph intended by the user as long as the request represents the user's intention. The request may be an image intended by the user. If a request has already been set, it may be compliant. In addition, the request may not always be valid. For example, an image of a type assumed by the user or a display indicating that deep learning cannot be performed may be displayed when there are no or a small number of evaluators.

  Subsequently, it is determined whether or not learning is requested (S77). In this step, it is determined whether or not the request set in step S75, for example, a request set with accompanying information such as nationality, gender, theme, etc. is transmitted to the learning unit 25 of the external device 20 through the communication unit 14. . As a result of the determination, if not requested, the process returns to step S51.

  If the result of determination in step S79 is that learning is requested, conditions are transmitted and renewed (S79). Here, the request set in step S75 is transmitted to the learning unit 25 through the communication units 14 and 26.

  As described above, in the camera control according to the present embodiment, it is confirmed whether or not the inference model used for the shooting guide or the like is updated (see S63). Etc.) and inferred using the inference engine 12 of the camera 10, and judgment is made based on the reliability of the inference result (see S65 to S69). For this reason, even an apparatus that uses an inference model using deep learning offline can hold an optimal inference model.

  Next, the operation of the external device 20 will be described using the flowchart shown in FIG. This operation is realized by the learning management unit 29b and the DB management unit 24b in the external device 20 controlling each unit in the external device 20 according to a program stored in the memory.

  When the control flow of the external device shown in FIG. 7 is started, it is first determined whether or not it is a learning request (S91). As described above, in steps S <b> 69 and S <b> 79, the camera 10 requests learning from the learning unit 25 of the external device 20 through the communication unit 14. If the result of this determination is that there is no learning request, a standby state is entered.

  If the result of determination in step S91 is that learning has been requested, a learning mother set is set (S93). All image data of the external image DB 21 and the usable external image DB may be used as a learning mother set. However, here, in order to learn efficiently, based on a request from the user, image data to be subjected to deep learning is extracted from an external image database and a learning population set is set.

  Subsequently, the population is narrowed down from the follower profile (S95). The request set in the setting unit 16 may include a follower profile that evaluates the target image as “like” in the deep learning. In this case, the population may be narrowed down using the follower profile. If the follower profile is not included in the request, an image having a high evaluation is extracted based on the accompanying information included in the request, and the follower profile of this image is acquired. If the profile of the follower is known, in this step, the learning population is narrowed down using this information.

  Next, the follow image is determined (S97). Here, it is determined whether or not the image that matches the request set by the user in the setting unit, that is, an image highly evaluated by a third party (an image evaluated as “like”) from the image data of the population To do. An image (follow image) determined to match the theme in this determination is used as an input image for deep learning in the input modeling unit 28.

  Next, deep learning is performed to generate an inference model (S99). Here, the input / output modeling unit 28 performs deep learning on the set population using a follow image to generate an inference model.

  When generation of the inference model is started, it is determined whether or not feature determination of the follow image is possible with a specific scale calculation (S101). Depending on the size of the population, the number and characteristics of follow images, and the circuit scale for performing deep learning, the time for performing deep learning may be considerably long. Here, the user determines whether or not the calculation is normally completed within an allowable time as the generation time of the inference model.

  If the result of determination in step S101 is that computation with a specific scale is possible, an inference model is transmitted (S103). Here, when the input / output modeling unit 28 completes the generation of the inference model by deep learning, the generated inference model is transmitted to the camera 10 (see S81 in FIG. 6).

  On the other hand, if the result of determination in step S101 is not possible with a specific scale calculation, the learning population is reset (S105). Since there is too much image data to be a learning mother set, the mother set that is the target of deep learning is changed and reset again using a request (accompanying information or the like).

  Once the learning population is reset, it is next determined whether or not there is a request for transmission even during learning (S107). As described above, deep learning often takes time. Therefore, as described above, if the inference model is generated with a certain degree of reliability, the user may request to transmit the learning intermediate model. In this step, it is determined whether or not this request has been made. If the result of determination in this step is that there is no transmission request, processing returns to step S99.

  Note that whether or not the inference model has a certain degree of high reliability may be determined based on whether or not the LOSS value is lower than a threshold value. The LOSS value is a difference between an inference result in an inference model generated by deep learning and an answer that is known in advance when deep learning is performed with a practice question whose answer is already known. Which evaluation method is used for the inference model by deep learning affects the reliability of the learning apparatus. In the present embodiment, the learning apparatus is evaluated using the LOSS value.

  On the other hand, if the result of determination in step S107 is that there is a transmission request, the mid-learning model is transmitted to the camera 10 (S109). Here, the learning mother set is reset, and an inference model with a certain degree of high reliability is generated within this range, and thus this learning intermediate model is transmitted to the camera 10. Even after the mid-learning model is transmitted, deep learning for inference model generation is performed, and when the inference model is generated, it is transmitted to the camera 10 (S103). Note that when the mid-learning model is transmitted, the generation of the inference model may be terminated. In this case, step S103 is skipped and the process returns to step S91.

  Returning to step S39, if the result of the determination in this step is not a learning request, it is determined whether or not the update is due to an inference confirmation request or the passage of time (S111). As described above, the camera 10 may confirm the update of the inference model and request re-learning of the inference model (S69 in FIG. 6). Further, the confirmation function may be performed in the external device 20, and in this case, in step 111, it is confirmed whether or not the inference model needs to be updated. For example, the update may be performed when a predetermined time elapses after learning. In addition, the confirmation method performed in step S63 may be performed in this step. If the result of this determination is that the request is not an inference confirmation request and not updated over time, image management is performed (S113). This image management will be described later with reference to FIG.

  If the result of determination in step S111 is an update due to an inference confirmation request or the passage of time, the latest sample that matches the previous condition is searched (S115). In this step, the latest sample (mother set, teacher data) that matches the condition (request) when the inference model was generated in the previous deep learning is searched.

  Subsequently, it is determined whether there is a sample request (S117). As described above, in step S65 (see FIG. 6), the camera 10 may request the learning unit 25 for the latest sample. In this step, determination is made based on whether or not the latest sample is requested from the camera 10.

  If the result of determination in step S117 is that there is a request for sample transmission, sample transmission is performed (S119). Here, the latest sample searched in step S115 is transmitted to the camera 10. On the other hand, when there is no sample transmission request, the reliability is confirmed and the result is transmitted (S121). Here, the inference engine performs inference using the sample searched in step S115, the reliability of the inference result is confirmed, and the result is transmitted to the camera 10. When the sample transmission in step S119 or the reliability result in step S121 is transmitted, the process returns to step S91 to execute the above-described operation.

  Thus, in the control flow of the external device 20, when a learning request is received (S91 Yes), a learning mother set is set (S93), an inference model is generated (S99), and the generated inference model is transmitted (S99). S103). When requested to confirm the inference, the latest sample is searched (S115), and if there is a sample request, it is transmitted to the camera 10 (S115, S117, S119). If there is no sample request, the reliability of the inference model is confirmed, and the reliability result is transmitted to the camera (S121). When the camera 10 receives a result indicating that the inference model has low reliability from the external device, the camera 10 requests re-learning (see S69 and S71 in FIG. 6).

  Next, the image management in step S113 will be described using the flowchart shown in FIG. This flow of image management is mainly executed by the CPU in the DB management unit 24b provided in the external image DB 21.

  When the image management flow shown in FIG. 8 starts, it is first determined whether or not an image is acquired (S121). An image captured by the user with the camera 10 may be uploaded to the external image DB 21. In this step, it is determined whether an image has been uploaded (image acquisition). If the result of determination in step S121 is that the image is to be acquired, the image is recorded (S123). Here, the uploaded image is recorded in the database of the external image DB 21.

  After the image is recorded, it is next determined whether or not there is a comment or the like (S125). Information such as comments may be associated with the image. In this step, it is determined whether or not there is information such as a comment. If there is a comment or the like as a result of this determination, the comment is recorded (S127). Here, the comment is recorded in the database of the partial image DB 21 in association with the image. If a comment is recorded in step S127, or if there is no comment as a result of the determination in step S125, the process returns to step S91 to execute the above-described operation.

  Returning to step S121, if the result of determination in this step is that the image is not acquired, it is next determined whether or not the image is displayed (S131). A third party may access the external image DB 21 and browse images recorded in the image database. In this step, it is determined whether or not there is a request for image browsing. If the result of determination in step S131 is image display, specific condition image display is performed (S131). In this step, an image specified by a third party is transmitted to the third party and displayed.

  Next, it is determined whether a comment or a favorite has been received (S135). In step S <b> 133, the user who has received the image may transmit a comment or evaluate it as “like” for the received image. In this step, it is determined whether or not there is a comment or a favorite display. If the result of determination in step S135 is a good or a comment, recording is performed (S139). Here, the likes and comments are recorded in association with the displayed image. If a comment or a good is recorded in step S139, or if there is no comment or good as a result of the determination in step S135, the process returns to step S91 to execute the above-described operation.

  Returning to step S131, if the result of determination in this step is that there is no image display, it is next determined whether or not it is a request for teacher data (S141). In some cases, the input / output modeling 28 extracts from the images recorded in the external image DB 21 as teacher data when inference is performed (see S115). In addition, the learning management unit 29b searches for the latest sample in step S115. The latest sample is a new image accompanied by an evaluation result, a newly evaluated old image, or the like. If there is a sufficient number of sample data (request data) for deep learning, it becomes teacher data. In this search, there is a case where a search is made from images recorded in the external image DB 21. In this step, it is determined whether or not it is a request for searching for these teacher data.

  If the result of determination in step S141 is a request for teacher data, an image with a specific condition is searched (S141). Here, when a teacher data search request is made, an image recorded in the image DB 21 is searched based on a given condition (specific condition). When the image search is completed, the learning date and history management, such as transmission, is performed (S145). Here, the searched image is transmitted to a request source such as the learning unit 25. Moreover, the date information learned by performing this search is acquired, and information such as date information is recorded as a history. If history management or the like is performed in step S145, the process returns to step S91 to execute the above-described operation.

  Returning to step S141, if the result of determination in this step is not a teacher data search request, it is next determined whether or not a predetermined time has passed since the previous learning (S151). As described above, when time elapses after the inference model is generated by deep learning, the situation changes, and the reliability of the inference model may decrease. Therefore, in this step, the determination is made based on the elapsed time since the deep learning was performed based on the history recorded in step S145.

  If the result of determination in step S151 is that a predetermined time has elapsed since the previous learning, a re-search is performed under the previous (request) condition (S153). Here, the mother set is generated under the conditions when deep learning was performed when an inference model satisfying a specific condition was obtained last time. Therefore, in this system, it is preferable to store what the previous request is and at what timing. For this purpose, each user may be recorded in the recording unit of the image management apparatus. Further, it may be possible to receive and determine what is recorded on the user side.

  Once the re-search is performed, it is next determined whether or not the upper data has fluctuated (S155). Here, it is determined whether or not the upper data of the population searched in step S153 is fluctuating.

  If the result of determination in step S155 is that the upper data has fluctuated, updating is recommended (S157). In this case, since the contents of the population have changed greatly, there is a high possibility that the inference model is not optimal and the reliability of the inference result may be low. Therefore, updating is recommended to the user of the camera 10. This update recommendation is received in S73 and S81 in FIG. 6 and the recommendation may be displayed as a recommendation. In S63, automatic determination may be performed to confirm. If updating is recommended in step S157 or if the result of determination in step S155 is that there is no change in the upper data, processing returns to step S91 and the above-described operation is executed.

  Returning to step S151, if the result of determination in this step is that the time has not elapsed since the previous learning, it is next determined whether or not the number of rice has rapidly increased (S161). When the user of the camera 10 uses an inference model suitable for photographing an image that is evaluated as “like”, the number of “like” affects the generation of the inference model. That is, when the preference of a third party changes, an image in which the number of likes rapidly rises appears.

  If the result of determination in step S161 is that the number of rice has risen rapidly, it is next determined whether there is a learning history of similar followers (S163). Even if the number of rice is soaring, the inference model may be created targeting an image where the number of rice is soaring, or the image may be of no interest to the user. This is because the viewer's preferences have changed rapidly with the passage of time, the number of viewers with new preferences has increased rapidly with SNS, and new images that match the preferences of the times (matching the preferences of many people) are uploaded. Since it is a case of evaluation, old learning is equivalent to becoming old. For example, if the request is a “photo that is evaluated by a woman in her 20s in France”, even if the number of rice is soaring, it should be considered if it is an image that a French woman has reacted to. However, even if a Japanese man evaluates locally as good, it is not the content that should be the target of this user's request. Therefore, it is determined whether or not there is a learned history for an image similar to an image in which the number of likes has risen sharply.

  If the result of determination in step S163 is that there is a learning history of similar followers, updating is recommended (S165). Since similar followers have been learned in the past, updating is recommended. If updating is recommended in step S165, or if both the determination results in steps S161 and S163 are No, the process returns to step S91 to execute the above-described operation.

  As described above, in the image management flow, when an image is uploaded (S121) and when an image browsing request is received (S131), the image is recorded and displayed. If there is a teacher data search request (S141), a search according to the request is performed. Furthermore, it is determined whether or not it is recommended to update according to the learning time from the previous time (S155). Further, it is determined whether or not it is recommended to update according to the change in the number of rice (S163).

  Note that, as in steps S151 to S165 of image management, the DB management unit 24b of the external image DB 21 determines whether or not to recommend updating. However, this flow may be determined by the learning management unit 29b of the learning unit 25.

  As described above, in the second embodiment, a camera is employed as the image acquisition device, and the learning management unit is disposed in the camera 10 and the external device 20. For this reason, whether or not the inference model needs to be updated is confirmed at a predetermined timing and condition, so that the optimum inference model can be held.

  Next, a third embodiment of the present invention will be described with reference to FIGS. 9 to 12B. Also in this embodiment, it is an example which employ | adopted the camera as an image acquisition apparatus which comprises a learning system. In the first embodiment, the guide for inserting the industrial endoscope 301 into the tube 311 is performed using the inference model. In the first embodiment, if the user continues to ignore the warning displayed by the guide, the inference model is updated by re-learning. In the second embodiment, a shooting guide for shooting an image evaluated by a third person is performed by an inference model. Whether or not to update the inference model is confirmed at a predetermined timing (or a predetermined condition), and when the reliability of the inference model is reduced, the learning device is requested to perform re-learning. It was updated to a new inference model. Also in the third embodiment, whether or not to update the inference model is confirmed at a predetermined timing (or a predetermined condition), and when the reliability of the inference model decreases, a re-learning is requested to the learning device. Update to the new inference model obtained as a result of learning. Compared to the second embodiment, the third embodiment is different in that a plurality of inference models are prepared in advance.

  FIG. 9 is a block diagram mainly showing an electrical configuration of the learning system according to the third embodiment. This learning system also includes a camera 10 and an external device 20. Details of the external device 20 are shown in FIG. 9, and details of the camera 10 are shown in FIG. The external device 20 may be an image publication server operated by each business operator. This server may publish an image without limitation, or may limit the publishing range such as publishing only to members or the like (the second embodiment also applies to the publishing range). The image recording unit 140, the model DB 130, and the control unit 110 may not be arranged in the same device, and these three parties may cooperate through communication. In addition, a plurality of image recording units 140 may be provided, and it is only necessary to access the image data in the plurality of external image recording units 140 from the control unit 110 through the communication unit and obtain information on the image data.

  The external device 20 includes a control unit 110, a communication unit 120, a model database (model DB) 130, and an image recording unit 140. All the units 110 to 140 in the external device 20 may be provided in the server 30, and some of the units 110 to 140 in the external device 20 are provided in the server 30. It may be provided in an external server or the like.

  The control unit 110 includes an access determination unit 111, a display control unit 112, a tendency determination unit 113, a favorite model learning unit 114, and a database (DB) management unit 115. The control unit 110 is a processor having a CPU (Central Processing Unit), peripheral circuits, a memory, and the like, and the CPU controls each unit of the external device 20 according to a program stored in the memory. Each unit 110 to 140 in the external device 20 is mainly realized by a program, but a part or all of the units 110 to 140 may be realized by a hardware circuit.

  The access determination unit 111 determines whether or not the external device 20 is accessed from the personal computer (PC) 250 used by the user A. The display control unit 112 controls the display when the image recorded in the image recording unit 140 is displayed on the PC 250 or the result learned by the rice model learning unit 114 is displayed on the PC 250.

  The tendency determination unit 113 analyzes the image data recorded in the camera 10 (or PC 250) and determines the tendency of the image of the user A. For example, from various viewpoints, such as the shooting target, the position of the shooting target, the size of the shooting target, the shooting direction (landscape, front, or looking up), the focal length at the time of shooting, the exposure control value, etc. The tendency of the captured image is determined. In addition, when determining the tendency of the image of the user A, the tendency determination unit 113 selects a favorite model that is closest to the tendency of the image of the user A from the first to n-th favorite models stored in the model DB 130. A search is performed, and the searched “like model” is transmitted to the PC 250.

  The rice model learning unit 114 includes an input layer I, intermediate layers N11 to N2n, and an output layer O as illustrated in FIG. 1B, and calculates an inference model. In other words, like the input / output modeling unit 28 described above, the rice model learning unit 114 performs deep learning to generate an inference model for obtaining a rice image. As described above, a good image is an image that has been highly evaluated by a third party such as “like” for an image uploaded to the Internet or the like.

  Also, the favorite model learning unit 114 creates guide information 33. The guide information 33 is an evaluation by a third party when the inference result is displayed together with the through image display on the display unit 8 of the camera 10 and the through image is predicted not to be highly evaluated by the third party. It is advice for making a high image. This guide display 33 is instructed according to the difference between the inference result and the through image. For example, as a result of inference, when the focal length is desirable on the telephoto side, but on the wide angle side in the through image, this difference in focal length becomes the photographing guide information. In addition, it is desirable that the inference result is that the aperture is open and the depth of field is shallow, whereas in the live view, when the aperture is narrowed down and the depth of field is deep, this aperture value The difference becomes shooting guide information. The guide display 33 may be generated by deep learning. The guide display 33 may be an example of an image (sample image) that has been highly evaluated by a third party.

  The large number of images input for learning by the rice model learning unit 114 is acquired by associating information that can be used to identify what evaluation each image has. For this reason, the favorite model learning unit can deeply learn the difference in image quality, composition, facial expression, hue, and the like between an image with a good evaluation and a bad image. Of course, it is also possible to identify a specific relationship and perform machine learning.

  Upon receiving a re-learning request from the camera 10, the DB management unit 115 creates a population set from image data stored in the model DB 130, and re-learns the like model learning unit 114 to generate an inference model. Let In addition, the DB management unit 115 may act as a proxy for confirming whether to perform inference performed by the learning management unit 1d of the camera 10 or the like model learning unit 114 of the external device 20.

  The communication unit 120 includes a communication circuit and can communicate with the PC 250 through the network 400. Of course, it may be possible to communicate directly with the camera 10. The captured image 31 is acquired from the PC 250 through the communication unit 120 (captured image UP), and the model information (inference model) 32 generated by the favorite model learning unit 114 is transmitted to the PC 250. Further, guide information 33 for displaying advice for taking an image based on the model information 32 is also transmitted to the PC 250. The PC 250 that has received the model information 32 and the guide information 33 can output the inference model and the guide information to the camera 10.

  The image recording unit 140 is an image database (DB), and not only the user A but also many users can store captured images in the image recording unit 140. In the example shown in FIG. 9, a large number of dog images 141 b and cat images 141 c are recorded in the image recording unit 140 as the image data 141 by classification. That is, the image data recorded in the image database is classified. The classification is an example, and there are various other classifications.

  Among the images 141b and 141d, the popularity information 142 is associated with the image with “Like”. In addition, when the shooting information (for example, Exif information) of the image is associated as the accompanying information 143 and the popularity information 142 is associated, the profile information of the person to whom the image contributor information, “like” is added. Various information such as follower information is associated.

  The model database (DB) 130 stores a first rice model 131 to an nth rice model 13n. The favorite model stored in the model DB 130 is an inference model that has already been learned by the favorite model 114.

  In the first rice model 131, an image feature input 131a, a first feature extraction 131b, a first feature learning 131c, and an output 131d are stored. The first rice model 131 corresponds to the learning model shown in FIG. 1B. The first image feature input 131a is a feature of an image input to the input layer I. The first feature extraction 131b is information such as the position, brightness, and color of a pixel extracted from the image data input to the input layer I. The first feature learning 131d is the strength of connection (weighting) between the neurons. The output 131d stores the result of deep learning. As described above, the tendency determination unit 113 examines the tendency of the image recorded in the camera 100 of the user A, searches for a favorite model that is close to the user A's preference, and is recommended to the user A, and transmits it to the PC 250. .

  Next, the configuration of the camera 10 will be described with reference to FIG. The camera 10 includes an image processing and control unit 1, an imaging unit 2, a clock unit 3, a recording unit 4, an accessory information acquisition unit 5, an operation determination unit 6, a sensor such as an attitude sensor 7a, a display unit 8, a communication unit 9, and the like. Have.

  The image processing and control unit 1 is provided with a display control unit 1a, a recording control unit 1b, and a shooting time information adding unit 1c. The control unit 1 is a processor having a CPU (Central Processing Unit), peripheral circuits, a memory, and the like, and the CPU controls each unit of the camera 10 according to a program stored in the memory. Each of the units 1a to 1c in the image processing and control unit 1 is mainly realized by a program and a peripheral circuit, but may be realized entirely by a peripheral circuit. The image processing and control unit 1 has an image processing circuit and performs image processing on the image data input from the imaging unit 2.

  The display control unit 1a has a display control circuit, displays a through image based on image data from the imaging unit 2, reproduces and displays a captured image recording unit in the recording unit 4, and an inference model recorded in the learning result recording unit 4b. The display control when performing various image displays on the display unit 8 such as the display of the result of inference using, the guide display recorded in the guide recording unit 4c, and the like is performed.

  The recording control unit 1 b includes a recording control circuit and controls recording of image data and the like on the recording unit 4. That is, the captured image is recorded in the captured image recording unit 4d. The inference assistant 1b1 records the model information (inference model) 32 transmitted from the external device 20 in the learning result recording unit 4b and the guide information 33 in the guide recording unit 4c. In addition, when advice is adopted when the camera 10 uses the inference model, such information is also recorded (see S191 in FIG. 11A).

  The shooting time information providing unit 1c includes a status unit 1c1, a setting unit 1c2, version information 1c4, and an inference unit 1c5. The situation unit 1c1 is the shooting information of the camera 10, for example, focal length, focal position, exposure control value (aperture value, shutter speed value, ISO sensitivity value, etc.), subject type, subject size, subject light ray hit. Information indicating various shooting conditions, such as the state of attachment and use of accessories, is acquired.

  The setting unit 1c2 sets request information related to the theme that the user A wants to acquire “like” in the same manner as the setting unit 16 in FIG. The set request information is transmitted to the external device 20 through the communication unit 9. The good model learning unit 114 of the external device 20 performs deep learning based on this request information, and an inference model as a learning result is transmitted to the camera 10 (see S211 and S231 in FIG. 11B). In addition, the feature of the image at the time of the inference at this time is recorded in the model DB 130 as the nth rice model. The version information 1c4 is accessory version information such as an interchangeable lens.

  The inference unit 1c5 performs inference using an inference model from the external device 20 in the same manner as the inference engine 13 in FIG. The inference result made during the through image display is displayed on the display unit 8. By referring to the inference result, the user A can perform shooting according to the intention set by the setting unit 1c2. The inference unit 1c5 functions as an inference unit that determines whether the image data input by the image input unit is evaluated by a third party based on the inference model received by the receiving unit.

  Similar to the learning management unit 17b (see FIG. 5A), the learning management unit 1d monitors the inference result using the inference model and requests the learning device to relearn. This confirmation operation is also performed when the user instructs confirmation of whether re-learning is required manually (see S183 and S185 in FIG. 11A).

  Similar to the image input unit 11 in FIG. 5A, the imaging unit 2 includes a photographic lens, an imaging element, an imaging control circuit, an image data processing circuit, and the like, and generates image data of a subject. The image data is subjected to image processing in the image processing and control unit 1. Further, the imaging unit 2 outputs information such as field angle information and distance information of the photographing lens to the image processing and control unit 1. These pieces of information are used by the situation unit 1c1 when determining the shooting situation. If the taking lens mounted on the camera 10 is an interchangeable lens, the interchangeable lens information 5b may be input.

  The accessory information acquisition unit 5 includes an interchangeable lens information unit 5b. The interchangeable lens information unit 5b includes focal length information, focal position information, aperture value information, version information, and the like of the interchangeable lens. In addition to the interchangeable lens information, camera accessory information such as information on a mounted strobe device may be used. The accessory information acquisition unit 5 transmits the acquired information to the image processing and control unit 1. The situation part 1c1 determines a situation also using the acquired information.

  The clock unit 3 has a timekeeping function and a calendar function. The image data output from the imaging unit 2 is associated with date information and the like.

  The communication unit 9 has a communication circuit and communicates with the PC 250. A captured image is transmitted from the camera 10 to the external device 20 directly or indirectly through the communication unit 9 and receives model information 32 and guide information 33. The operation determination unit 6 determines operation states of various operation members such as a release button, a dial, and an operation button, and outputs a determination result to the image processing and control unit 1. The communication unit 9 functions as a transmission unit that transmits the shooting situation and / or the image recorded in the image recording unit to the learning device. The communication unit 9 functions as a receiving unit that receives an optimal inference model selected from the inference models generated by the learning device based on the shooting situation and / or the image recorded in the image recording unit.

  The posture sensor 7a may be an acceleration sensor, a gyro, or the like, and detects the posture of the camera 10. A GPS (Global Positioning System) 7b measures the position of the camera 10 by a satellite positioning system. The direction sensor 7c has a direction magnet and detects the direction in which the photographing lens of the camera 10 is facing. The outputs of the attitude sensor 7a, the GPS 7b, and the azimuth sensor 7c are output to the image processing and control unit 1, and the status unit 1c1 determines the status of the camera 10 based on the outputs of these sensors. Although all of attitude sensor 7a, GPS 7b, and direction sensor 7c may be provided, any of them may be omitted.

  The display unit 8 includes a display panel and the like, similarly to the display unit 15 in FIG. 5A, and performs display based on the image data acquired in the imaging unit 2. In addition, the inference result is displayed in the inference unit 1c5. In addition, display based on guide information recorded in the guide recording unit 4c may be performed. The request set in the setting unit 1c2 may be displayed as shown in FIG. The touch panel 8 b is provided on the front side of the display unit 8 and detects that the user A has touched the display surface of the display unit 8. This detection result is output to the image processing and control unit 1.

  The recording unit 4 includes a learning result recording unit 4b, a guide recording unit 4c, and a captured image recording unit 4d. The learning result recording unit 4b is set by the setting unit 1c2, and the result (inference model or mid-learning model) that is deeply learned by the rice model learning unit 114 is recorded. Further, a learning result (inference model) recommended by the tendency determination unit 113 is also recorded in accordance with the situation of the camera 10 detected by the situation unit 1c1.

  The guide recording unit 4 c records the guide display 33 transmitted from the external device 20. In this guide display, data for performing a guide display which serves as a reference to the user A when the generated inference model deeply learned by the rice model learning unit 114 is executed is recorded. The guide display may be a sample image that is evaluated as “like” in addition to advice on shooting operations such as an exposure control value and a focal length.

  Note that all or part of the CPU 110 in the control unit 110, image processing and control unit 1 may be realized by a CPU (Central Processing Unit) and a program code, such as a DSP (Digital Signal Processor). It may be realized by a circuit executed by program code, may be a hardware configuration such as a gate circuit generated based on a program language described by Verilog, and is executed by a hardware circuit. But of course it does n’t matter. Further, a part of the CPU function may be realized by a circuit executed by a program code such as a DSP, or may be a hardware configuration such as a gate circuit generated based on a program language described by Verilog. Also, it may be realized by a hardware circuit.

  Next, the operation of the learning system according to the present embodiment will be described using the flowcharts shown in FIGS. 11A to 12B.

  The flowcharts shown in FIGS. 11A and 11B show the operation of the camera 10, and the image processing and control unit 1 executes the flowcharts according to the program stored in the memory. When the camera control flow starts, it is first determined whether or not the shooting mode is set (S171). In the present embodiment, if the shooting mode is the default mode and no other mode is set by an operation member or the like, it is determined as the shooting mode.

  If the result of determination in step S171 is that the camera is in shooting mode, then various sensors are activated and accessory determination is performed (S173). Here, sensors such as the attitude sensor 7a, the GPS 7b, and the direction sensor 7c are activated, and the sensor detection result is input. Further, the accessory information acquisition unit 5 determines whether an accessory (for example, an interchangeable lens) is attached to the camera 10. Various information such as focal length is acquired from the interchangeable lens.

  Next, imaging is performed and a through image is displayed (S175). Here, image data is acquired by the imaging unit 2, and the image processing and control unit 1 performs image processing to display a through image on the display unit 8.

  Subsequently, inference is performed (S177). As will be described later, since the inference model generated by the rice model learning unit 114 is received from the external device 20 (see S213), in this step, the inference unit 1c5 performs inference using the received inference model. Do. In this inference, for example, an image displayed as a through image may receive a high evaluation (evaluation of “like”) from a third party under a request (condition) set by the user A. Whether or not.

  Once the inference is performed, it is next determined whether or not the specific condition is met (S179). Here, it is determined whether a through image display and an accessory (for example, an interchangeable lens) match a specific condition (request) set by the user A.

  If the result of determination in step S179 is that the specific conditions are met, a corresponding guide UI (User Interface) is displayed (S181). Here, when the result of inference in the inference unit 1c5 is expected to receive a high evaluation by a third party (evaluation of “like”), the display unit 8 together with the through image “user A has set Displays that the condition is met and is likely to be evaluated as “like”. The match may not be a complete match. In this case, guide display may be performed for shooting for complete matching.

  On the other hand, if the result of determination in step S179 does not match the specific condition, that fact is not displayed. However, when it is not likely to be evaluated as “like”, the display may be performed based on the guide information 33 recorded in the guide recording unit 4c. As guide information in this case, for this theme, advice is given on how to shoot as “Like”. For example, the difference between the image evaluated as “good” and the current through image may be displayed as an advice. That is, an aperture value, a shutter speed value, a shooting direction, a focal length, and the like for shooting an image similar to the image evaluated as “good” are displayed. In addition, a sample image evaluated as “like” may be displayed.

  If the corresponding guide UI is displayed in step S181 or if the result of determination in step S179 is that the specific conditions are not met, it is next determined whether or not a user guide has been adopted (S183). As described above, the user guide is displayed on the display unit 8 based on the inference result in step S177. Even when the specific condition is incompletely met or not met, a shooting guide that can be evaluated by a third party is provided. However, in such a case, the user may ignore the guide and perform shooting different from the guide. In this step, it is determined whether or not the user has adopted the guide displayed on the display unit 8.

  If the result of determination in step S183 is that the user guide has not been adopted, learning with new teacher data is requested (S185). The reason that the user did not adopt the guide is presumed that the reasoning does not match the current situation. Therefore, the DB management unit 115 of the external device 20 is requested to create new teacher data and perform the relearning by the like model learning unit 114. Note that the external device 20 may perform deep learning using pre-renewed teacher data and store the generated inference model in the model DB unit 130 in some cases. In this case, the inference model stored in the model DB 130 is selected and transmitted to the camera without re-learning.

  If the learning that renewed the teacher data is requested in step S185, or if the user guide is adopted as a result of the determination in step S183, it is next determined whether or not to shoot (S187). Here, the determination is made based on whether or not the release button of the camera 10 is fully pressed. The user A can refer to the corresponding guide UI display in step S61 depending on whether or not to perform shooting. If the result of this determination is that there is no shooting, processing returns to step S171 and the above-described operation is executed.

  As a result of the determination in step S187, in the case of shooting, it is next determined whether or not shooting is performed using the advice (S189). When the user A performs photographing according to the corresponding guide UI display displayed in step S181, the user A displays that fact by the operation member or the touch panel. Or, if it can be determined based on various sensor outputs and / or operation states of operation members, etc., that the corresponding guide UI display, such as aperture, shutter speed, shooting direction, focal length, etc., can be automatically advised You may make it determine with having employ | adopted.

  If the advice is adopted as a result of the determination in step S189, the inference assistant 1b1 records the image and the situation at that time, and records the employment guide information in association with the recording unit 4 (S191). In this case, the image and the shooting situation are recorded, and the adopted advice (guide information) is recorded in association with the image. By accumulating the result of whether or not to accept the advice, it is possible to determine the advice that the user who uses the camera 10 likes and does not like. The advice that has not been adopted is considered to be unacceptable for the user. Such extra advice is often bothersome for the user, and the same advice may not be displayed when the same situation occurs next time.

  In addition, in step S191, by recording what kind of inference model is used, it is possible to record as evidence that the image is obtained only by its own pass / fail judgment or by a guide. . As a result, it is preferable to perform a new learning when the evaluation is low despite taking a picture according to the guide. As a result, it can be used as a trigger for renewal of learning, which is a feature of the present embodiment. Specifically, if a large number of such images are posted but are not evaluated as good, you may think that this inference model is out of date, and no good teacher data in the past. You might also say that.

  On the other hand, if the result of determination in step S189 is that no advice has been adopted, the image and status are recorded (S193). By recording the images and situations where the advice was not adopted, it can be used as reference for the next inference. Note that the re-learning request in step S185 may be executed in step S193. By recording such a situation, it is possible to reflect the determination of the inference model required by the user or the creation of the necessary teacher data population. In addition, when such an image is uploaded, if it is not evaluated as “like”, it is the user's responsibility, and it becomes reference data when the camera or the external device determines that it is better to use the inference engine.

  Once recording has been performed in step S191 or S193, it is next determined whether or not a predetermined time has passed (S195). It is determined whether a predetermined time has elapsed after shooting, for example, whether a so-called REC view display time has elapsed. As a result of this determination, if the predetermined time has not elapsed, the image taken in step S187 is displayed on the display unit 8. On the other hand, when a predetermined time has passed as a result of the determination, the process returns to step S171 to execute the above-described operation.

  Returning to step S171, if the result of this determination is that the camera is not in shooting mode, it is determined whether or not it is in playback mode (S201). The reproduction mode is a mode for reproducing and displaying the captured image recorded in the recording unit 4 on the display unit 8. This is executed when the user A selects a playback mode by operating a playback button or the like. If the result of determination in this step is that the playback mode has been selected, the playback mode is activated (S203). When the reproduction mode ends, the process returns to step S171.

  If the result of determination in step S201 is not playback mode, it is determined whether or not it is image transmission mode (S205). The image transmission mode is a mode in which the captured image 31 is transmitted from the camera 10 to the external device 20 through the PC 250. The image transmission mode is set by the user A operating the image transmission button or the like. If the image transmission mode is selected as a result of the determination in this step (see S233 in FIG. 12A), the operation is performed in the transmission mode (S207). When the transmission mode ends, the process returns to step S171.

  If the result of determination in step S205 is not image transmission mode, shooting status data and the like are transmitted (S209). When the mode is not the image transmission mode, the remaining mode, that is, the inference model acquisition mode is set. Here, the information regarding the photographing situation collected by the situation unit 1c1 and the image recorded in the recording unit is transmitted to the external device 20 through the PC 250. The external device 20 searches for an inference model most suitable for the user A based on the received information regarding the shooting situation, and transmits it to the camera 10. The search for the optimum inference model will be described later with reference to FIG. 12B. As in the second embodiment, the user may create a request and request the external device 20 for learning.

  When the shooting situation data or the like is transmitted, it is determined whether or not an inference model or the like has been acquired (S211). As described above, the external device 20 sends back the optimal inference model based on the information regarding the shooting situation.

  If the inference model or the like cannot be acquired as a result of the determination in step S211, the process returns to step S171 and the above-described operation is executed. On the other hand, when an inference model or the like is acquired, the acquired inference model or the like is recorded in the recording unit 4 (S213).

  The flowcharts shown in FIGS. 12A and 12B show the operation in the external device 20, and the control unit 110 executes the flowcharts according to the programs stored in the memory. When the flow of the image management server starts, it is first determined whether or not to view an image (S221). In some cases, the user A desires to view an image recorded in the image recording unit 140 on the PC 250 or the camera 10 or the like. In this case, there is an access to that effect from the PC 250 or the like to the access determination unit 111. When accessing the image management server, the PC 250 or the like transmits image search conditions and the like.

  If the result of determination in step S221 is that the image is to be viewed, a list of images that have been uploaded according to the search conditions is displayed (S223). In this case, the search is performed from the images recorded in the image recording unit 140 according to the search condition. When the search is completed, the display control unit 112 transmits the data to the PC 250 or the like after allowing the selected images to be displayed as a list.

  Subsequently, it is determined whether or not a favorite image has been selected (S225). If there is a favorite image among the images displayed on the display unit 8 by the user A, the user A clicks “Like” and transmits the favorite image. Here, it is determined whether or not the user A has evaluated “like”.

  If the result of determination in step S225 is that a selection has been made as a favorite image, the profile of the person who selected “like” is associated with the image (S227). Here, the person who selected “like” (that is, user A), the person who evaluated “like” (that is, the contributor of the selected image), and the selected image are associated with each other. Because certain people often rate a particular person ’s image as “like”, if you record both the profile of the person who chose “like” and the profile of the person who ’s rated “like”, The relationship between the two can be estimated. That is, it is preferable that the evaluation is performed in consideration of the relationship between the submitting person and the evaluated person. Furthermore, since there are people who give a high evaluation to any image and people who do not give a high evaluation, the profile information may include the personality of such an evaluator.

  Subsequently, aggregation such as categorization is performed (S229). In step S227, the relationship between each image and each evaluator is determined. An image evaluated as “Like” may be replaced by various people. By analyzing this evaluation, what kind of evaluation is given by the person of what profile? I understand. Further, by analyzing the profiles of the evaluator and the evaluator, it is possible to determine what type of image gives or receives an evaluation. Furthermore, it is possible to perform categorization and the like such that people with such a common profile tend to like such images.

  In step S229, as described above, the control unit 110 performs classification or the like using the information associated in step S227. From information attached to the image (for example, Exif information) etc., information such as focal length, focal position, aperture value, shutter speed value, photographing lens direction, flash device usage at the time of photographing is obtained. Classify based on information. Further, by analyzing the image, it may be classified according to the theme of the photo, for example, landscape, person, cooking, and the like. Further, if the profile of the poster (photographer) is attached to the image, the image may be classified according to the nationality, gender, age group, etc. of the poster. Moreover, when a comment is attached to the image, the image may be classified based on the comment.

  Once the totalization such as categorization has been carried out, it is next determined whether or not it is finished (S231). Here, the access determination unit 111 determines whether or not the access from the user A has ended. If the result of this determination is not end, processing returns to step S223 and the aforementioned operation is executed. On the other hand, in the case of completion, the process returns to step S221 to execute the above-described operation.

  If the result of determination in step S221 is that there is no image viewing, it is determined whether or not a camera is connected (S233). Here, it is determined whether it is connected to the image management server directly from the camera 10 or via the PC 250.

  If the result of determination in step S233 is camera connection, it is determined whether or not the image is organized (S235). Here, it is determined whether or not to organize the captured images recorded in the image recording unit 140, or to record the captured images in the image recording unit 140 and organize them. Since the user A designates the work content when connected to the external device 20, the determination is made based on the work content.

  If the result of determination in step S235 is image organization, processing such as photo organization, folder division, albuming, posting, etc. is executed (S237). When the user A wants a third party to view the image, the image is posted. Such a posted image can be made into an environment where a community is created by SNS or the like and can be easily reviewed. As described above, since it is better to consider the human relationship between the evaluator and the evaluator when evaluating the image, the profile of the user A is also recorded at this time. By recording the profile, it is possible to eliminate the case of evaluating “like” simply by acquaintance. However, even if such “likes” are not excluded, if it is clear whether there are many “likes” or not, images that are not evaluated subjectively even if they are learned in total are distinguished. be able to. When the photo organization in step S237 is performed, the process returns to step S221 to execute the above-described operation.

  If the result of determination in step S233 is that the camera is not connected, it is determined whether it is a request or update (S241). The camera 10 may request transmission of the inference model (S211, S213), and may request to update the inference model by renewing and re-learning the inference model generation set. Yes (S185). In this step, it is determined whether or not there has been such a request or update request. If there is no request as a result of this determination, the process returns to step S221 to execute the above-described operation.

  If the result of determination in step S241 is that there is a request or update request, a good image determination is performed (S243). Here, for the user who can access the external device 20, an operation for generating an inference model is executed in steps S143 to S261. First, in this step, an image that has been highly evaluated by a third party (evaluated as “like”) is extracted from the images recorded in the image recording unit 140. This image is used as an input image (set to the input layer I) when deep learning is performed by the rice model learning unit 114.

  If the rice image determination is performed, next, all-field deep learning is performed (S245). Here, in general, deep learning is performed to generate an inference model of an image to be evaluated as “like”. In this deep learning, as described with reference to FIG. 1B, the neutral layer N11 of the intermediate layer is output so that the output layer O outputs the rice when the images (the mother set) of all fields are input to the input layer I. Adjust the strength of N2n bond, weighting, etc. The adjusted neutral layer strength, weighting, and the like are the inference model. In deep learning, the shooting situation is also reflected in the creation of the inference model. This is for use in recommending an optimal inference model (see S275).

  When all-field deep learning is performed, a first inference model is created (S247). The intermediate layer neutral connection strength, weighting, and the like when deep learning is finished are recorded in the model DB unit 130 as the first model. Note that this inference model is a general-purpose inference model. When an inference model suitable for a specific category has not been generated, this general-purpose inference model is transmitted to an external device.

  Once the first inference model is created, a first guide is created (S249). When the first inference model is created, first guide information for enabling photography with the first inference model is created. The first guide is created by detecting a difference between an image to be evaluated and an image not to be evaluated. In this detection, a difference is detected by using a learning inference model for evaluating an input image at the time of photographing or observation, and a learning inference model to which category the input image belongs. It may be. In any case, it is only necessary to create an inference model in which learning is performed so that a difference between an evaluated image and an image that has not been evaluated is detected.

  In steps S243 to S249, a general reasoning model (dictionary) and guide with high versatility can be created. However, a customized inference model may be obtained based on the narrowed request. For example, the inference model may be generated by renewing the teacher data and re-learning, and the first inference model may be updated.

  Once the first guide is created, transmission and the like are performed, the learning date is recorded, and the history is managed (S251). Here, the first guide created in step S249 is transmitted to a camera or the like. In addition, the date (date information) on which deep learning is performed to create this inference model is recorded in the model DB unit 130 or the DB management unit 115. In addition, as the history, various information such as the conditions for creating the inference mother set, conditions relating to the mother set, information about the teacher data, the destination of the inference model, etc. And recorded in the model DB unit 130 or the DB management unit 115. By managing the history, conditions such as the timing of the next renewal and relearning can be determined.

  If history management or the like is performed in step S251, it is next determined whether or not the number of “likes” has increased in a specific category (S253). For example, there are cases where the number of “likes” increases for “photos evaluated by women in their 20s in France”. In this step, the number of “likes” is watched for each category, and a determination is made based on this number. If the result of this determination is that the number of likes in the specific category has not increased, processing returns to step S221 and the above-described operation is executed.

  If the result of determination in step S253 is that the number of “likes” has increased in a specific category, deep learning is performed for a specific field (S255). Here, for a specific category in which the number of “likes” has increased, the population of image data is limited and deep learning is performed.

  When deep learning in a specific field is performed, an nth inference model is created (S255). The intermediate layer neutral connection strength, weighting, and the like when deep learning is completed are recorded in the model DB unit 130 as the nth model. A plurality of inference models can be recorded, and the nth inference model, the (n + 1) th inference model, the n + 2 inference model,.

  Once the nth model is created, an nth guide is created (S259). When the nth inference model is created, advice for enabling photography with the nth inference model is created. The advice may include various advices such as an exposure control value, a focal length, a focal position, and a photographing lens direction. Further, it may be a sample photograph that can be taken with the nth model.

  When the nth guide is created, next, as in step S251, transmission is performed, the date at the time of learning is recorded, and history management is performed (S261). Here, the nth guide created in step S259 is transmitted to a camera or the like. In addition, the date (date information) on which deep learning is performed to create this inference model is recorded in the model DB unit 130 or the DB management unit 115. In addition, as the history, various information such as the conditions for creating the inference mother set, conditions relating to the mother set, information about the teacher data, the destination of the inference model, etc. And recorded in the model DB unit 130 or the DB management unit 115.

  The history management in step S261 is related to an inference model for a specific category, and this inference model is learned from teacher data created for recruitment under conditions in which a specific scene limitation is included in the request condition. It is preferable to specify scene information and the like strictly. This identification may be performed by the camera, but the external device may not classify the image according to the specific separation. Therefore, unlike the general-purpose model, it is necessary to devise a method for creating the population. . Therefore, it is preferable that the history management here is information that can correctly create a population. For example, even if you want to specify a scene such as a starry sky, the external device may only have a night view, and in order to eliminate such inconsistencies, further image features, elevation information, and attached comments from the night view Since it is necessary to narrow down by such as, history management becomes important. In addition, because of the difficulty in setting the population, there is a high possibility that the previous learning has failed, so it may be better to recommend updates more frequently by the user. If history management etc. are performed in step S261, it will return to step S221 and will perform the above-mentioned operation | movement.

  Returning to step S235, if the result of determination in this step is that the image is not organized, the shooting status and the tendency of the captured image are determined (S271). In step S209 (FIG. 11B), the shooting situation collected by the situation unit 1c1 and the shot image recorded in the shot image recording unit 4d are transmitted to the external device 20. In this step, the tendency determination unit 113 determines the shooting situation of the camera 10 and the tendency of user A's shooting preference.

  Next, the teacher data is updated (S273). Here, the teacher data is renewed in accordance with the change in the shooting situation and the tendency of the shot image determined in step S271. That is, the condition of image data to be extracted as teacher data for deep learning is changed.

  When the teacher data is renewed, it is next determined whether or not there is an optimum model (S277). The trend determination unit 113 performs the trend determination performed in step S271. This tendency determination searches for a favorite model having a high degree of coincidence with image feature input, feature extraction, feature learning, etc. in the favorite model such as the first favorite model 131, and sets this as the optimum model. In this step, it is determined whether or not there is an optimum model recommended to the user A from the model DB 130.

  If the result of determination in step S275 is that there is an optimal model, the optimal model is recommended (S277). Here, the model extracted in step S275 is set as the optimum model.

  Subsequently, it is determined whether or not there is a user request (S279). Here, it is determined whether or not there is an optimal model transmission request from user A. Note that when the shooting situation or the like is transmitted by the operation of the user A, it may be automatically determined that the user has requested.

  If the result of determination in step S279 is that there is a user request, the old model and guide are deleted (S281). Here, the old inference model and the old guide sent last time are deleted. The old inference model, etc. is deleted when considering the energy-saving design, the heat generation problem associated with it, portability and space-saving, and having an infinite number of inference models corresponding to all usage scenes This is because it is not suitable for manufacturing. If the apparatus is in a field where the above-mentioned problems are permitted, it may not be erased. You may devise a method to lower the priority of the inference model because there is a possibility that the unnecessary judgment is inconsistent.

  Subsequently, a new model and a new guide are transmitted (S285). The optimum model (new model) set in step S277 and the guide created based on this optimum model are transmitted to the camera 10. As described above, the camera 10 records the received inference model and guide (see S213).

  When the new model and the new guide are transmitted, or when there is no optimum model as a result of the determination in step S275, or when there is no user request as a result of the determination in step S279, the process returns to step S221 to execute the above-described operation. .

  Thus, in the third embodiment of the present invention, when the user does not adopt the guide or advice inferred using the inference model, the external device 20 is requested to regenerate the inference model. (S183, S185, S193). For this reason, when the shooting situation, shooting preference, and the like change, the inference model can be updated, and guidance and advice based on the inference model adapted to the user's preference can be performed.

  As described above, in each embodiment of the present invention, an inference model is set in the inference unit (for example, see S1 in FIG. 3, S81 in FIG. 6, and S213 in FIG. 11B), input image data, and inference Using the model, an output for displaying a guide to the user is obtained (for example, see S3 in FIG. 3, S55 in FIG. 6, and S177 in FIG. 11A), and the guide display is performed based on the output of the inference unit. (For example, see S9 in FIG. 3, S57 in FIG. 6, and S181 in FIG. 11A). Also, the update of the inference model is determined, and based on the result of the determination, the external learning device is requested to update the inference model or whether the inference model needs to be updated (for example, FIG. 3). S15, S19, S63 in FIG. 6, S183, S185 in FIG. 11A). For this reason, the inference model can be updated so that an appropriate guide or the like can be performed even if the situation or the like changes.

  Further, in each embodiment of the present invention, a control unit in a device such as the camera 10 or the industrial endoscope 301, and a program incorporated in the control unit are incorporated in a computer that can cooperate with a social network system. Execute the method. The computer embedded software is software that operates by being installed in an embedded device, and the computer embedded method means a method realized by this software. In addition, the social network system allows content items to be posted (posted) at any time in time, connected users can be added, and the posted content items can be posted at any time in the social network system. It is possible to evaluate the quality (may include preferences) from the user. For example, as described above, when an image is uploaded to a social network, it becomes possible for a third party to browse, and a quality evaluation such as “like” is received from a third party, and the uploaded image can be associated with the quality evaluation. it can. The computer-embedding method according to the present embodiment also includes an inference model for inferring quality evaluation from the characteristics of a content item using, as teacher data, a specific content item accompanied by quality evaluation information. However, it can be determined that the inference result of the initial aim can no longer be output due to a change in the content item added over time or a change in the evaluation tendency of the content item (for example, FIG. 6). S63 to S69, S111 to S121 of FIG. 7, S151 to S165 of FIG. 8, S185 of FIG. 11A, S241 to S261 of FIG.

  In addition, a computer incorporation method capable of cooperating with the above-described social network system, the computer incorporation method includes a step of recording, as a history, conditions when machine learning is performed using content items as teacher data. Then, new teacher data is generated according to the recorded history (see, for example, S153 in FIG. 8). For example, a mother set may be generated according to conditions such as a request when deep learning was performed last time, and an inference model may be generated using this mother set.

  Further, it is a computer incorporation method capable of cooperating with the above-mentioned social network system, and this computer incorporation method uses a condition when a specific user performs machine learning using a content item as teacher data as a history. Recording step (see, for example, S251, S261, etc. in FIG. 12B). For example, when an inference model is generated, it is recorded as a history of conditions used for generation, for example, date and time, request (scene, evaluator, etc.), population information, and the like. By managing this history, it can be used the next time an inference model is generated.

  Also, it is a computer-embedded method that can be linked with a social network system. The social network system allows content items to be posted (posted) at any time, and connected users can be added. The posted content items can be quality evaluated (may include preferences) from users of social network systems at any time. Also, the computer-embedding method records as a history when a specific user performs machine learning using a content item as teacher data (for example, to eliminate the inconsistency of categories between the camera and the external device). Step (see, for example, S261 in FIG. 12B).

  In each embodiment of the present invention, learning in the learning device is deep learning. However, the learning is not limited to this, and any learning that uses artificial intelligence such as machine learning may be used. In addition, the inference model is performed when the learning device receives a request for generating an inference model from a camera or the like. However, an inference model may be generated in a camera or the like by incorporating a learning device and cooperating with an external database or the like. In addition, an inference engine was installed in the camera and so on to make inferences. However, the present invention is not limited to this, and an inference engine may be arranged on an external server provided in the cloud, always connected via the Internet or the like, and display or the like may be performed based on the inference result of the inference engine in the cloud.

  In each embodiment of the present invention, a digital camera has been described as an apparatus for shooting. However, a digital single lens reflex camera, a mirrorless camera, or a compact digital camera may be used as a camera. A camera for moving images such as a camera may be used, and a scientific device such as a mobile phone, a smartphone, a personal digital assistant (PC), a personal computer (PC), a tablet computer, a game machine, etc., a medical camera, a microscope, etc. It may be a camera for a car, a camera mounted on a car, or a camera for monitoring. In any case, the present invention can be applied to any device that can input image data and can directly or indirectly access an external image database.

  In recent years, artificial intelligence that can collectively determine various determination criteria is often used, and improvements that collectively perform each branch of the flowchart shown here are also included in the scope of the present invention. Needless to say, it is a part of this. If the user can input good or bad for such control, the user's preference can be learned and the embodiment shown in this application can be customized in a direction suitable for the user.

  Of the techniques described in this specification, the control mainly described in the flowchart is often settable by a program and may be stored in a recording medium or a recording unit. The recording method for the recording medium and the recording unit may be recorded at the time of product shipment, may be a distributed recording medium, or may be downloaded via the Internet.

  Further, in one embodiment of the present invention, the operation in the present embodiment has been described using a flowchart. However, the processing procedure may be changed in order, or any step may be omitted. Steps may be added, and specific processing contents in each step may be changed.

  In addition, regarding the operation flow in the claims, the specification, and the drawings, even if it is described using words expressing the order such as “first”, “next”, etc. It does not mean that it is essential to implement in this order.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Image processing and control part, 1a ... Display control part, 1b ... Recording control part, 1b1 ... Inference assistance part, 1c ... Shooting time information provision part, 1c1 ... Situation part DESCRIPTION OF SYMBOLS 1c2 ... Setting part, 1c4 ... Version information, 1c5 ... Inference part, 1d ... Learning management part, 2 ... Imaging part, 3 ... Clock part, 4 ... Recording part 4b: Learning result recording unit, 4c: Guide recording unit, 4d: Captured image recording unit, 5: Accessory information acquisition unit, 5b: Interchangeable lens information unit, 6: Operation Determination unit, 7a ... posture sensor, 7b ... GPS, 7c ... direction sensor, 8 ... display unit, 8b ... touch panel, 9 ... communication unit, 10 ... camera, 11 ... Image input part, 12 ... Inference engine, 13 ... Storage part, 13a ... Learning product, 14. · Communication unit, 15 ··· Display unit, 16 ··· Setting unit, 17 · · · Control unit, 20 · · · external device, 21 · · · external image DB, 22 · · · communication unit, 23 ··· Image classification unit, 23a ... follower profile, 23b ... theme, 25 ... learning unit, 26 ... communication unit, 27 ... population generation unit, 28 ... input / output modeling unit , 29 ... control unit, 30 ... server, 31 ... photographed image, 32 ... model information, 33 ... guide information, 110 ... control unit, 111 ... access determination unit, DESCRIPTION OF SYMBOLS 112 ... Display control part, 113 ... Trend determination part, 114 ... Like model learning part, 120 ... Communication part, 130 ... Model DB part, 131 ... 1st rice model, 131a ... Image feature input, 131b ... First feature extraction, 131c ... 1 feature learning, 131d ... output, 140 ... image recording unit, 141 ... image data, 141b ... dog image data, 141c ... cat image data, 142 ... popular information, 143 ... Accompanying information, 200 ... Learning device, 200A ... Learning device, 200B ... Learning device, 201A ... Old data set, 201B ... New data set, 203 ... Inference engine , 203A ... inference engine, 203B ... inference engine, 204A ... learning start unit, 205A ... input unit, 205B ... external output unit, 206A ... output unit, 207A ... learning Management control unit, 210 ... external device, 250 ... PC, 301 ... industrial endoscope, 302 ... insertion unit, 302a ... tip, 302b ... bending part, 302c・ ・Flexible tube, 303 ... Imaging unit, 311 ... Tube, 311a ... Tube, 311b ... Tube, 312 ... Seam, 313 ... Rust, 400 ... Net

Claims (20)

An input unit for inputting an inference model;
An inference unit that inputs image data and obtains an output for performing a guide display to the user using the inference model;
A display unit for performing the guide display based on the output of the inference unit;
An update confirmation unit that confirms the update of the inference model and requests an external learning device to update the inference model or determines whether the inference model needs to be updated based on the result of the confirmation When,
A learning management apparatus comprising:   The update confirmation unit acquires the latest teacher data, detects the reliability of the currently used inference model using the latest teacher data, and if the reliability is low, the external learning The learning management apparatus according to claim 1, wherein the apparatus requests the apparatus to update the inference model.   The update confirmation unit requests the external learning device to update the inference model when a predetermined time elapses after requesting the external learning device to update the inference model, or updates the inference model. The learning management apparatus according to claim 1, wherein the learning management apparatus requests to determine whether or not it is necessary.   The learning management device according to claim 1, wherein the update confirmation unit monitors a change in the current state and requests the external learning device to update the inference model when the current state changes.   The learning management apparatus according to claim 4, wherein the update confirmation unit determines that the current state has changed when the user does not adopt the guide display and does not reflect the guide display.   The learning management apparatus according to claim 1, wherein the update confirmation unit always confirms whether or not the inference model needs to be updated.   The learning according to claim 1, wherein the external device confirms whether or not the inference model needs to be updated, and the update confirmation unit makes a determination based on a confirmation result from the external device. Management device. A determination unit that determines whether or not a request for confirmation of the inference model has been input, or whether or not a predetermined time has elapsed since the last update of the inference model;
The determination unit inputs a request for confirmation of the inference model, or if it is determined that a predetermined time has elapsed since the last update of the inference model, a population generation unit that sets a learning population from the database; ,
Using the learning population created by the population creation unit, based on the reliability of the requested inference model, an update confirmation unit that determines the need for updating,
A learning management server characterized by comprising:   The learning management server according to claim 8, wherein the population generation unit sets a learning population from the database based on a condition used when a previous inference model is generated. A database that accumulates usable data when generating inference models by learning;
A time determination unit that determines whether or not a predetermined time has elapsed since learning to generate the inference model;
If the time determination unit determines that the predetermined time has elapsed, the database is searched based on the conditions used when the previous inference model was generated, and whether or not the upper data has fluctuated. A determination unit for determining whether or not
When the determination unit determines that the upper data is fluctuating, a recommendation unit that recommends updating of the inference model;
A learning management server characterized by comprising: A database that accumulates data that can be used to generate inference models by learning and that includes third party assessments;
A determination unit for determining whether or not the evaluation of the third party has changed,
When there is a change in the evaluation of the third party by the determination unit, a recommendation unit that recommends updating the inference model,
A learning management server characterized by comprising: Set the inference model in the inference section,
Input image data and use the above inference model to obtain an output for the guide display to the user,
Based on the output of the inference section, the guide display is performed,
Determine the update of the inference model, and based on the result of the determination, request an external learning device to update the inference model, or request whether the inference model needs to be updated,
A learning management method characterized by that.   The latest teacher data is acquired, and the reliability of the currently used inference model is detected using the latest teacher data. If the reliability is low, the external learning device is connected to the inference model. The learning management method according to claim 12, wherein an update is requested.   When a predetermined time has elapsed after requesting the external learning device to update the inference model, the external learning device is requested to update the inference model, or whether the inference model needs to be updated is determined. The learning management method according to claim 12, wherein a request is made.   13. The learning management method according to claim 12, wherein a change in the current state is monitored, and when the current state changes, the external learning device is requested to update the inference model.   The learning management method according to claim 15, wherein when the user does not adopt the guide display but reflects it in the operation, it is determined that the current state has changed.   The learning management method according to claim 12, wherein confirmation is always performed as to whether or not the inference model needs to be updated. A computer embedded method capable of cooperating with a social network system,
In the social network system, content items can be posted at any time, and the posted content items can be quality evaluated from the user of the social network system at any time,
In the computer embedded method, an inference model for inferring the quality evaluation from the feature of the content item using the specific content item accompanied by the quality evaluation information as the teacher data among the posted content items is as follows: It is determined that the inference result of the initial aim cannot be output due to the change of the content item added by the temporal transition and the temporal change of the evaluation tendency of the content item.   It has a step of recording as a history a condition when machine learning is performed using the content item as teacher data, and it is recommended that new teacher data be generated according to the recorded history The computer built-in method according to claim 18.   19. The computer-embedded method according to claim 18, further comprising a step of recording, as a history, conditions when a specific user performs machine learning using the content item as teacher data.

Images