JP2019200536A - Image acquisition apparatus, image acquisition method, and image acquisition program - Google Patents

Abstract

To improve the effectiveness of machine learning by recording information about the use of machine learning.SOLUTION: An image acquisition apparatus comprises: an image acquisition unit for acquiring images; an inference unit for performing inference using the image acquired by the image acquisition unit as an input with using a predetermined inference model; a presentation unit for presenting the inference result of the inference unit; a determination unit for determining whether or not the inference result is adopted; and a control unit for generating usage information regarding the use of the inference model on the basis of the determination result by the determination unit, and recording the generated usage information as metadata of the image acquired by the image acquisition unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image acquisition apparatus, an image acquisition method, and an image acquisition program that can acquire an image using machine learning.

  In recent years, machine learning such as deep learning has been used. Machine learning learns the characteristics, time-series information, spatial information, etc. of known input information and makes inferences based on the learning results to obtain inference results for unknown matters. That is, in machine learning, first, a learned model for enabling inference of a determinable output result is obtained from specific input information. Machine learning is one of the elemental technologies of artificial intelligence, and one of the algorithms is a neural network. This is because the human brain uses a network of nerve cells called neurons to provide various places such as images. It realizes a recognition process that mimics the recognition of the target object from the features of this, and tries to reproduce the neural network in the brain and the process using it on a computer. Deep learning is a neural network that is constructed in multiple layers. Using various features such as images as input, it is repeated trial and error to find out if it is useful for recognition, and the most effective recognition model is found and learned. To do.

  A large amount of information whose relation between input and output is known is used as learning data when generating a learned model so that an inference result can be obtained with high reliability. For example, when machine learning is realized by a neural network, a network design is designed so that an expected output is obtained with respect to a known input. A learned model (hereinafter also referred to as an inference model) obtained by such a process can be used independently of the learned neural network.

  Such machine learning can be used in various fields. For example, there is a Watson (trademark) system that performs natural language processing to read information and perform inference. Patent Document 1 discloses a technique for providing a mechanism, a method, a computer program, and an apparatus for identifying commonality between answer candidates generated by a question answering system such as a Watson system.

  Further, for example, it is conceivable to use such machine learning at the time of image acquisition by a photographing device or the like. The use of machine learning may facilitate acquisition of an image desired by the user.

Japanese Patent Laying-Open No. 2015-109068

  However, even if an inference model is used, an expected output may not be obtained depending on the difference between the data used for learning and the input information to be inferred. However, in the conventional image acquisition system, a technique for effectively using machine learning is not considered in consideration of such restrictions.

  It is an object of the present invention to provide an image acquisition device, an image acquisition method, and an image acquisition program that can effectively use an inference model based on machine learning by recording information about the use of machine learning.

  An image acquisition device according to an aspect of the present invention includes an image acquisition unit that acquires an image, an inference unit that performs an inference using the image acquired by the image acquisition unit as an input, using a predetermined inference model, and the inference The presenting part that presents the inference result of the part, the determination part that determines whether or not the inference result is adopted, and the usage information on the use of the inference model based on the determination result of the determination part are created and created A control unit that records usage information as metadata of the image acquired by the image acquisition unit.

  An image acquisition method according to an aspect of the present invention includes an image acquisition step of acquiring an image, an inference step of performing an inference using the image acquired in the image acquisition step as an input, using a predetermined inference model, and the inference The presenting step for presenting the inference result in the step, the determination step for determining whether or not the inference result is adopted, and the usage information on the use of the inference model based on the determination result in the determination step were created and created And a control step of recording usage information as metadata of the image acquired in the image acquisition step.

  An image acquisition program according to an aspect of the present invention includes: an image acquisition step for acquiring an image in a computer; an inference step for performing an inference using the image acquired in the image acquisition step as an input using a predetermined inference model; A presentation step for presenting the inference result in the inference step, a determination step for determining whether or not the inference result has been adopted, and use information on the use of the inference model based on the determination result in the determination step. And a control step of recording the created usage information as metadata of the image acquired in the image acquisition step.

  According to the present invention, by recording information about the use of machine learning, there is an effect that an inference model based on machine learning can be used effectively.

1 is a block diagram showing an imaging device including an image acquisition device according to a first embodiment of the present invention. Explanatory drawing for demonstrating dictionary 12a1, 12a2 memorize | stored in the memory | storage part 12a of the inference engine 12. FIG. 7 is a flowchart showing the operation of the image acquisition device 10. 5 is a flowchart showing the operation of the external device 30. Explanatory drawing for demonstrating operation | movement of 1st Embodiment. Explanatory drawing for demonstrating operation | movement of 1st Embodiment. The flowchart which shows the operation | movement flow employ | adopted in the 2nd Embodiment of this invention. The flowchart which shows the operation | movement flow employ | adopted in the 2nd Embodiment of this invention. FIG. 3 is an explanatory diagram illustrating a state in which a subject is imaged by the imaging device 20 of FIG. 1. Explanatory drawing which shows the captured image displayed on the display screen 15a of the display part 15. FIG. Explanatory drawing which shows a dictionary menu screen. Explanatory drawing which shows a dictionary menu screen.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing an imaging apparatus including an image acquisition apparatus according to the first embodiment of the present invention.
Machine learning, such as deep learning, is built in multiple layers, imitating that the human brain recognizes its objects from features such as images using a network of neurons called neurons. Therefore, since the obtained “inference model” causes a problem that input / output is performed in a black box, it is important to identify which “inference model” is used.
In this embodiment, in inputting predetermined information, whether or not an inference result using a predetermined inference model is adopted, and if inferred, which inference model is adopted is effective for inference. It improves the performance. In the present embodiment, information on the use of such an inference model (hereinafter referred to as inference model use information) is recorded. For example, image metadata synchronized with the timing at which the inference model is used. Inference model usage information is recorded. The inference model use information may be recorded as metadata of various types of information, not limited to images, and the inference model use information is recorded alone together with information for specifying a use scene, for example, time information. Also good.

  The user may be able to set the use of the inference model. In this case, by recording the inference model usage information as evidence, the inference model usage information can be used to determine whether or not the user's judgment regarding the use of the inference model is correct, and the validity of the inference model can be determined. It becomes a material. Inference model usage information can also be used to clarify the scope of application of an inference model.

  The imaging device 20 in FIG. 1 includes an image acquisition device 10. The imaging device 20 images a subject, and the image acquisition device 10 in the imaging device 20 records an image obtained by imaging. As the imaging device 20, not only a digital camera and a video camera but also a camera built in a smartphone or a tablet terminal may be adopted. Furthermore, as the imaging device 20, a microscope, an endoscope, a CT scanner, or the like may be adopted, and various imaging devices that obtain images using white light, ultraviolet rays, infrared rays, X-rays, ultrasonic waves, or the like are adopted. be able to. 1 illustrates an example in which the image acquisition device 10 is configured in the imaging device 20, the imaging device 20 and the image acquisition device 10 may be configured separately.

  As will be described later, the image acquisition apparatus 10 can use an inference model when acquiring an image. However, the image acquisition apparatus 10 may acquire an image using an inference model that is installed in advance. Alternatively, the inference model may be acquired from the external device 30. That is, the external device 30 is used as necessary.

  The imaging device 20 includes a control unit 11 and an imaging unit 22. The control unit 11 may be configured by a processor using a CPU or the like, and may operate in accordance with a program stored in a memory (not shown) to control each unit, or may be a part of a function by a hardware electronic circuit. Or you may implement | achieve all.

  The imaging unit 22 includes an imaging element 22a and an optical system 22b. The optical system 22b includes a lens and a diaphragm (not shown) for zooming and focusing. The optical system 22b includes a zoom (magnification changing) mechanism, a focus and a diaphragm mechanism (not shown) that drive these lenses.

  The image pickup device 22a is constituted by a CCD, a CMOS sensor, or the like, and an optical image of the subject is guided to the image pickup surface of the image pickup device 22a by the optical system 22b. The image sensor 22a photoelectrically converts the subject optical image to obtain a captured image (imaging signal) of the subject.

  The imaging control unit 11a of the control unit 11 can adjust the zoom, the aperture, and the focus by driving and controlling the zoom mechanism, the focus mechanism, and the aperture mechanism of the optical system 22b. The imaging unit 22 performs imaging under the control of the imaging control unit 11a, and outputs an imaging signal of a captured image (moving image and still image) to the control unit 11 serving as an image acquisition unit.

  The imaging device 20 is provided with an operation unit 13. The operation unit 13 includes a release button, a function button (not shown), various switches for shooting mode setting, parameter operation, etc., a dial, a ring member, and the like, and outputs an operation signal based on a user operation to the control unit 11. The control unit 11 controls each unit based on an operation signal from the operation unit 13.

  The control unit 11 captures a captured image (moving image and still image) from the imaging unit 22. The image processing unit 11b of the control unit 11 performs predetermined signal processing, for example, color adjustment processing, matrix conversion processing, noise removal processing, and other various signal processing on the captured image.

  The imaging device 20 is provided with a display unit 15, and the control unit 11 is provided with a display control unit 11 f. The display unit 15 has a display screen such as an LCD (Liquid Crystal Display), for example, and this display screen is provided on the rear surface of the housing of the imaging device 20 or the like. The display control unit 11f displays the captured image signal-processed by the image processing unit 11b on the display unit 15. Further, the display control unit 11f can display various menu displays, warning displays, and the like of the imaging device 20 on the display unit 15.

  The imaging device 20 is provided with a communication unit 14, and the control unit 11 is provided with a communication control unit 11 e. The communication unit 14 is controlled by the communication control unit 11 e so that information can be transmitted to and received from the external device 30. The communication unit 14 is capable of communication by short-range wireless such as Bluetooth (registered trademark) and wireless LAN such as Wi-Fi (registered trademark). Note that the communication unit 14 is not limited to Bluetooth or Wi-Fi, and can employ communication using various communication methods. The communication control unit 11 e can receive inference model information from the external device 30 via the communication unit 14.

  The control unit 11 is provided with a recording control unit 11c. The recording control unit 11c can compress the captured image after the signal processing and give the compressed image to the recording unit 16 for recording. The recording unit 16 is configured by a predetermined recording medium, and can record information given from the control unit 11 and output the recorded information to the control unit 11. As the recording unit 16, for example, a card interface can be adopted, and the recording unit 16 can record image data on a recording medium such as a memory card.

  In the present embodiment, the recording unit 16 includes an image data recording area 16a and a metadata recording area 16b, and the recording control unit 11c records image data in the image data recording area 16a. Yes. The recording control unit 11c records the inference model usage information as metadata in the metadata recording area 16b. Note that the recording control unit 11c can also read and reproduce information recorded in the recording unit 16.

  In the present embodiment, the imaging apparatus 20 is provided with an inference engine 12 as an inference unit. The inference engine 12 includes a storage unit 12a, and one or more dictionaries (two dictionaries 12a1 and 12a2 in FIG. 1) are provided in the storage unit 12a. Each of the dictionaries 12a1 and 12a2 is configured by a network obtained by completing learning in machine learning, that is, an inference model. The dictionaries 12a1 and 12a2 can be identified by assigned dictionary IDs. For example, even when a dictionary is imported from the external device 30, only a necessary dictionary can be acquired by the dictionary ID.

  FIG. 2 is an explanatory diagram for explaining the dictionaries 12 a 1 and 12 a 2 stored in the storage unit 12 a of the inference engine 12. In FIG. 2, a large amount of data sets corresponding to the input A and the output B are given to the predetermined network N1 as learning data. As a result, the network N1 is determined so that an output B corresponding to the input A is obtained. As the network N1 employed for machine learning, various known networks may be employed. For example, R-CNN (Regions with CNN features) using CNN (Convolution Neural Network), FCN (Fully Convolutional Networks), or the like may be used. In addition to the deep learning, an inference model may be acquired using various known machine learning techniques.

  By providing a large amount of data set to the network N1, the design of the network N1 is determined so that the relationship between the input and the output similar to the relationship between the input A and the output B can be obtained with high reliability. Thus, the learned network N1 can be used as the inference model IM1.

  In the storage unit 12a of the inference engine 12, a dictionary 12a1 corresponding to the inference model IM1 is stored. The storage unit 12a corresponds to an inference model obtained by using a network in which a large amount of data sets having input / output relationships different from the relationship between the input A and the output B described above are provided as learning data. A dictionary 12a2 is also stored.

  The control unit 11 is provided with a setting control unit 11d, and the setting control unit 11d can control the inference engine 12 to perform inference using the inference engine 12. The control unit 11 may control each unit based on the inference result of the inference engine 12. For example, when the inference engine 12 detects a target object for focus control, the inference engine 12 determines whether or not the target object exists in the captured image when the captured image is given. If it exists, the position in the image is output to the control unit 11. In this case, the imaging control unit 11a performs focus control so that the focus is matched at the position of the detected object.

  In the present embodiment, the setting control unit 11d can control the display control unit 11f as a presentation unit to display the result of inference by the inference engine 12 on the display screen of the display unit 15. ing. For example, when detecting an object to be focused by inference by the inference engine 12, the display control unit 11f displays a display for recognizing the detection result, for example, a frame display surrounding the detected object. You may come to let me.

  Note that the setting control unit 11d is not limited to display, but may be able to present the result of inference by the inference engine 12 to the user by various methods. For example, the setting control unit 11d may present the inference result by voice, or may present the inference result by mechanical control of the drive unit.

Furthermore, in the present embodiment, the setting control unit 11d as a determination unit performs inference by performing determination based on a user operation on the operation unit 13 or determination based on image analysis on the captured image signal-processed by the image processing unit 11b. It is possible to determine and set whether to use inference using the engine 12 and which dictionary to use inference when using inference.
Thus, since the operation unit 13 is operated by the user reflecting a specific intention, the inference result is useful information for determining whether or not the inference result is effective for the user. In particular, in a personal use device, an operation by a user who operates the device can be considered to be based on the judgment of the user. On the other hand, even if the device is used by various people, the same effect can be expected by providing a function for individual determination such as providing fingerprint authentication on the operation unit 13 or performing voiceprint authentication using voice during use. In addition, the number of devices operated by voice is increasing, and in this case, the function of acquiring voice and determining the content of the voice is the operation unit, but at this time, voiceprint authentication can be easily used together.
In addition, the setting control unit 11d may determine which operation on the operation unit 13 does not adopt what inference result by using information registered in advance. For example, the recording unit 16 may be provided with an inference function and an operation relation database 16c. The inference function / operation relation database 16c is a database of what kind of control the inference result corresponds to and what operation unit the control is associated with. The setting control unit 11d has an inference function, By referring to the operation relation database 16c, it is possible to determine an operation that adopts or rejects the inference result (hereinafter also referred to as a related operation). For example, if the inference model is for realizing the focus function, if the operation to operate the focus ring is performed for the result display of the inference function to achieve the focus function, The control unit 11d can determine that the user has rejected the inference result.

  In addition, the setting control unit 11d is configured to give information (inference model use information) regarding this setting to the recording control unit 11c. Thereby, the recording control unit 11c stores the inference model usage information in the metadata recording area 16b of the recording unit 16 as metadata of the captured image in synchronization with the captured image obtained by the imaging unit 22. ing. In this way, evidence is recorded on the use of the inference model.

  The imaging device 20 can also acquire an inference model from the external device 30 via the communication unit 14. The external device 30 includes a learning unit 31 and an external image database (DB) 32. The learning unit 31 has a communication unit 31b, and the external image DB 32 has a communication unit 33. The communication units 31 b and 33 can communicate with each other, and the communication unit 31 b can also communicate with the communication unit 14.

  The learning unit 31 includes a population creation unit 31a, an output setting unit 31c, and an input / output modeling unit 31d, and the external image DB 32 includes an image classification function unit 34. The image classification function unit 34 classifies and records a plurality of images for each type of object included in the image. In the example of FIG. 1, the image classification function unit 34 shows an example of recording the first-type image group of the object and the second-type image group of the object, but the number of types to be classified can be set as appropriate. It is.

  The mother set creation unit 31a reads an image from the external image DB 32 and creates a mother set as a basis of learning data. The output setting unit 31c sets an output for a population image. For example, it is conceivable to use the apparatus shown in FIG. 1 for detecting an object that is a target for focusing a captured image. For example, when focusing on the human eye in the captured image, the image portion of the eye is detected by inference. In this case, the population creation unit 31a uses the image of the eye as the population, and the output setting unit 31c sets the parameters used at the time of capturing the image and the focus position together with information indicating that the image is an eye. Set.

  The input / output modeling unit 31d uses, for example, a learning model (inference model) in which the relationship between the image set created by the set creation unit 31a and the output set by the output setting unit 31c is learned by the method shown in FIG. Generate. The learning unit 31 is configured to transmit the generated inference model to the image acquisition device 10 via the communication units 31 b and 14 when requested by the control unit 11 of the image acquisition device 10. The control unit 11 can store the inference model acquired via the communication unit 14 in the storage unit 12a of the inference engine 12 as a dictionary.

  Next, the operation of the embodiment configured as described above will be described with reference to FIGS. 3 and 4 are flowcharts for explaining the operation of the first embodiment. FIG. 3 shows the operation of the image acquisition apparatus 10 and FIG. 4 shows the operation of the external device 30. 5 and 6 are explanatory diagrams for explaining the operation of the first embodiment.

  3 to 6 are for explaining the operation when an industrial endoscope is constituted by the imaging apparatus 20 of FIG. For example, it is assumed that the imaging device 20 is configured by housing the imaging unit 22 of FIG. 1 in the distal end portion 23a of the insertion portion of the industrial endoscope. In addition, the imaging part 22 shall be able to image the front end side of the front-end | tip part 23a. FIG. 5 shows such a movement of the distal end portion 23a by a change in the position of the distal end portion 23a at every predetermined time interval, and an insertion portion on the proximal end side from the distal end portion 23a is not shown. In the example of FIG. 5, the distal end portion 23 a enters from the inlet side of the pipe 41 and progresses in the direction of the deep portion 43 of the lumen of the pipe 41 as indicated by an arrow. Image P1, P2,... In FIG. 5 indicates images sequentially obtained by being imaged by the imaging unit 22 in accordance with the movement of the distal end portion 23a.

  At the timing at which the images P1 to P3 are obtained, the distal end portion 23a faces the direction of the deep part 43, and the image 43a of the deep part 43 of the lumen is included in the approximate center of the images P1 to P3. A convex portion 42 is formed on the inner wall of the pipe 41, and the image 42a of the convex portion 42 that is visible in the image P4 is captured by the tip portion 23a approaching the convex portion 42. As the distal end portion 23a gets closer to the convex portion 42, the image 42a is captured in a large size as shown in the image P5. Further, when the distal end portion 23a advances to the deep portion 43 side, the image 42a is not included in the image P6.

  In the images P <b> 1 to P <b> 6, it can be seen that the image 43 a of the deep portion 43 is always located at substantially the center of the image, and the distal end portion 23 a proceeds toward the deep portion 43. It is assumed that the storage unit 12a of the inference engine 12 stores an image similar to the images P1 to P6 and an inference model obtained as a result of learning about changes thereof as a dictionary. That is, the inference engine 12 can infer changes in the captured image when the insertion unit is correctly inserted.

  FIG. 3 shows an example in which whether or not the insertion is correctly performed is determined by inference when the endoscope insertion portion is inserted. When the insertion portion is inserted, the imaging is performed by the imaging portion 22 housed in the distal end portion 23a. The control unit 11 of the imaging device 20 captures an image obtained by the imaging unit 22 in step S1 of FIG. The display control unit 11f of the control unit 11 gives the captured image to the display unit 15 for display. In addition, the control unit 11 gives the captured image to the inference engine 12 to infer whether or not the image is correctly inserted.

  That is, in step S2, the recording control unit 11c of the control unit 11 provides the captured image to the inference engine 12 while capturing the captured image to the recording unit 16 and temporarily recording the captured image. The inference engine 12 compares the successively captured images and detects whether there is a change (step S3). Further, when the inference engine 12 detects a change in two images, the process proceeds from step S3 to step S4 to infer whether the image change before and after the change is a change when it is correctly inserted. The inference result is output to the control unit 11. Note that if no image change is detected in step S3, the inference engine 12 proceeds to step S11 and determines whether or not there has been a shooting operation. The inference engine 12 infers what is a smooth change when the insertion is performed correctly as shown in FIG. In this case, the control unit 11 shifts the processing from step S5 to step S11, and determines whether or not there is a photographing operation.

  Here, the insertion of the insertion portion is assumed to be as shown in FIG. FIG. 6 shows a state of insertion when it is inferred that the insertion portion is not correctly inserted by the same description method as FIG. In the example of FIG. 6, the distal end portion 23 a proceeds toward the deep portion 43 until the time when the images P <b> 11 to P <b> 14 are captured. At the time when the image P15 is captured, the tip 23a moves in a direction toward the inner wall of the pipe 41 with the orientation deviating from the direction of the deep portion 43. As a result, the image 43a of the deep portion 43 is shifted from the center of the image, and at the time when the image P16 is captured, the image 43a of the deep portion 43 is greatly deviated from the center of the image to the extent that a collision between the distal end portion 23a and the inner wall of the pipe 41 is expected. It's off. In addition, in the image P16, a display 44 indicating the predicted position of the collision between the distal end portion 23a and the inner wall of the pipe 41 is also displayed.

  When the insertion shown in FIG. 6 is performed, the inference engine 12 outputs an inference result indicating that the captured image has changed unsuccessfully to the control unit 11 in step S4. In step S5, the control unit 11 determines that the change in the captured image is not a smooth change, and the process proceeds to step S6. In step S6, the display control unit 11f displays on the display screen of the display unit 15 a warning display indicating that the insertion has not been performed correctly. Note that the control unit 11 may generate a warning sound. Next, the control part 11 determines whether operation | movement is continued in step S7. For example, the control unit 11 can determine whether or not the movement of the distal end portion 23a is continued by image analysis on the captured image. For example, when the progress of the distal end portion 23a is stopped, the control unit 11 determines that the operation is not continued and returns the process to step S1.

  On the other hand, if it is determined that the operation is continued, the control unit 11 determines in step S8 that the warning is ignored, automatically shoots, and records evidence. That is, the captured image signal-processed by the image processing unit 11b is recorded in the image data recording area 16a of the recording unit 16 by the recording control unit 11c. In addition, the setting control unit 11d generates inference model use information indicating that the inference by the inference engine 12 is ignored and is not used, and supplies the inference model use information to the recording control unit 11c. Thereby, the recording control unit 11c records the inference model usage information as metadata of the captured image to be recorded in the image data recording area 16a. The recorded image and inference model use information indicate that the insertion of the distal end portion 23a is continued ignoring the inference result that the distal end portion 23a is not correctly inserted, and the endoscopic image at that time is clear. Become.

  In the next step S9, the control unit 11 determines whether or not the shooting and recording in step S8 has been repeated a predetermined number of times. The control part 11 transfers a process to step S11, when not having reached the predetermined number of times. Further, when the shooting and recording in step S8 has reached the predetermined number of times, the control unit 11 determines that there is a problem with the warning presentation method, and changes the warning method in the next step S10. For example, the warning presentation method is changed, such as increasing the size of the warning display, changing the color, changing the timing, generating a warning sound or changing the volume as well as the warning display.

  In step S11, the control unit 11 determines whether or not a shooting operation has been performed. The control unit 11 returns the process to step S1 when the photographing operation is not performed, and performs photographing and recording at step S12 when the photographing operation is performed.

  FIG. 4 illustrates a method for creating the inference model described above. The external image DB 32 of the external device 30 stores an image when the endoscope is inserted. In step S21, the population creation unit 31a of the learning unit 31 sets the image change upon insertion as the population. In step S22, the input / output modeling unit 31d converts the image change at the time of good insertion (OK) into teacher data, and at step S23, converts the image change at the time of poor insertion (NG) into teacher data to generate an inference model. (Step S24). If there is request data, the learning unit 31 performs inference using the request data (step S25). The input / output modeling unit 31d determines whether or not the inference reliability in step S25 is equal to or greater than a predetermined value (step S26).

  If the reliability is not equal to or higher than the predetermined value, the input / output modeling unit 31d moves the process to step S27, resets the learning matrix, etc., and then moves the process to step S24 to change the inference model. Generate. If the reliability becomes equal to or higher than the predetermined value, the input / output modeling unit 31d moves the process to step S28 and transmits the generated inference model to the image acquisition device 10 via the communication unit 31b. Thus, the inference model for determining whether or not the insertion is correctly performed is stored in the inference engine 12 of the image acquisition apparatus 10.

  The inference engine 12 can store a plurality of inference models (dictionaries), and the inference engine 12 includes an inference model for each insertion object of the insertion unit. The setting control unit 11d can change the inference model used for each insertion object.

As described above, in the present embodiment, inference using an inference model is performed, and inference model use information indicating whether or not an inference result is adopted and which inference model is adopted is recorded. It is supposed to be. The inference result by the inference model is not always effective. By recording the inference model usage information, it is easy to determine the boundary of whether the inference model is valid, and the reason for using the inference model can be promoted by clarifying the use range of the inference model. In addition, when the inference result by the inference model is ignored for a predetermined number of times, it is possible to determine that the warning method based on the inference result is inappropriate, which contributes to the improvement of the warning method and the like.
In the case of this embodiment, it is registered in the inference function / operation relation database 16c that operations such as an insertion operation and a stop operation are operations that do not adopt the inference result for an inference function such as a guide display. Has been. Such an insertion operation, a stop operation, and the like can be determined from a change in the captured screen. Furthermore, regarding this inference function, an operation in which the user presses the “failure” button may be included in the database.

(Second Embodiment)
7 and 8 are flowcharts showing an operation flow employed in the second embodiment of the present invention. The hardware configuration of this embodiment is the same as that shown in FIG. This embodiment shows an example in which a digital camera is configured by the imaging device 20 of FIG.

  FIG. 9 is an explanatory diagram showing a state in which a subject is imaged by the imaging device 20 of FIG. Each part of the imaging device 20 in FIG. 1 is housed in a housing 20a in FIG. A display screen 15a constituting the display unit 15 is disposed on the back surface of the housing 20a. Further, a lens (not shown) constituting the optical system 22b is disposed on the front surface of the housing 20a, and a shutter button 13a constituting the operation unit 13 is disposed on the upper surface of the housing 20a.

  FIG. 9 shows an example in which a snake butterfly (butterfly) 56 stopped on a plant 55 is photographed as a subject. For example, the user 51 holds the casing 20a with the right hand 52 and displays the display screen 15a of the display unit 15. While viewing the butterfly 56 within the field of view, shooting is performed by pressing the shutter button 13a with the finger 52a of the right hand. The butterfly 56 has a pattern resembling human eyes on the wing 57.

  In the present embodiment, an object used for focus control is determined using an inference model. That is, the inference engine 12 stores an inference model for detecting an object for focus control. For example, it is assumed that an inference model (hereinafter referred to as a human dictionary) for detecting human eyes is stored in the inference engine 12.

  The controller 11 determines whether or not an image acquisition mode is designated in step S31 of FIG. When the image acquisition mode is designated, the control unit 11 performs image input and display in step S32. That is, the imaging unit 22 images a subject, and the control unit 11 captures a captured image from the imaging unit 22 and gives the captured image as a through image to the display unit 15 for display.

  In the present embodiment, the setting control unit 11d causes the inference engine 12 to perform inference for detecting an object for focus control. The inference engine 12 uses the inference model (human dictionary) stored in the storage unit 12a to detect the image portion of the human eye that is the target of focus control from the captured image. The inference engine 12 outputs the inference result to the control unit 11.

  FIG. 10 is an explanatory diagram showing a captured image displayed on the display screen 15 a of the display unit 15. As described above, the user 51 tries to photograph the butterfly 56 on the plant 55. The image P21 shows a through image at a certain moment. An image 61 of butterflies 56 is displayed on the through image displayed on the display screen 15a. At this time, an inference result indicating that the reliability is low as an eye image is given to the control unit 11 from the inference engine 12. The control unit 11 determines whether or not inference has been performed in step S33, and when an inference result is obtained, in step S34, it is determined whether or not an inference result whose reliability is higher than a predetermined threshold is obtained. To do. In this case, since the reliability is low, the control unit 11 shifts the process to step S39 and determines whether or not a photographing operation has been performed. If the photographing operation is not performed, the control unit 11 returns the process to step S31.

  Next, it is assumed that the image P22 of FIG. 10 is displayed on the display screen 15a as a through image. The image P22 includes an image 62 of the butterfly 56 with the wings spread. The image 62 includes an image portion similar to the human eye, and the inference engine 12 infers the blade pattern as the human eye image portion. As a result, the inference engine 12 outputs an inference result whose reliability is higher than a predetermined threshold value to the control unit 11.

  If it does so, the control part 11 will determine with the reliability of the inference result of the inference engine 12 being high in step S34, and will transfer a process to step S35. In step S35, the setting control unit 11d controls the display control unit 11f to display the inference result. The display control unit 11f displays a frame image 64 indicating the position where the eye image portion is detected as an inference result (image P23 in FIG. 10). The frame image 64 shows a focus position (AF point) in focus control.

  When the autofocus is set, the setting control unit 11d gives the imaging control unit 11a information that sets the image portion of the eye detected by inference as the focus position. The imaging control unit 11a drives and controls the optical system 22b so that the in-focus state is obtained at the designated focus position. Thereby, a captured image in which the image portion of the eye detected by inference is in focus is obtained.

  An image P23 in FIG. 10 shows a through image displayed on the display screen 15a in this case, and has an image portion 63a that is in focus and an image portion 63b that is not in focus (broken line). By using a human (person determination or face detection) dictionary for detecting human eyes, the image P23 is an image in which the pattern portion of the wings of the snake-eye butterfly 56 is focused, and the user desires It is thought that it is not in the focused state.

Therefore, it is assumed that the user 51 performs an operation of changing the focus by the operation unit 13 such as a dial operation. In step S36, the control unit 11 determines whether or not the user has performed a focus change operation. When the control unit 11 detects the focus change operation, the process proceeds to step S39. The control unit 11 determines whether or not a shooting operation has been performed in step S39. If the shooting operation has not been performed, the process returns to step S31. If the shooting operation has been performed, the process proceeds to step S40. Take a picture. Step S40 does not adopt the focus control based on the inference result by the inference model (human (person determination or face detection) dictionary), and the focus control by the focus change operation performed by the user is performed and the photographing is performed. To be executed. In step S <b> 40, the control unit 11 performs shooting and records evidence regarding the use of inference. That is, the captured image signal-processed by the image processing unit 11b is recorded in the image data recording area 16a of the recording unit 16 by the recording control unit 11c. Further, the setting control unit 11d generates inference model use information indicating that the inference by the inference engine 12 has not been used, and provides the inference model use information to the recording control unit 11c. Thereby, the recording control unit 11c records the inference model usage information as metadata of the captured image to be recorded in the image data recording area 16a. Based on the recorded captured image and the inference model usage information, it is clear that shooting was performed in accordance with the focus change operation independently performed by the user without using the inference using the inference model stored in the inference engine 12. Become.
However, for function items that are automatically set by using an inference model, it is important to determine the operation of the operation unit that corresponds to the same or similar function item, and operations that are not related to the output of the inference model are performed. Even if it is, it is not dissatisfaction with the inference result, and therefore does not reflect the history of not using the inference model usage information. That is, in order to correctly determine the determination unit that determines whether or not the inference result is adopted according to the operation result of the operation unit and the content of the inference result, the determination unit is It is important to make a decision according to whether the operation of the operation unit affects the items related to the content of the inference result, considering the range of involvement of the inference unit It is determined whether or not the inference result is adopted. For example, if the focus operation is being performed on the focus result (whether the focus is correct, the display of the focused position is as expected by the user, the focus position, the display result, etc.) This can be considered as a decision to reject the inference result. In order to make such a determination, an inference function and an operation relation database that can understand what kind of control the inference is related to and what kind of operation unit it is associated with may be provided in the recording unit. For example, since face detection is used for focusing, the function controlled thereby is focusing, and the focus ring or the like is an operation unit corresponding to the function. For the display function of the part to be focused which will be described later, for example, focus switching using a touch panel may be used as a corresponding operation.

  Further, it is assumed that the user is not satisfied with the focus control using the human dictionary and desires the focus control using another dictionary (inference model). In this case, the user operates the operation unit 13 to end the acquisition mode and display a menu related to the dictionary. When this operation is performed, the control unit 11 controls the display control unit 11f to display a dictionary menu on the display screen 15a of the display unit 15.

  11A and 11B are explanatory diagrams showing the dictionary menu screen 70. FIG. On the dictionary menu screen 70, an inference model (dictionary) registered in the inference engine 12 is displayed. In the example of FIG. 11A, it is indicated that only a human dictionary is registered by an icon 71 described as “human dictionary”. The user 51 operates the add button 72 in order to add a dictionary suitable for shooting the butterfly 56. For example, when a touch panel (not shown) is provided on the display screen 15a, the user 51 performs a touch operation on the add button 72.

  The control unit 11 determines whether or not a dictionary transfer is requested in step S41. When the add button 72 is operated, the control unit 11 determines that the user desires to transfer a new dictionary. Then, the process proceeds to step S42. In step S42, the display control unit 11f displays a target setting screen and a re-learning object setting screen to allow the user to specify the target and the re-learning target. The control unit 11 makes a learning request or a re-learning request for the target object or the re-learning object specified by the user to the external device 30.

  FIG. 8 shows an inference model creation process in the external device 30. In FIG. 8, the same steps as those in FIG. In step S51, the learning unit 31 of the external device 30 determines whether a learning request or a re-learning request has been received from the control unit 11 of the image acquisition device 10 via the communication units 14 and 31b. When receiving the learning request or the re-learning request, the learning unit 31 sets an object included in the request content in step S52. For example, it is assumed that there is a learning request for the butterfly dictionary based on the operation of the user 51 from the control unit 11. In this case, in step S53, the population creation unit 31a converts the butterfly image, which is the object image, and the focus position into teacher data. In step S54, the population creating unit 31a converts an image that is not the object image into teacher data separately from the object image.

  The input / output modeling unit 31d generates an inference model by learning using the teacher data generated in steps S53 and S54 (step S24). The learning unit 31 performs inference using the request data in step S25, and determines whether or not the reliability of the inference is a predetermined value or more (step S26).

  If the reliability is not equal to or greater than the predetermined value, the input / output modeling unit 31d moves the process from step S26 to step S55, resets the teacher data, and then resets the predetermined number of times in step S56. It is determined whether or not. If the resetting has not been performed a predetermined number of times or more, the input / output modeling unit 31d returns the process to step S24. When the resetting is performed a predetermined number of times or more, the input / output modeling unit 31d moves the process from step S56 to step S57, and determines that the image of the target object is not suitable for inference. Then, after transmitting poor image information to the image acquisition apparatus 10, the process proceeds to step S28. If the input / output modeling unit 31d determines that the reliability has become equal to or higher than the predetermined value in step S26, the process proceeds to step S28.

  In this way, the learning unit 31 transmits an inference model whose reliability is equal to or higher than a predetermined value or an inference model corresponding to weak image information to the image acquisition device 10 via the communication unit 31b. In step S44, the control unit 11 of the image acquisition apparatus 10 stores the received inference model in the inference engine 12 and records weak image information in the recording unit 16.

  FIG. 11B shows the dictionary menu screen 70 displayed on the display screen 15a in this case. In the example of FIG. 11B, on the dictionary menu screen 70, the human dictionary is registered by the icon 71a described as “human dictionary”, and the butterfly dictionary is registered by the icon 73 expressed as “butterfly dictionary”. It is shown. The icon 71a and the icon 73 in FIG. 11B indicate that different display is performed by a broken line and a solid line, the broken line indicates that no dictionary is selected, and the solid line indicates that the dictionary is selected. Show. Note that a dictionary corresponding to the icon is selected by the user touching the icon.

  An image P24 in FIG. 10 shows a through image displayed on the display screen 15a when focus control is performed using the butterfly dictionary in the acquisition mode. The inference engine 12 uses the butterfly dictionary to infer and detect the image portion of the butterfly 56 from the image P24, and sets the focus position information to be set for butterfly imaging and the butterfly imaging as the inference result. Information on various shooting parameters to be supplied is given to the control unit 11. The imaging control unit 11a controls the imaging unit 22 according to the inference result. As a result, an image 65 in which the entire butterfly 56 is in focus is displayed in the image P24 on the display screen 15a, and the focus position set for the detected butterfly as an inference result using the butterfly dictionary. A frame image 66 is displayed.

  Here, it is assumed that the user 51 depresses the shutter button 13a without performing the focus change operation by himself. That is, in this case, the control unit 11 detects a photographing operation in step S37 subsequent to step S36. In the next step S <b> 38, the control unit 11 takes a picture and records the evidence regarding the use of inference. That is, the captured image signal-processed by the image processing unit 11b is recorded in the image data recording area 16a of the recording unit 16 by the recording control unit 11c. Further, the setting control unit 11d generates inference model use information including information such as a dictionary ID indicating that the inference by the inference engine 12 is used and that the used dictionary is a butterfly dictionary, and a recording control unit 11c. Thereby, the recording control unit 11c records the inference model usage information as metadata of the captured image to be recorded in the image data recording area 16a. From the recorded captured image and the inference model usage information, it becomes clear that the image was taken using the inference using the butterfly dictionary stored in the inference engine 12.

  In the above description, an example in which the user explicitly adds a dictionary has been described, but a dictionary can be automatically added without any operation by the user. If the transfer of the dictionary is not requested in step S41, the control unit 11 determines in step S45 whether the result of inference unnecessary is a majority based on the recorded inference model usage information. Determine. If the inference result is not adopted in the set dictionary when the inference result is adopted more than the case where the inference result is adopted, the control unit 11 does not need to infer all the dictionaries held by the inference engine 12 in step S46. It is determined whether the result is a majority.

  If the controller 11 does not make a determination that the result of inference is not a majority for all of the retained dictionaries, the process proceeds to step S47, the normal dictionary is switched to another dictionary, and the process returns to step S31. . If a majority of the retained dictionaries do not require inference, the controller 11 determines that the inference engine 12 does not store a dictionary suitable for focus control of the subject that the user prefers to shoot. Then, the process proceeds to step S42, and the external device 30 is requested to create and transfer a dictionary. In this case, the control unit 11 may prompt the user to specify the type of dictionary requested in step S42, that is, the object to be detected by the dictionary. Even when there is no operation for specifying the target object, it is possible to automatically specify the target object and request creation and transfer of the dictionary. For example, the control unit 11 determines which target object is the main subject by image analysis of the captured image, and requests creation and transfer of a dictionary for detecting the target object based on the determination result. It may be.

  In addition, when the control unit 11 determines, for example, that the frequency of shooting the butterfly is high by image analysis of the captured image, the control unit 11 determines that the focus change operation is repeated a predetermined number of times in step S36, and the butterfly dictionary is updated. You may set so that priority may be given.

  Note that the appearance of the image varies significantly depending on the color of the image. Therefore, the control unit 11 may control to switch the inference model to be used according to the hue by determining the hue of the captured image by image analysis of the captured image. By accumulating inference model use information regarding the adoption and non-adoption of the inference model used in this case, it becomes easy to determine the inference model to be selected according to the hue.

Thus, in the present embodiment, the same effect as that of the first embodiment can be obtained. Furthermore, in this embodiment, not only the inference model to be used can be switched based on the user operation, but also the inference model to be used is automatically switched or a new inference model is determined based on the determination based on the inference model usage information. It is also possible to request an external device and automatically incorporate and use an inference model. By recording the inference model usage information in this way, it is possible to determine the validity and use range of the inference model and promote effective use of the inference model.
In the future, guidance, automatic control, and semi-automatic control using AI will be used for various devices, but these “inference functions and operation-related databases” are important in any field of devices and equipment. Consider whether the inferred result affects what control of which function and how it affects the user, and input whether the user is satisfied with the guide, automatic control, or semi-automatic control. By doing so, it can be determined whether or not the function is valid. In other words, it is not possible to know whether the inference model is suitable for the user unless the operation that cancels the function performed or being performed by the inference is determined. In the case of an imaging unit for an autonomous driving vehicle, it is suitable for verifying the validity of the inference model as in the present application even when the user is stepping on the brake when the car is moving according to the inference result. is there. If these situations occur frequently, the inference model should be customized for the user. Without such a contrivance as in the present application, it is impossible to correctly determine the necessity of such customization. In the case of a camera, if an image captured using an inference model has been deleted by a user with a high probability, it can be determined that the inference model is inappropriate. For this purpose, a mechanism is required to determine whether or not the image is taken with an inference model. In this case, the inference function is an automatic inference function in the “inference function, operation relation database”. Therefore, it is only necessary that a record of an image erasing operation is registered as the related operation. In this example, the inference model usage information needs to be recorded independently of the target image. Alternatively, if the inference model usage information is recorded as metadata of the image, before deleting the image, the metadata that the image was taken with this inference model, along with the information that it was deleted, along with external information You may make it output to the server for analysis. The server can analyze the captured inference model usage information and determine whether a new inference model is required. Further, in the case of a camera that uses a reasoning model to present a guide for instructing a shooting opportunity, an operation such as a release switch is a related operation if it is determined that shooting is not performed in response to the instruction. Further, when the inference model is suitable for the user, it can be determined by the same device. When an image photographed by the inference model is always reproduced many times, it can be determined that the user likes the inference model. The same thing can be said when a backup is recorded somewhere. Thus, the related operations do not have to be one. Nor is an inference related to a particular single function. When a face is detected, not only focus but also exposure may be adjusted. In this case, the “inference function and operation relation database” may have two functions and a plurality of related operations. Of course, the relationship between the use of the inference engine and the user operation may be determined by a specific program instead of the database, and the case where the determination is weighted with a specific calculation formula is also included in the technical scope of the present invention.

  In the above-described embodiment, the image acquisition device requests the external device to create and transfer an inference model. However, the inference model may be created in any device, for example, a computer on the cloud. May be used.

  In the above embodiment, the digital camera is used as the imaging device. However, the camera may be a digital single lens reflex camera or a compact digital camera, and a video camera such as a video camera or movie camera. Of course, a camera built in a portable information terminal (PDA: Personal Digital Assist) such as a mobile phone or a smartphone may be used. The imaging unit may be separated from the image acquisition device. For example, a machine such as a scanner in which only the imaging unit moves although the machine itself does not move is also an assumed range. In applications such as observing microorganisms, the microscope or stage may move. Further, although an example of an endoscope has been described, the present invention can also be applied to an observation apparatus such as a capsule endoscope or a CT scanner.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  It should be noted that even if the operation flow in the claims, the description, and the drawings is described using “first,” “next,” etc. for convenience, it is essential to carry out in this order. It doesn't mean. In addition, it goes without saying that the steps constituting these operation flows can be omitted as appropriate for portions that do not affect the essence of the invention.

  Of the techniques described here, the control mainly described in the flowchart is often settable by a program and may be stored in a recording medium or a recording unit. The recording method for the recording medium and the recording unit may be recorded at the time of product shipment, may be a distributed recording medium, or may be downloaded via the Internet.

  In the embodiment, the portion described as “part” (section or unit) may be configured by combining a dedicated circuit or a plurality of general-purpose circuits, and pre-programmed software as necessary. May be configured by combining a microcomputer that operates according to the above, a processor such as a CPU, or a sequencer such as an FPGA. In addition, a design in which an external device takes over part or all of the control is possible. In this case, a wired or wireless communication circuit is interposed. Communication may be performed by Bluetooth, WiFi, a telephone line, etc., and may be performed by USB. A dedicated circuit, a general-purpose circuit, or a control unit may be integrated into an ASIC. The moving unit or the like is configured by various actuators and, if necessary, a moving connection mechanism, and the actuator is operated by a driver circuit. This drive circuit is also controlled by a microcomputer or ASIC according to a specific program. For such control, detailed correction and adjustment may be performed according to information output from various sensors and their peripheral circuits. In addition, although an example in which the judgment is made based on the learning result determined by the artificial intelligence in terms of an inference model or a learned model has been described, this can also be replaced by a simple flowchart, conditional branching, or numerical judgment involving computation. There is a case. In addition, the learning of machine learning may be performed in the imaging apparatus by improving the computing capability of the control circuit of the camera or by narrowing down to a specific situation or object.

  DESCRIPTION OF SYMBOLS 10 ... Image acquisition apparatus, 11 ... Control part, 11a ... Imaging control part, 11b ... Image processing part, 11c ... Recording control part, 11d ... Setting control part, 11e ... Communication control part, 11f ... Display control part, 12 ... Inference Engine, 12a ... Storage unit, 12a1, 12a2 ... Dictionary, 13 ... Operating unit, 14 ... Communication unit, 15 ... Display unit, 16 ... Recording unit, 16a ... Image data recording area, 16b ... Metadata recording area, 20 ... Imaging Apparatus, 22 ... Imaging unit, 22a ... Imaging element, 22b ... Optical system, 30 ... External device, 31 ... Learning unit, 31a ... Population creation unit, 31b ... Communication unit, 31c ... Output setting unit, 31d ... Input / output model Conversion unit, 32 ... external image DB, 34 ... image classification function unit.

Claims (12)

An image acquisition unit for acquiring images;
Using a predetermined inference model, an inference unit that performs inference using the image acquired by the image acquisition unit as an input,
A presentation unit for presenting the inference result of the inference unit;
A determination unit for determining whether or not the inference result is adopted;
Based on the determination result of the determination unit, create usage information regarding the use of the inference model, and record the generated usage information as metadata of the image acquired by the image acquisition unit;
An image acquisition apparatus comprising: The image acquisition device has an operation unit for a user to operate,
The image acquisition apparatus according to claim 1, wherein the determination unit determines whether the inference result is adopted according to the operation result of the operation unit and the content of the inference result. The determination unit determines whether or not the inference result is adopted according to whether or not the operation of the operation unit has an influence on an item related to the presentation content of the inference result. The image acquisition device according to claim 2. The inference section has a plurality of inference models,
The image acquisition apparatus according to claim 1, wherein the control unit records the metadata including the information indicating which inference model is used in the usage information as the metadata. When the determination unit obtains a determination result indicating that the inference result is not adopted by the determination unit, the control unit automatically records the image acquired by the image acquisition unit together with the metadata. The image acquisition apparatus according to claim 1, wherein: The image acquisition device according to claim 1, wherein the presentation unit issues a warning when the determination unit obtains a determination result indicating that the inference result is not adopted. The presenting unit performs warning by changing a warning method when a determination result indicating that the inference result is not adopted by the determination unit is obtained a predetermined number of times or more. 6. The image acquisition device according to 6. The image acquisition apparatus according to claim 1, wherein the presentation unit includes a display unit that displays the image, and displays a display indicating the inference result on the display unit. The control unit requests an external device to create and transfer a new inference model used by the inference unit when the determination unit obtains a determination result indicating that the inference result is not adopted. The image acquisition apparatus according to claim 1. The control unit performs control to switch the inference model used by the inference unit when the determination result indicating that the inference result is not adopted by the determination unit is obtained a number of times equal to or greater than a predetermined threshold. If the determination result indicating that the inference result is not adopted by the determination unit is obtained a number of times greater than or equal to a predetermined threshold, the creation and transfer of a new inference model used by the inference unit is externally performed. The image acquisition apparatus according to claim 1, wherein the image acquisition apparatus requests an apparatus. An image acquisition step of acquiring an image;
An inference step for performing an inference using the image acquired in the image acquisition step as an input, using a predetermined inference model;
A presenting step for presenting the inference result in the inference step;
A determination step of determining whether or not the inference result is adopted;
A control step of creating usage information regarding the use of the inference model based on the determination result in the determination step, and recording the generated usage information as metadata of the image acquired in the image acquisition step;
An image acquisition method comprising: On the computer,
An image acquisition step of acquiring an image;
An inference step for performing an inference using the image acquired in the image acquisition step as an input, using a predetermined inference model;
A presenting step for presenting the inference result in the inference step;
A determination step of determining whether or not the inference result is adopted;
A control step of creating usage information regarding the use of the inference model based on the determination result in the determination step, and recording the generated usage information as metadata of the image acquired in the image acquisition step;
An image acquisition program characterized in that

Images