JP2021052319A - Imaging device and machine learning processing method - Google Patents

Abstract

To provide technology for efficiently collecting video data.SOLUTION: An imaging device of the present invention comprises multiple types of optical filters, an optical filter switching mechanism, an imaging unit and an imaging drive unit. The multiple types of optical filters have mutually different optical characteristics. The optical filter switching mechanism selectively switches the multiple types of optical filters and inserts one in the imaging optical path of a subject. The imaging unit images the subject via the optical filter inserted in the imaging optical path and outputs filter video data. The imaging drive unit drives the imaging unit in synchronism with optical filter insertion timing while switching the optical filters to be inserted in the imaging optical path by the optical filter switching mechanism, and thereby generates multiple types of filter video data that corresponds to the subject.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging device and a machine learning processing method.

In recent years, image inspection technology using machine learning processing has attracted attention and has been introduced into various fields such as medical care, agriculture, food and precision equipment factories. In this kind of technology, by using a machine learning device that has been machine-learned in advance, it is possible to obtain an inspection result that captures an image feature of an object. In order to improve the accuracy of such image inspection, it is necessary to carefully learn the machine learning device, and it is necessary to prepare a large amount of various video data related to the image inspection.

Data augmentation (data augmentation method) is known as a method of preparing a large amount of video data for learning. In this data augmentation, the number of video data is increased by performing image processing such as rotation and translation on one video data.

For example, Patent Document 1 discloses a technique of "generating a large number of images by interpolating a small amount of video data with an image interpolation device or transforming data with an image transforming device".

Japanese Unexamined Patent Publication No. 9-237340

In machine learning processing, a technique for efficiently collecting video data for learning is required.
Therefore, an object of the present invention is to provide a technique for efficiently collecting video data.

In order to solve the above problems, a typical image pickup apparatus of the present invention includes a plurality of types of optical filters, an optical filter switching mechanism, an image pickup unit, and an image pickup drive unit.
The plurality of types of optical filters have different optical characteristics from each other.
The optical filter switching mechanism selectively switches a plurality of types of optical filters and inserts them into the shooting optical path of the subject.
The imaging unit captures the subject through an optical filter inserted in the photographing optical path, and outputs the filter image as data.
The image pickup drive unit drives the image pickup unit in synchronization with the insertion timing of the optical filter while switching the optical filter to be inserted in the shooting optical path by the optical filter switching mechanism, thereby displaying a plurality of types of filter images corresponding to the subject. Generate data.

According to the present invention, video data can be efficiently collected.

Issues, configurations and effects other than those described above will be clarified by the description of the following embodiments.

FIG. 1 is a block diagram showing the configuration of the image pickup apparatus of the first embodiment. FIG. 2 is a diagram showing an example of the basic configuration of the machine learning device 20. FIG. 3 is a diagram illustrating the outer shape of the image pickup apparatus and the internal arrangement of the optical filter switching mechanism 11. FIG. 4 is a diagram illustrating the positional relationship between the opening holes 110A and 110B and the image sensor 103. FIG. 5 is a diagram illustrating the positional relationship between the opening holes 110A and 110B and the image sensor 103. FIG. 6 is a diagram illustrating the relationship between the rotation angle of the rotary actuator 105 and the electronic shutter control (charge accumulation control, etc.) of the image sensor 103. FIG. 7 is a diagram illustrating an example of a filter image compositing process. FIG. 8 is a diagram illustrating an example of image processing of a filtered image. FIG. 9 is a diagram illustrating an example of combined processing in which video processing and composition processing are combined. FIG. 10 is a flow chart (1/2) for explaining the imaging mode of the imaging apparatus. FIG. 11 is a flow chart (2/2) for explaining the imaging mode of the imaging apparatus.

Hereinafter, examples of the present invention will be described with reference to the drawings.

<Configuration of imaging device>
First, the configuration of the image pickup apparatus of the first embodiment will be described.
FIG. 1 is a block diagram showing the configuration of the image pickup apparatus of the first embodiment.
In the figure, the image pickup apparatus includes a group of a plurality of types of optical filters 10, an optical filter switching mechanism 11, an image pickup unit 12, an image pickup drive unit 13, an image processing unit 14, an image output unit 17, and a machine learning device 20.

The plurality of types of optical filters 10 are a group of optical elements having different optical characteristics. Examples of such an optical filter 10 include a visible light filter that selectively transmits visible light, an infrared filter that selectively transmits near-infrared light, and the like.

The optical filter switching mechanism 11 selectively switches a plurality of types of optical filters 10 and inserts them in the photographing optical path between the subject and the imaging unit 12.

The imaging unit 12 images the subject through the optical filter 10 inserted in the photographing optical path, and outputs the filter image as data.

The image pickup drive unit 13 drives the image pickup unit 12 in synchronization with the insertion timing of the optical filter 10 while switching the optical filter 10 to be inserted in the photographing optical path by the optical filter switching mechanism, thereby for the same subject. Generate data for multiple types of filtered video.

The image processing unit 14 selectively performs a plurality of types of image processing 15 and a plurality of types of compositing processing 16 on data of a plurality of types of filtered images, whereby the number of types of filtered images for the same subject is increased. Further increase.
In this way, the image pickup apparatus can collectively generate a plurality of types of filter images for the same subject.

The video output unit 17 outputs a plurality of types of filtered video to an external device via the video processing unit 14. For example, the video output unit 17 outputs a plurality of types of filtered video at a predetermined frame rate according to an interface such as USB, GiGE, CameraLink, CoaXPress, SD-SDI, HD-SDI, or NTSC.

A plurality of types of filtered images that have passed through the image processing unit 14 are input to the machine learning device 20 as learning materials or estimation materials. When a plurality of types of filter images are input to the machine learning device 20 as learning materials, incidental information such as a teacher value regarding the subject and the type of the filter image is also input to the machine learning device 20.

<About machine learning processing method>
Subsequently, the machine learning processing method in the first embodiment will be described.

FIG. 2 is a diagram showing an example of the basic configuration of the machine learning device 20.
In the figure, the machine learning device 20 includes a preprocessing unit 31, an input layer 32, a convolutional layer / activation function 33, a pooling layer 34, a neural network 35, an output layer 36, and a learning algorithm unit 37.

The pre-processing unit 31 performs pre-processing on the input filter image according to the inspection content of the image inspection. For example, the preprocessing unit 31 can set an analysis area (crop, resize, bird's-eye view conversion, viewpoint conversion, distortion correction, padding before convolution, etc.), difference processing (background subtraction, contour extraction, etc.), and vector analysis (motion vector). Detection, motion tracking) and signal processing (feature extraction, shape extraction, contour enhancement, gradation correction, dynamic range scaling, level normalization, noise suppression, moire removal, white balance adjustment, color processing, color reduction processing, monochrome And other pretreatments.

The filter image preprocessed by the preprocessing unit 31 is input to the input layer 32.

The filter image of the input layer 32 is processed via the convolution layer / activation function 33 and the pooling layer 34.

The convolutional layer / activation function 33 includes a neuron sequence that activates according to local features of the image space. The pooling layer 34 has a function of appropriately aggregating the neuron values activated by the neuron sequence. With such a function, the convolutional layer / activation function 33 and the pooling layer 34 generate a feature map that abstracts the image features of the filtered image. The convolutional layer / activation function 33 and the pooling layer 34 are multi-layered in order to abstract the image features in multiple stages.

The neuron values of the feature map via the multi-layered convolutional layer / activation function 33 and the pooling layer 34 are serialized and then input to the neural network 35.

The neural network 35 is configured by connecting (for example, fully connecting) neuron layers at the number of stages required for image inspection. From the output layer 36 at the final end of the neural network 35, data and a data pattern indicating the estimation result (likelihood and the like) of the filter image are output.

The learning algorithm unit 37 combines a plurality of types of input filter images and a teacher value (or a teacher value related to the filter image itself) regarding the subject of the filter image to form a data set, and internally serves as learning data of the machine learning device 20. Data is stored in the memory. The learning algorithm unit 37 performs machine learning of the machine learning device 20 by using the learned data accumulated in the data. For example, the learning algorithm unit 37 has a convolution layer / activation function 33 and a neural network 35 based on an error backpropagation method or the like so that the estimation result when the filter image is given to the machine learning device 20 approaches the teacher value. Adjust the weighting coefficient and bias value of each neuron.

In the machine learning processing method described above, if the types of filter images are significantly different even for the same subject, the difference in image features may become large and the machine learning processing may not be performed collectively. In that case, it is preferable to provide a dedicated machine learning device 20 for each type of filter image to perform the machine learning processing method.

<About the configuration example of the optical filter switching mechanism 11>
Next, an example of the optical filter switching mechanism 11 will be described.

FIG. 3 is a diagram illustrating the outer shape of the image pickup apparatus and the internal arrangement of the optical filter switching mechanism 11.

In the figure, the lens mount 102 is provided in the housing 101 of the image pickup apparatus. A photographing lens for forming a subject light is attached to the lens mount 102. An image sensor 103, which is a main component of the image pickup unit 12, is arranged at the image formation position of the photographing lens.

An optical filter switching mechanism 11 including a rotary actuator 105 is arranged in the photographing optical path of the image sensor 103. The rotary actuator 105 is rotated by a motor (hereinafter referred to as "built-in motor") in the actuator. The rotary actuator 105 is divided into 6 in the circumferential direction according to the rotation angle. The six categories include a visible light filter 106 (optical filter that cuts the near-infrared region) that selectively transmits visible light and an infrared filter 107 (optical filter that cuts the near-infrared region) that selectively transmits near-infrared light. Optical filters) are arranged alternately. Six types of optical filters 10 having different optical characteristics may be attached to the six categories. Further, the angle divisions of the six divisions are related to the exposure time of the image sensor 103 as described later. Therefore, the angle division may be increased for the optical filter 10 having a long exposure time, and the angle division may be made small for the optical filter 10 having a short exposure time.

Opening holes 110A and 110B for detecting the rotation position are formed in the six divisions of the rotary actuator 105 so as to be displaced in the radial direction. Inside the image pickup apparatus 104, a light source 108 is provided at a position for illuminating the opening holes 110A and 110B. The illumination light of the light source 108 is turned on at the moment when the six divisions of the rotary actuator 105 cross over the image sensor 103, and is turned off during the other periods. The outer peripheral portion of the rotary actuator 105 is shielded from light so that stray light does not enter through a gap such as the side of the rotary actuator 105.

<Synchronous control between the optical filter switching mechanism 11 and the frame rate>
4 and 5 are views for explaining the positional relationship between the opening holes 110A and 110B and the image sensor 103.

In the center line of the image pickup surface of the image sensor 103, an area A through which the light spot of the opening hole 110A passes is set as an area on the left side, and an area B through which the light spot of the opening hole 110B passes is set as an area on the right side. Will be done.

The image pickup unit 12 sequentially reads out the line signal of the center line from the image pickup element 103. The light spot of the opening hole 110A passing through the area A and the light spot of the opening hole 110B passing through the area B are alternately detected.

FIG. 5A shows the timing at which the optical filter 10 inserted in the photographing optical path switches from the visible light filter 106 to the infrared filter 107. The imaging unit 12 detects this timing by detecting the light spot of the opening hole 110A in the area A. By this timing detection, the image pickup apparatus initializes the rotation position of the rotary actuator 105.

FIG. 5B shows the timing at which the optical filter 10 inserted in the photographing optical path switches from the infrared filter 107 to the visible light filter 106. The imaging unit 12 detects this timing by detecting the light spot of the opening hole 110B in the area B. By this timing detection, the image pickup apparatus initializes the rotation position of the rotary actuator 105.

FIG. 6 is a diagram illustrating the relationship between the rotation angle of the rotary actuator 105 and the electronic shutter control (charge accumulation control, etc.) of the image sensor 103.

As shown in the figure, when the rotary actuator 105 is step-rotated by an angle of about (φ / 2) from the initialized rotation position, the partition portion of the rotary actuator 105 is detached from the image sensor 103. Therefore, the image pickup drive unit 13 starts charge accumulation of the image pickup device 103 by electronic shutter control. When the rotary actuator 105 is step-rotated by an angle of about (2θ) from the time when the charge accumulation starts, the partition portion of the rotary actuator 105 is applied to the image sensor 103 again. Therefore, the image pickup drive unit 13 completes charge accumulation by electronic shutter control. When the rotary actuator 105 is step-rotated by about (φ/2) angles from the completion of the charge accumulation, the rotation position of the rotary actuator 105 is initialized. The rotational stability of the rotary actuator 105 is confirmed by periodically detecting the initialization of the rotational position.

In this way, the electronic shutter control of the image sensor 103 and the output of the filter image accompanying the electronic shutter control are repeated at the timing synchronized with the rotation speed. As a result, the rotation speed of the rotary actuator 105 and the frame rate of the image pickup apparatus are linked. For example, when the rotary actuator 105 rotates at 40 rotations / second, three frames of visible light images and three frames of near-infrared images are sequentially imaged for each rotation of the rotary actuator 105. Therefore, the frame rate of the visible light image is 120 fps, and the frame rate of the near infrared image is 120 fps. Further, if it is desired to reduce the frame rate to 60 fps, which is half of both images, the rotary actuator 105 may be rotated at half 20 rpm.

In this way, by synchronizing the rotation speed of the rotary actuator 105 with the frame rate, the optical filters 10 (visible light filter 106, infrared filter 107) are appropriately switched at the timing adapted to the variable frame rate of the image pickup apparatus. Will be done.

In the first embodiment, the rotary actuator 105 is divided into six categories. However, the present invention is not limited to this. If a high frame rate is not required, the number of divisions of the rotary actuator 105 may be reduced. Further, when it is desired to reduce the size of the image pickup apparatus, the diameter of the rotary actuator 105 may be reduced by reducing the number of divisions of the rotary actuator 105. Further, when it is desired to increase the number of types of the optical filter 10, the number of divisions of the rotary actuator 105 may be increased according to the number of types of the optical filter 10.

Further, in the first embodiment, the variable frame rate of the image pickup apparatus is synchronized with the rotation speed of the rotary actuator 105 (switching of the optical filter 10). However, the present invention is not limited to this. For example, a multiplication signal obtained by multiplying the frame rate synchronization signal (vertical synchronization signal or horizontal synchronization signal) of the image pickup apparatus by PLL may be generated, and the rotary actuator 105 may be step-rotated by the multiplication signal. In this case, the rotation speed of the rotary actuator 105 (switching of the optical filter 10) can be synchronized with the variable frame rate of the image pickup apparatus.

<About the composition process of filtered video>
Next, the compositing process for a plurality of types of filtered images will be described.

FIG. 7 is a diagram illustrating an example of a filter image compositing process.

The image processing unit 14 includes a visible light image (low-pass visible light image shown in FIG. 7) taken through the visible light filter 106 and a near-infrared image (high-pass near-infrared image shown in FIG. 7) taken through the infrared filter 107. Monochrome image) and is acquired as a filter image. A new filter image (composite image) is generated by superimposing and synthesizing the visible light image and the near infrared image under predetermined composition conditions.

For example, near-infrared images show impurities such as dust and scratches that do not absorb (reflect) near-infrared light. On the other hand, the image in the visible range is not captured. On the other hand, in the visible light image, impurities such as dust and scratches are shown together with the image in the visible range. Therefore, a new filter image (composite image) from which dust and scratches have been removed can be obtained by performing a composition process for removing dust and scratches from the visible light image using a near-infrared image as a removal mask. Further, by performing a compositing process of multiplying (logically producting) the near-infrared image and the visible light image, a new filtered image (composited image) emphasizing dust and scratches can be obtained.

By superimposing and synthesizing the visible light image and the near-infrared image with predetermined settings in this way, the visible light image that can be visually confirmed by humans and the near-infrared image that cannot be normally visually confirmed by humans are clearly separated. In addition, by controlling each exposure time and gain before compositing, it is possible to obtain a filtered image (composite image) in which a part that is dark and invisible to the human eye is brightly emphasized.

In such a composition process, a limited number of filter images can be obtained by increasing the number of combinations of filter images to be combined, setting additional types of composition conditions, and repeating the composition process in multiple stages. It will be possible to further increase the number of types.

<About image processing of filtered images>
Next, image processing for the filtered image will be described.

FIG. 8 is a diagram illustrating an example of image processing of a filtered image.
In the figure, the image processing unit 14 performs the following image processing on each of the filtered images.
(1) Color tone correction / conversion, (2) Contour correction, (3) Binarization, (4) Contour marking, (5) Gamma correction, (7) Other video processing

In such image processing, it is possible to further increase the number of types of filtered images, which is limited, by further setting the types of image processing or by repeating image processing in multiple stages.

<Composite processing that combines video processing and compositing processing>
Next, a composite process that combines video processing and composition processing will be described.

FIG. 9 is a diagram illustrating an example of combined processing in which video processing and composition processing are combined.

The image processing unit 14 cuts out an image level portion of a predetermined threshold or higher with respect to the near infrared image (high-pass near-infrared monochrome image shown in FIG. 9) to generate an under-level cut image.

Next, the image processing unit 14 synthesizes the under-level cut image with respect to the visible light image (low-pass visible light image shown in FIG. 9) at a plurality of types of blend ratios (80:20, 50:50 ...). To do.

In such combined processing of video processing and compositing processing, it is possible to further increase the number of types of a limited number of filter images by further setting a combination of video processing and compositing processing.

<About the imaging mode of the imaging device>
Subsequently, the imaging mode of the first embodiment will be described.
10 and 11 are flow charts illustrating an imaging mode of the imaging apparatus.
Hereinafter, the steps will be described in the order of the step numbers shown in the figure.

Step S01: The image pickup drive unit 13 turns on the position confirmation light source 108 and initializes the rotation position of the rotary actuator 105.

Step S02: The image pickup drive unit 13 determines whether the mode is single or mode switching based on the setting of the image pickup apparatus. In the case of the mode alone, the image pickup driving unit 13 shifts the operation to step S03. In the case of mode switching, the image pickup drive unit 13 shifts the operation to step S14.

Step S03: The image pickup driving unit 13 determines whether it is a visible mode or a near infrared mode based on the setting of the image pickup apparatus. In the visible mode, the image pickup driving unit 13 shifts to the operation in step S04. In the near-infrared mode, the image pickup driving unit 13 shifts to the operation in step S05.

Step S04: The imaging drive unit 13 inserts the visible light filter 106 into the photographing optical path by rotating the rotary actuator 105 from the initialized position by a predetermined step angle. After this operation, the image pickup driving unit 13 shifts to the operation in step S06.

Step S05: The imaging drive unit 13 inserts the infrared filter 107 into the photographing optical path by rotating the rotary actuator 105 from the initialized position by a predetermined step angle.

Step S06: The image pickup drive unit 13 turns off the position confirmation light source 108.

Step S07: The image pickup driving unit 13 sets an appropriate exposure time as an image pickup condition according to the inserted optical filter 10. Further, the focus may be adjusted according to the difference in the imaging distance of the inserted optical filter 10.

Step S08: The image pickup drive unit 13 drives the image pickup unit 12 (image sensor 103) according to the set imaging conditions, and generates a filter image according to the inserted optical filter 10.

Step S09: The image processing unit 14 determines whether or not to perform image processing based on the setting of the image pickup apparatus. Here, when performing video processing, the video processing unit 14 shifts the operation to step S10. When no video processing is performed, the video processing unit 14 shifts the operation to step S11.

Step S10: The image processing unit 14 performs a plurality of types of image processing on the filtered image to increase the number of types of the filtered image.

Step S11: The video processing unit 14 or the video output unit 17 has a synchronization signal input from the outside of the image pickup apparatus (hereinafter referred to as “external synchronization”) or a synchronization signal generated inside the imaging apparatus (hereinafter referred to as “internal synchronization”). ), And perform timing processing to rearrange multiple types of filter images.

Step S12: The video processing unit 14 or the video output unit 17 encodes the filtered video and generates a video signal for output.

Step S13: The video processing unit 14 outputs a video signal to the machine learning device 20. In the case of still image shooting, the image pickup driving unit 13 temporarily completes the image pickup operation here. In the case of moving image shooting, the imaging drive unit 13 returns to step S01 to continue moving image shooting.

Step S14: The image pickup drive unit 13 starts the rotation of the rotary actuator 105 in accordance with the external synchronization or the internal synchronization.

Step S15: The image pickup drive unit 13 waits until the rotation of the rotary actuator 105 stabilizes. For example, it waits until the time interval of the initialization timing of the rotation position of the rotary actuator 105 becomes constant.

Step S16: The image pickup driving unit 13 determines which of the visible light filter 106 and the infrared filter 107 is the optical filter 10 that next passes through the photographing optical path based on the detection results of the opening holes 110A and 110B. In the case of the visible light filter 106, the image pickup driving unit 13 shifts to the operation in step S17. In the case of the infrared filter 107, the image pickup driving unit 13 shifts to the operation in step S20.

Step S17: The image pickup driving unit 13 sets an appropriate exposure time as an image pickup condition according to the visible light filter 106.

Step S18: The image pickup drive unit 13 temporarily turns off the light source 108, drives the image pickup unit 12 (image sensor 103), and drives a visible light filter image (for example, a color image) according to the inserted visible light filter 106. ) Is generated. When the imaging is completed, the imaging drive unit 13 turns on the light source 108 again.

Step S19: The image processing unit 14 performs a plurality of types of image processing on the filtered image of visible light to generate a plurality of types of filtered images. After this operation, the video processing unit 14 shifts the operation to step S23.

Step S20: The image pickup driving unit 13 sets an appropriate exposure time as an image pickup condition according to the infrared filter 107. Further, the focus may be adjusted according to the imaging distance of the near infrared light.

Step S21: The image pickup drive unit 13 temporarily turns off the light source 108, drives the image pickup unit 12 (image sensor 103), and filters an image of near-infrared light (for example, according to the inserted infrared filter 107). Monochrome video) is generated as data. When the imaging is completed, the imaging drive unit 13 turns on the light source 108 again.

Step S22: The image processing unit 14 performs a plurality of types of image processing on the near-infrared light filter image to generate a plurality of types of filter images.

Step S23: The video processing unit 14 temporarily records an increased number of types of filtered video in the internal memory.

Step S24: The image pickup driving unit 13 determines whether or not all the switching of the optical filters 10 is completed. When all the switching of the optical filters 10 is completed, the image pickup driving unit 13 shifts to the operation in step S25. If the switching of the optical filters 10 is not completed, the operation is returned to step S16 in order to perform imaging on the remaining optical filters 10.

Step S25: The image processing unit 14 creates a plurality of combinations of the filter images on the internal memory, and performs a plurality of types of synthesis processing on the plurality of combinations. This compositing process synergistically increases the number of types of filtered images. It should be noted that a plurality of types of image processing may be added to the filtered image after the composition processing. Further, the image processing and the composition processing may be repeated.

Step S26: The image processing unit 14 sorts the filtered images with an increased number of types in the order of output based on the settings of the imaging device. After this operation, the video processing unit 14 shifts the operation to step S11.
By the series of operations described above, the operation according to the image pickup mode of the image pickup apparatus is performed.

<Effects of Example 1>
(1) In the first embodiment, a plurality of types of optical filters 10 having different optical characteristics are selectively switched, and the imaging unit is driven in synchronization with the insertion timing of the optical filters 10 to obtain a plurality of the same subject. Generate data of various types of filter images. Therefore, it becomes possible to efficiently collect video data for learning required in machine learning processing.

(2) Further, in the first embodiment, the number of types of filter images acquired for the same subject by performing image processing or compositing processing on a plurality of types of filter images captured by switching the optical filter 10. Can be further increased. Therefore, it becomes possible to more efficiently collect the video data for learning required in the machine learning process.

(3) In particular, in the first embodiment, the number of types of filter images can be increased stepwise as follows, for example.
First stage: The number of types of the filter image is set to N (N> 1) by switching the optical filter 10 of the “N” type.
Second stage: The number of types of the filter image is set to "N × M" by performing image processing according to M (M> 1) on the filter image of the "N" type.
Third stage: When L-like composition processing is performed for each combination of "N x M" types of filter images, it is possible to increase the number of types of filter images to "(total number of combinations) x L". Become.
As described above, in the first embodiment, since the plurality of types of optical filters 10, the plurality of types of image processing, and the composition processing are combined, the number of types of filter images can be synergistically increased. Therefore, it becomes possible to more efficiently collect the video data for learning required in the machine learning process.

(4) Further, in the first embodiment, the plurality of types of optical filters 10 include at least a visible light filter that selectively transmits visible light and an infrared filter that selectively transmits near-infrared light. Since the near-infrared image passed through the infrared filter is an image that cannot be seen by humans, image inspection that cannot be visually inspected becomes possible. Therefore, it becomes possible to obtain a filter image as a learning material or an estimation material necessary for performing an image inspection (machine learning process, etc.) that cannot be visually inspected.

<Supplementary matters of the embodiment>
In Example 1, a convolutional neural network (see FIG. 2) was used as the machine learning device 20. However, the present invention is not limited to this. For example, such machine learning devices include decision tree learning, correlation rule learning, neural networks, genetic programming, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, expression learning, principal component analysis, and extremes. • A machine learning device based on at least one of the learning machines and other machine learning techniques may be employed.

Further, in the first embodiment, the optical filter switching mechanism 11 using the rotary actuator 105 is disclosed. However, the present invention is not limited to this. Any mechanism may be used as long as it is a mechanism for selectively inserting a plurality of types of optical filters 10 into the photographing optical path. For example, an optical filter switching mechanism for switching the optical filter 10 to a slide type may be adopted. Further, in the optical filter switching mechanism 11, as one of the filter images, a transparent optical filter (empty frame, antireflection filter, etc.) may be switched in order to generate data of an optically unprocessed image. ..

Further, in Example 1, a visible light filter and an infrared filter are disclosed as the optical filter 10. However, the present invention is not limited to this. Any optical element that optically processes the incident light may be used. For example, among the incident light, light having a predetermined property such as a wavelength range may be selectively transmitted, reflected, absorbed, or removed, or an optical low-pass filter that locally shifts and superimposes the incident light may be used. It may be an ND filter that reduces the amount of transmitted light of the incident light, an optical system that displaces or rotates the incident light, a soft focus filter that diffuses the incident light with a predetermined diffusion distribution, or reacts to the incident light. It may be a filter that changes the transmitted wavelength or the transmittance, a polarized optical element that processes the polarization component of the incident light, a focal distance conversion optical system that changes the image magnification of the subject, or a focusing of the subject. It may be an in-focus distance conversion optical system (close-up filter, etc.) that changes the distance, an optical system that divides the incident light in multiple directions to form multiple images, or a part of the subject image (central part or peripheral part). It may be an optical element that optically processes only the upper half, etc., a foggy filter that adds halo (bleeding) to the incident light, or a cross filter that adds a cross-shaped light beam to the incident light.

Further, the present invention is not limited to the first embodiment, and includes various modifications. For example, Example 1 has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations. Further, it is also possible to add / delete / replace a part of the configuration of the first embodiment with another configuration.

10 ... Optical filter, 11 ... Optical filter switching mechanism, 12 ... Imaging unit, 13 ... Imaging drive unit, 14 ... Video processing unit, 17 ... Video output unit, 20 ... Machine learning device, 31 ... Preprocessing unit, 32 ... Input Layer, 33 ... Folding layer / activation function, 34 ... Pooling layer, 35 ... Neural network, 36 ... Output layer, 37 ... Learning algorithm unit, 101 ... Housing, 102 ... Lens mount, 103 ... Imaging element, 105 ... Rotary Actuator, 106 ... Visible light filter, 107 ... Infrared filter, 110A ... Opening hole, 110B ... Opening hole

Claims (5)

Multiple types of optical filters with different optical characteristics and
An optical filter switching mechanism that selectively switches between a plurality of types of the optical filters and inserts them into the shooting optical path of the subject.
An imaging unit that images a subject through the optical filter inserted in the photographing optical path and outputs filter images as data.
By switching the optical filter inserted in the photographing optical path by the optical filter switching mechanism and driving the imaging unit in synchronization with the insertion timing of the optical filter, a plurality of types of the filter images corresponding to the subject are displayed. An image pickup device equipped with an image pickup drive unit that generates data. The imaging device according to claim 1.
An imaging device including an image processing unit that further increases the number of types of the filtered images corresponding to the subject by performing image processing or compositing processing on the plurality of types of the filtered images. The imaging device according to any one of claims 1 and 2.
An image pickup apparatus comprising at least a visible light filter that selectively transmits visible light and an infrared filter that selectively transmits near-infrared light, as the plurality of types of the optical filters. A data enhancement step of acquiring a plurality of types of the filter images corresponding to the subject by the imaging device according to any one of claims 1 to 3.
Machine learning of the machine learning device is performed by giving learning data including a data set in which a plurality of types of the filtered images generated for the subject and a teacher value related to the subject or the filtered image are combined to the machine learning device. Machine learning steps and
A machine learning processing method including an estimation step in which an image to be estimated is input to the machine learning device and estimation is performed. The machine learning processing method according to claim 4.
The estimation step
A machine learning processing method characterized in that a plurality of types of the filter images corresponding to the estimation target are acquired by the imaging device, input to the machine learning device, and estimation is performed based on the output data of the machine learning device. ..

Images