JP6904757B2 - Camera system - Google Patents

Description

The present invention relates to a camera system including a camera body and a receiving unit.

The outline of the conventional camera system is shown in FIGS. 1 to 3.
The surveillance camera 101 performs signal processing on the image signal output from the image sensor and outputs a video signal. The video signal processing unit 16 includes a storage processing unit 37, a gain-up device 31, a gamma correction 35, a knee processing device 33, a color correction device (masking) 32, an enhancer 34, a WDR (wide dynamic range), and a fog haze correction device ( DEFOG) 39, DNR (digital noise reduction) processor 36, electronic zoom, etc. The receiving unit 102 receives compressed / uncompressed video signals such as HD-VLC (trademark) or HD-SDI (High Definition-Serial Digital Interface) from the camera, and format-converts them so that they can be connected to various monitors and recording devices. VBS (Video (video signal), Burst (color burst signal), Sync (synchronous signal)) output, DVI-D output, analog RGB (Red, Green, Blue) output, HD-SDI output can now be performed. ing.

The biggest merit of the configuration consisting of the surveillance camera 101 and the receiving unit 102 is that it can be transmitted by the coaxial cable 103 with a maximum of 300 m by outputting from the camera by the HD-VLC compression transmission method and restoring it to the HD-SDI signal by the receiving unit. be. This made it possible to change the camera from an analog camera to an HD-SDI camera using the existing coaxial cable. Further, the conventional surveillance camera 101 and the receiving unit 102 are equipped with FPGAs (Field Programmable Gate Arrays) 12 and 22 for performing video signal processing. The merit of FPGA, which is a kind of dynamically reconfigured device, is that signal processing can be easily customized, and it is possible to immediately respond to requests such as adding functions. On the other hand, the disadvantage is that it consumes more power than commercially available signal processing LSI chips (SoCs), and the power consumption increases further due to the addition of functions, and the temperature of the camera body rises accordingly.

In the past, functions were added to satisfy various functions with the camera alone, so it was unavoidable that the temperature of the main unit would rise as the power consumption of the camera increased.
Therefore, it is not necessary to add a function only to the camera by forming a set of the camera and the receiving unit, and it is possible to reduce the power consumption of the camera by adding the function to the receiving unit side (for example, Patent Document). 1 to 3).

Japanese Unexamined Patent Publication No. 2010-154462 Japanese Unexamined Patent Publication No. 2007-221684 Patent Registration No. 5874519 Japanese Unexamined Patent Publication No. 2011-135532 Japanese Unexamined Patent Publication No. 2007-036787 Japanese Unexamined Patent Publication No. 2010-136032 Japanese Unexamined Patent Publication No. 2014-036414 Japanese Unexamined Patent Publication No. 09-139939 Patent Registration No. 5535476 Patent Registration No. 2995683 Japanese Unexamined Patent Publication No. 2011-259047 Japanese Unexamined Patent Publication No. 2014-171119

“Partial & Dynamic Reconfiguration to Improve the Multifunctionality of 28nm FPGAs”, [online], 2010, Altera Corporation, [Search February 28, 2017], Internet <https://www.altera.co .jp / ja_JP / pdfs / literature / wp / wp-01137-stxv-dynamic-partial-reconfig_j.pdf ＞

However, if the camera to be combined and the receiving unit are not fixed, duplication or lack of functions may occur depending on the combination. Even if the functions are duplicated, there is no problem if one of the functions is stopped, but that means that the hardware resources are not being used effectively.
Further, the image processing required by the user is various, and even if the camera and the receiving unit are configured as a set, there is a possibility that the processing that can be executed is insufficient. When trying to support various functions according to demand, there is a problem that the types of processing to be performed on the camera side increase and many models must be prepared.

An object of the present invention is to provide a camera system capable of appropriately adding necessary functions to a camera or a receiving unit in view of the above problems.

The camera system of the present invention is a camera system that connects a camera and a receiving unit with a coaxial cable, and the surveillance camera is a first dynamic reconstruction device that processes a signal output from an image pickup element and a first operation. The first non-volatile memory that holds the first configuration data that sets the configuration of the target reconfiguration device, and the first that is set by loading and loading the first configuration data into the first dynamic reconfiguration device. It has a first CPU that controls the video signal processing performed by the dynamic reconstruction device so that identifiable information is output from the cable, and the receiving unit outputs the video signal output from the surveillance camera and receives the video signal via the cable. A second dynamic reconfiguration device that processes signals, a second non-volatile memory that holds multiple second configuration data that sets the configuration of the second dynamic reconfiguration device, and identifiable information output from the surveillance camera. It has a second CPU that selects and loads what should be loaded from a plurality of second configuration data based on the above, and the video signal of a predetermined format output from the receiving unit is the first dynamic reconstruction device and the second operation. It is characterized in that it is a signal that has undergone video signal processing shared by the target reconstruction device.

Further, the receiving unit has a memory, and the second dynamic reconstruction device uses the memory to perform video signal processing accompanied by inter-frame calculation, while the first dynamic reconstruction device performs video signal processing accompanied by inter-frame calculation. It is preferable not to perform the treatment.

Further, the first dynamic reconstruction device has a first color corrector that converts the input video signal of four or more colors into a video signal of three colors, and the receiving unit converts the video corrected by the color corrector. It has artificial intelligence that learns and recognizes a predetermined subject in the image, and the color corrector performs color correction processing using a clustering device, mapping table, transformation matrix table, subtractor, matrix multiplier, and adder, and is artificial. The intelligence is preferably composed of a multi-layer neural network, a learning processor, a learning data store, a learning history store, and a second color corrector.

According to the present invention, by combining a camera equipped with an FPGA and a receiving unit equipped with an FPGA of the same type, it is possible to switch functions while suppressing heat generation of the camera.

The configuration diagram of the conventional surveillance camera 101 and the receiving unit 102, and the ready-made receiving unit are for 1ch and 4ch. The configuration block diagram of the conventional camera system and the receiving unit do not have a video signal processing unit. In the example of the block diagram of the video signal processing unit 163 of the conventional camera, the power consumption is increased because the function is added to the camera side. The block diagram of the camera system of the embodiment. The block diagram of the video signal processing unit 16 of the camera 1. The block diagram of the video signal processing unit 2526 of the receiving unit 2. The block diagram of the video signal processing unit 16 of the modified example. The block diagram of the artificial intelligence 50 on the receiving unit 2 side of the modified example.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 4 shows the configuration of the camera system according to the embodiment. The camera system includes a surveillance camera 1 and a receiving unit 2. As a feature of this embodiment, a video signal processing unit 26 is added to the receiving unit 2.
The surveillance camera 1 includes an image sensor (CMOS) 11, an FPGA 12, a memory 13, a ROM (Read Only Member) 14, and a power supply separator 15.
CMOS 11 is an image sensor that is mainly sensitive to visible light, has a digital video interface such as LVDS that outputs a video signal, and a control interface such as I2C, and is connected to the FPGA 12 by both interfaces. .. It is desirable that CMOS 11 can output at least one type of video signal in 1080p, 1080i or 720p.

The FPGA 12 includes a video signal processing unit 16, an HD-VLC compression unit 17, and a CPU (Central Processing Unit) core 18. A device in which a CPU and PFGA are fused in this way is also called a programmable SoC.
The memory 13 is, for example, a general DDR2 (Double Data Rate 2) DRAM (Dynamic Random Access Memory), and is used as a frame memory or the like.

The ROM 14 is a flash memory or the like that stores the configuration data of the FPGA 12 and the instructions and data for the CPU core 18.
The power separator 15 is connected to the coaxial cable 3 and separates the video signal transmitted from the surveillance camera 1 to the receiving unit 2 and the power supplied from the receiving unit 2 to the surveillance camera 1.

The video signal processing unit 16 is a reconfigurable hardware resource in the FPGA 22, and a soft macro for video signal processing is materialized. The video signal processing unit 16 is configured so that the memory 13 can be accessed.
The HD-VLC compression unit 17 embodies hard macros such as Dirac-type video compression processing and a cable driver. HD-SDI video signals are input, converted into signals preferable for cable transmission, and output. do. The HD-VLC compression unit 17 can reproducibly transmit HD-SDI including ancillary data by the receiving unit.

The CPU core 18 is a CPU or DSP (Digital Signal Processing) built in the FPGA 12, and includes an interrupt controller, a timer, a DMA, a cache memory, a memory controller, and the like like a general one-chip microcomputer. Then, the CPU core 18 transfers control information or video data between the video signal processing unit 16, the HD-VLC compression unit 17, and the CMOS 11 via registers, memory-mapped I / O, and a serial bus, and operates them. Can control and play a part in video signal processing. Further, as a feature of this example, the CPU core 18 selects some streams to be loaded into the video signal processing unit 16 from among a plurality of bitstreams for partial configuration stored in the ROM 14, and actually You can control the loading to. Further, the result of stream selection or the information specifying the image processing pipeline materialized on the camera 1 side can be passed to the HD-VLC compression unit 17 as ancillary data.

The receiving unit 2 includes a format conversion IC 21, an FPGA 22, a memory 23, a ROM 24, and a power supply superimposing device 25. The FPGA 22, the memory 23, and the ROM 24 are the same as the FPGA 12, the memory 13, and the ROM 14 in terms of hardware. The FPGA 22 is different from the FPGA 12 in that the HD-VLC restoration unit 27 is materialized instead of the HD-VLC compression unit 17. The ROM 24 differs from the ROM 14 in that configuration data and instructions that are required only on the camera side are omitted, and configuration data and instructions that are required only on the receiving unit side are added.

The format conversion IC 21 is a dedicated IC that inputs predetermined digital video data, converts it into a video interface such as VBS (Video, Blanking, and Sync), RGB, DVI-D, and outputs it. When an HD-SDI signal is input and VBS is output, the format conversion IC 21 performs resample (downsample), progressive / interlaced conversion, and the like.

The power superimposing device 25 is connected to the coaxial cable, receives the video signal transmitted from the camera 1, and supplies the coaxial cable 3 with DC 12V as the power source for the surveillance camera 1.

The video signal processing unit 26 is a reconfigurable hardware resource in the FPGA 22, and a soft macro for video signal processing is materialized. The video signal processing unit 26 is configured so that the memory 23 can be accessed.

The HD-VLC restoration unit 27 embodies hard macros such as a Dirac-type video decoding process and a cable equalizer, and develops the video signal received from the paired HD-VLC compression unit 17 to develop the video signal. Outputs a video signal equivalent to HD-SDI. The ancillary data is embedded in HD-SDI and also provided to the CPU core 28.

The CPU core 28 controls to select some streams to be loaded into the video signal processing unit 26 from among a plurality of streams for partial configuration stored in the ROM 24, and to actually load the streams. be able to. Further, when the result of stream selection on the camera 1 side or the information specifying the materialized image processing pipeline is received from the HD-VLC restoration unit 27 as ancillary data, a partial or whole of the FPGA 22 is received accordingly. Perform the figuring of. When performing partial and dynamic updates, the reconfiguration can be achieved without stopping the video by providing a signal bypass wiring.

FIG. 5 is a block diagram of the video signal processing unit 16 of the surveillance camera 1 of the present embodiment. Here, the camera 1 is assumed to be a color single-panel camera. The image processing pipeline of the surveillance camera 1 is composed of a gain-up device 31, a color corrector 32, a knee processor 33, an enhancer 34, and a gamma correction 35. As a feature of this example, the function that requires memory processing is deleted to reduce power consumption. The input / output I / F of each stage after the gain-up device 31 is unified. For example, the bit depth of the pixel data is 14 bits, and the synchronous I / F is used. In the case of a color image, it is desirable that the data after color separation is unified into a predetermined color format such as RGB 4: 4: 4 or YUV 4: 2: 2.

The gain-up device 31 receives pixel data directly from CMOS 11, digitally gain-ups it by multiplying it by a predetermined coefficient, and outputs it to the CSC. When the number of pixels of CMOS 11 is large, binning processing may be performed, and when different exposures are made for each line, HDR processing may be performed. The bit depth of the pixel data from the CMOS 11 may be different from the bit depth of the data output by the gain up 31.

The color corrector 32 appropriately demosizes the image taken through the color filters of RGB and CMYG (Cyan (light blue), Magenta (magenta), Yellow (yellow), Green (green)), and then demosizes the image. A color-corrected video signal is output by white balance adjustment, conditional linear calculation as in Patent Document 10 (Patent No. 2995683), projection conversion in a color space as in Patent Document 11, and the like.

The knee processor 33 performs a process of making the saturation of the brightness less likely to occur even in a small dynamic range by reducing the inclination (knee slope) of the tone mapping with respect to the brightness exceeding a predetermined knee point. Further, when the brightness at which saturation starts in the color most likely to be saturated in the color channel of CMOS 11 is exceeded, a process for preventing false color may be performed by setting the color difference to 0 or the like.

The enhancer 34 performs spatial filter processing for emphasizing the details of the image and spatially smoothing the gradation. Alternatively, a filter process for suppressing outliers such as a median filter is performed, especially when a high gain is given by a gain-up device (low illuminance).

Gamma correction 35 performs gamma processing corresponding to display gamma of about 1.9 to 2.4 in accordance with ITU-R Rec.709, BT.1886, etc., and if necessary, a format compatible with HD-SDI (YCbCr). The color space is converted to (Y (luminance), Cb (blue color difference), Cr (red color difference)) or RGB) and output to the HD-VLC compression unit 17.

FIG. 6 is a block diagram of the signal processing unit 23 of the receiving unit 2 of the present embodiment. The signal processing unit 23 receives a baseband video signal compatible with HD-SDI from the HD-VLC restoration unit 27. The image processing pipeline of the signal processing unit 2523 is composed of a digital noise reduction processing unit 36, a storage processing unit 37, a wide dynamic range processing unit 38, a fog haze correction device 39, and a memory IF 40.

The digital noise reduction processor 36 is an adaptive noise filter that performs time domain filtering with weights optimally controlled according to the speed of movement of the subject, SN, and the like. Generally, it is often implemented as a recursive filter.
The storage processor 37 simply adds and outputs images of a plurality of frames. For example, when accumulating N frames, the output frames are updated every N frames.

The wide dynamic range processor 38 performs tone management by a method such as histogram normalization, histogram adjustment, and Retinex. Although this process does not require a large amount of time domain manipulation on the pixel data, it requires a considerable amount of memory bandwidth for histogram calculation, illumination light model, or visual response model generation.

The memory IF 40 provides read and write access to the memory 23 to the digital noise reduction processor 36, the storage processor 37, and the wide dynamic range processor 38. It should be noted that these do not necessarily have to be able to operate at the same time, and only a part of them can be operated within the range allowed by the memory bandwidth.

The fog haze corrector 39 removes components of scattered light due to fog and haze by using a dark channel image composed of the smallest values in the color channels in the vicinity of the pixel of interest. This processing has an effect of improving visibility similar to the wide dynamic range processing, and can be realized with a processing amount that is not much different from that of the enhancer 34.

In the present embodiment, the FPGA 12 mounted on the surveillance camera 1 and the FPGA 22 mounted on the receiving unit 2 are the same or highly compatible devices, so that the configuration data corresponding to the functions from the gain-up device 31 to the memory IF 40 are supported. Can be shared at the stream level, the hardware description language level, or at least the higher language level, and as a result, the design assets can be effectively utilized. It is not necessary to make each functional part from the gain-up device 31 to the memory IF40 independently partially reconfigurable, and a configuration stream of a plurality of patterns of a combination of these functional parts is prepared and loaded in a selectable manner. In some cases, this can be achieved more easily. Here, there is no intention of completely denying the duplication of the functions of the camera 1 and the receiving unit 2, and in order to reduce the types of streams to be prepared, the functions that consume less power and resources may be duplicated.

By selectively embodying the functional parts commonly held in the ROM 14 of the surveillance camera 1 and the ROM 24 of the receiving unit 2, the surveillance camera 1 and the receiving unit 2 can be placed between them as if they were. Behave so that the functions can be interchanged. For example, if an ex post facto notices that a camera has been installed in an environment that is specifically prone to high temperatures, the function of such a camera can be simplified to the one shown in FIG. 4 to reduce heat generation. be able to. In particular, stopping wide dynamic range processing that uses the frame memory is effective in reducing heat generation. At night when there is no risk of overheating, the accumulation process in the surveillance camera 1 can be appropriately enabled.

In this example, since the transmission between the surveillance camera 1 and the receiving unit 2 is unidirectional, the image processing performed by the surveillance camera 1 needs to be performed autonomously without any instruction from the receiving unit 2. Further, it is necessary for the receiving unit 2 to know the contents of the processed processing. Therefore, the CPU core 18 acquires the parameters set in the CMOS 11 and the parameters and internal states used by each function unit of the video signal processing unit 16 through registers and the like, and includes them in the ancillary data to include the receiving unit. It is desirable to send to 2.

For transmission between the surveillance camera 1 and the receiving unit 2, any well-known method, for example, a method capable of bidirectional communication such as HDcctv ™ and SMPTE 2022-6 can be used.

Hereinafter, a modified example of the embodiment will be described with reference to FIGS. 7 and 8. In the modified example, the artificial intelligence 50 is provided on the receiving unit 2 side, and a communication means capable of transmitting from the artificial intelligence 50 to the surveillance camera 1 is provided.
FIG. 7 is an internal block diagram of the color corrector 42 of the video signal processing unit 16 according to the modified example. The modified video signal processing unit 16 includes a color corrector 42 as a replacement for the color correction unit 32 or additionally.
The color corrector 42 executes a part of mapping (dimensional compression) processing that needs to be performed in real time in a process called manifold learning or multidimensional scaling.
Specifically, the color corrector 42 receives a video signal of four or more channels of color, performs non-linear color conversion in real time so as to preserve subtle differences in color as much as possible, and performs a video signal of three channels of color. Is output. The input signal is, for example, a complementary color sensor of Cy, Mg, Ye, G, or an image in a color format in which one of the G channels of the bayer sensor is changed to another sensitivity wavelength (for example, near infrared rays). Called a video signal.

The clustering device 43 classifies each pixel of the four-color video signal into N clusters, and outputs the result as a cluster ID. Clustering is performed in advance by manifold learning or the like, and each cluster corresponds to a submanifold. The classification can be performed by well-known methods such as kd tree, R tree, and LSH. Non-linear methods such as LLE, t-SNE, and Laplacian eigenmap can be used for manifold learning and the like. Alternatively, clustering can also be achieved by DSMAP, or simple evenly spaced divisions of the color space.

When the mapping table 44 receives the cluster ID, it outputs the representative vector (source vector) of the corresponding cluster and the mapped vector of the mapping destination. In this example, the representative vector is four-dimensional and the mapped vector is three-dimensional. The representative vector may be output from the clustering device 43.
When the transformation matrix table 45 receives the cluster ID, it outputs a 3-by-4 projected transformation (submersion) matrix A optimized in the vicinity of the representative vector of the cluster.

The contents of the mapping table 44 and the transformation matrix table 45 are initially set to those acquired by manifold learning or the like for the image of a predetermined subject, and then the learning of artificial intelligence at the output destination of the receiving unit 2 is performed. In the process of, it can be modified by backpropagation. If the clustering device 43, the mapping table 44, and the transformation matrix table 45 are materialized as an embedded block RAM (Random Access Memory) in the FPGA, the contents can be easily rewritten. The version of each table is managed at the time of rewriting, and by notifying the used version as ancillary data downstream from the camera 1, the artificial intelligence 50 can know what kind of color conversion was performed by the surveillance camera 1. can.

When performing manifold learning online, it is desirable that the video signal processing unit 16 also has a configuration in which pixels are randomly selected, a configuration in which variance is calculated and held for each cluster, and the like. On the other hand, the selection of the representative vector, the eigenvalue calculation for obtaining the projective transformation matrix A, the steepest descent algorithm, and the like can be processed by the CPU core 18 or the HD-VLC restoration unit 27. Manifold learning and the like have a degree of freedom, so if some constraints are required, it is possible to impose constraints that make the mapping close to the general RGB color space. Further, if consistency with vision is emphasized, injectiveness may be sacrificed. Also, if the color distribution of the subject is biased, it does not have to be surjective. When trying to recognize various types of subjects, the nature of the original local isometry can be prioritized without imposing additional restrictions. Note that local isometry is not unique and can be learned.

The subtractor 46 subtracts the representative vector from the input four-color video signal and outputs the difference vector.

The matrix multiplier 47 multiplies the difference vector, which is a column vector, by the projective transformation matrix from the back and outputs the result.
The adder 48 adds a mapped vector to the output of the matrix multiplier 47 and outputs it as a result of the color correction unit 42.

The color corrector 42 can individually correct all the pixels of the image, but even if the pixels are thinned out to 1/2 according to the format such as 4: 2: 2 or 4: 2: 0, the color corrector 42 may be executed. good.

FIG. 8 is a configuration diagram of an artificial intelligence 50 that learns and recognizes an image corrected by the color corrector 42. The artificial intelligence 50 includes a convolutional neural network (CNN) 51, a learning processor 52, a learning data store 53, a learning history store 54, and a color corrector 55, and can perform semi-supervised learning online. As the hardware of the artificial intelligence 50, a computer suitable for high performance computing (HPC), particularly a workstation or a PC equipped with GPGPU (General-Purpose computing on Graphics Processing Units) can be used.

CNN51 is a multi-layer NN that recognizes known objects contained in an image, such as Faster R-CNN, SSD (Single Shot MultiBox Detector), YOLO, etc., and outputs the score and area coordinates of the recognized object. do. It is assumed that the CNN 51 has fully learned about known objects before it is used for recognition, and in this example it has learned about multiple symptoms for a particular crop (eg, plant leaves). Part or all of the CNN 51 can be realized by a GPGPU, main memory, or the like provided in the computer.

The learning processor 52 is a processor that learns the CNN 51, and can be realized by a CPU and a main memory provided in the computer.

The training data store 53 stores labeled images that are training data. The learning data store 53 can be realized by a non-volatile storage device such as an SSD included in the computer.
Labeled images are images containing leaves and stems of plants, which are deficient or excessive in nitrogen, phosphoric acid, potassium, magnesium, calcium, etc., as well as insufficient or excessive sunshine or irrigation, and are affected by specific pests. A label is attached to indicate the symptoms of the time. The labeled image is added with color expression information indicating what kind of color space is expressed or color correction is received. The training data store 53 temporarily holds the input image, adds a label and color expression information to the partial image of the temporarily held image according to the instruction of the learning processor 52, and registers it as a new labeled image. can do.

The color representation information is, for example, a color matrix (3 * 3, 4 * 3,) for converting a labeled image into a reference color system, for example, sRGB (IEC 61966-2-1) or vice versa. Or each coefficient of 4 * 4), information (color temperature, ID) that identifies such a color matrix. When the labeled image is acquired by the camera 1, the version of each table in the color corrector 42 can be used as the color expression information.

The learning history store 54 stores the initial contents of each table used for mapping by the color corrector 42, and holds the history when those tables are modified by the backpropagation performed by the learning processor 52. do. The number of corrections in one modification (version upgrade) is small, and it is more efficient to apply the modification to the initial table according to the history and generate it each time, rather than keeping a snapshot of the table for each modification. Since it is not necessary to reproduce the history past the oldest labeled image stored in the training data store 53, the contents of each table at the time of the oldest labeled image acquired by the camera 1 are initially set. Can be content.

The color corrector 55 approximately converts the labeled image stored in the training data store 53 into an image that has been color-corrected based on the latest learning results. When the labeled image is acquired as a general sRGB image, the color corrector 55 can achieve color correction with the same configuration and / or method as the color corrector 42. When the labeled image is acquired by the camera 1 and corrected by the color corrector 42, it is converted using the conversion table generated based on the contents of each table at the time of acquisition and each of the latest tables. The conversion table is generated by the following steps 1 to 4.

Step 1: Associate all the mapped vectors in the table at the time of acquisition with the mapped vectors in the latest table that best correspond in the original space (the four-color video signal input to the color corrector 42). This is necessary when clustering or the representative vector is updated after the time of acquisition, and is because each cluster used in the clusterer 43 at the time of acquisition is associated with the latest cluster that corresponds spatially best. And expressed as an old cluster ID-new cluster ID conversion table. Alternatively, the method of searching the latest mapping table for the representative vector closest to each representative vector of the mapping table 44 at the time of acquisition can also be performed.

Step 2: Set the clusterer of the color corrector 55 so that all the mapped vectors associated with the old cluster ID are classified in the three color space and the new cluster ID corresponding to the old cluster ID is output. do.

Step 3: The mapping table of the color corrector 55 is set to the latest contents.

Step 4: For all the transformation matrices associated with the new cluster ID in the latest transformation matrix table, the inverse matrix of the transformation matrix of the old cluster ID corresponding to the new cluster ID in the transformation matrix table at the time of acquisition is displayed from the right. Multiply to get a transformation matrix with 3 rows and 3 columns. Then, they are associated with the new cluster ID and set in the transformation matrix table of the color corrector 55. The matrix multiplier of the color corrector 55 is configured to multiply a transformation matrix of 3 rows and 3 columns.

Here, learning by the learning processor 52 will be described. Learning generally has the following first to third processes.
[First process]
First, the learning processor 52 learns the CNN 51 using the initially prepared labeled image. That is, the image of the labeled image is input to the CNN 51, and the error (difference from the correct answer value of the label) of the value of the output node of the CNN 51 corresponding to the label is calculated. Then, the output value of the intermediate layer node (which is an error for the intermediate layer node) and the weight that the output node should input in order to output the correct answer value are adjusted. A stochastic steepest descent method can be used for this adjustment. By performing this adjustment while going back to the input side, the entire CNN 51 can be learned. Other learning specific to the CNN 51 used may also be performed. For example, in order to improve the generalization ability for differences in position, size, and viewpoint, internal parameters may be learned using labeled images to which various affine transformations are applied.

[Second process]
Somewhere in the middle layer of CNN51, the RGB-by-RGB features are converted to features on another scale. When backpropagating from this intermediate layer, the weight of each RGB color was adjusted in the first process.
After those adjustments have converged, the second process instead adjusts the color conversion weights of the color corrector 42. That is, by updating each table of the color corrector 42 by backpropagation, the color weights are adjusted with a much higher degree of freedom than the adjustment for three colors. Actually, learning is performed by backpropagating the color corrector 52 that reproduces the correction of the color corrector 42 and adjusting the color weight in a pseudo manner on the labeled image input to the CNN 51. NS.

[Third process]
The CNN 51 identifies an image acquired by the camera 1 and processed by the color corrector 42, and uses the result to perform weakly supervised learning. In the simplest weakly supervised learning, among the identification results, those that are sufficiently reliable are assumed to be correct answers and stored in the learning data store 53 as a new labeled image. Then, learning is repeated using these added labeled images.

The modified examples described above are not limited to the identification of plants, but various identifications based on images, such as diagnosis of the appearance of concrete structures, detection of pollution of the sea and rivers, red tide and other conditions, identification of oil, etc. It can be used to detect leaks of substances.

Further, the present invention is, for example, as a method or device for executing a process according to the present invention, a program for realizing such a method on a computer, a non-transient tangible medium for recording the program, or the like. It can also be provided.

The camera system according to the above-described embodiment is configured by combining a surveillance camera equipped with an FPGA for signal processing and a receiving unit equipped with the same type of FPGA, so that signal processing can be performed by the FPGA in the camera. The function that was being used can be incorporated into the receiving unit. Also, when adding a new function, the power consumption of the camera can be reduced by installing it in the receiving unit. Of course, it is possible to replace the functions on the receiving unit side with those on the camera side, so it is possible to consider combining the necessary functions with the camera and the receiving unit.
The effect of reducing power consumption depends on the replacement of functions, but in particular, by installing a wide dynamic and digital noise reduction function that uses frame memory on the receiving unit side, a reduction effect of several watts can be expected.

The present invention can be widely used in digital cameras, video cameras and the like. It is not limited to those that transmit images in real time, but can also be applied when a camera stores an image on a recording medium and reproduces it on a receiving unit such as a PC.

101: Surveillance camera, 102: Receiving unit, 103: Coaxial cable, 1: Camera, 2: Receiving unit, 3: Coaxial cable, 11: Imaging element (CMOS), 12: FPGA, 13: Memory, 14: ROM, 15 : Power separator, 16: Video processing unit, 17: HD-VLC compression unit, 18: CPU core, 19: Optical system, 21: Format conversion IC, 22: FPGA, 23: Memory, 24: ROM, 25: Power supply Superimposition, 26: Video processing unit, 27: HD-VLC restoration unit, 28: CPU core, 31: Gain up unit, 32: Color corrector (masking), 33: Knee processing unit, 34: Enhance, 35: γ (Gamma) correction, 36: Digital noise reduction processor, 37: Accumulation processor, 38: Wide dynamic range processor, 39: Fog haze corrector, 40: Memory IF, 42: Color corrector, 43: Clustering device, 44: Mapping table, 45: Transformation matrix table, 46: Subtractor, 47: Matrix multiplier, 48: Adder, 50: Artificial intelligence, 51: Convolutional neural net (CNN), 52: Learning processor, 53: Learning Data store, 54: Learning history store, 55: Color corrector, 56: Switcher.

Claims (3)

A camera system that connects a surveillance camera and a receiving unit with a coaxial cable.
The surveillance camera is a first non-volatile device that holds a first dynamic reconstruction device that processes a signal output from an imaging element and a first configuration data that sets the configuration of the first dynamic reconstruction device. a memory, along with loading the first configuration data to the first dynamically reconfigurable device, information capable of specifying the video signal processing performed by the first dynamically reconfigurable devices configured by the load is the It has a first CPU that controls to output from a coaxial cable, and has.
The receiving unit sets a second dynamic reconstruction device that processes a video signal output from the surveillance camera and received via the coaxial cable, and a plurality of configurations of the second dynamic reconstruction device. Based on the second non-volatile memory that holds the second configuration data and the identifiable information output from the surveillance camera, the second CPU that selects and loads what should be loaded from the plurality of second configuration data. And have
A camera characterized in that the video signal of a predetermined format output from the receiving unit is a signal that has undergone video signal processing shared by the first dynamic reconstruction device and the second dynamic reconstruction device. system. The camera system according to claim 1.
The receiving unit has a memory, and the second dynamic reconstruction device uses the memory to perform video signal processing accompanied by inter-frame calculation.
The first dynamic reconstruction device is a camera system characterized in that it does not perform video signal processing accompanied by inter-frame calculation. The camera system according to claim 1.
The first dynamic reconstruction device has a first color corrector that converts an input video signal of four or more colors into a video signal of three colors.
The receiving unit has artificial intelligence that learns and recognizes a predetermined subject reflected in the image corrected by the first color corrector.
The first color corrector performs color correction processing using a clustering device, a mapping table, a transformation matrix table, a subtractor, a matrix multiplier, and an adder.
The artificial intelligence is a camera system including a multi-layer neural network, a learning processor, a learning data store, a learning history store, and a second color corrector.

Images