JP2019047214A - Smart camera, image processing device, smart camera system, data transmission method and program - Google Patents

Abstract

To maintain continuity of image processing before and after video switching.SOLUTION: A smart camera is communicable with an image processing device 15, and includes multiple imaging parts 1a-1m, a selection part 15b, a switching part 15c, a feature data generation part 15d, a synchronization processing part 20, a multiplexing part 19, and a transmission part 21. Each of the imaging parts generates video data. The selection part selects an imaging part generating video data corresponding to image processing in the image processing device. Every time when another imaging part is selected by the selection part, the switching part switches and outputs the video data from the selected imaging part, while synchronizing mutual frame phases. The feature data generation part generates feature data of a video from the selected imaging part, over a prescribed period including a moment in time of switching and outputting. The synchronization processing part synchronizes the switched and output video data with the feature data. The multiplexing part multiplexes the synchronized video data and feature data into a transmission stream.SELECTED DRAWING: Figure 2

Description

  Embodiments of the present invention relate to technology for smart cameras.

  Attention is focused on so-called smart cameras provided with an imaging sensor and a processor. In recent years, due to the miniaturization and price reduction of sensor devices, smartphones and on-vehicle cameras equipped with a plurality of cameras have also been sold. The generation of stereo images using compound eye cameras and the generation of images having distance information (distance images) have also been studied. An array camera in which a plurality of camera devices are arranged in an array is also known. Furthermore, a multispectral camera (also called a hybrid camera) in which a visible light camera, a near infrared camera, and a far infrared camera are mounted in a common housing is also known. These next-generation cameras are expected to be connected to a center device via a wired network or a wireless network and applied to a remote monitoring system or the like.

JP, 2005-328479, A

"Compound-view distance image CMOS image sensor", [online], [Search on June 15, 2017], Internet, <URL: http://www.toshiba.co.jp/rdc/detail/13_t24.htm> “Next-generation Camera for Multifunctionalization”, [online], [search on June 15, 2017], Internet, <URL: http://toshiba.semicon-storage.com/design_support/elearning/keytechnology/__icsFiles /afieldfile/2010/11/05/edn1011_39_47feature02.pdf>

  It is rare to send the video data of all the cameras of the array camera to the center device, and in many cases the image of one of the cameras is switched and output. For example, in order to detect a person, fixed point observation is performed with a visible light camera during the daytime, but it is switched to an infrared camera during the nighttime. In this way, the occupied bandwidth required for transmission of a stream containing video is minimized.

  However, when the video is switched, processing on the receiving side of the transmission stream can not catch up, and some time-series image analysis data may be lost. Technically, this is referred to as "a discontinuity in image processing." For example, if a color image is suddenly switched to a monochrome image, it is difficult to continue the image processing although the center apparatus having received the image acquires an image of the same field of view. There are various factors that cause discontinuities such as differences in color tone between cameras, differences in wavelength, differences in contrast, differences in focus, differences in screen size, and differences in angle of view. Image processing may be reset if discontinuities become extreme.

  As described above, there is a technical problem in the technical field that it is difficult to maintain the continuity of image processing before and after video switching (frame switching). The situation is the same in a system in which a plurality of monocular cameras observe a common field of view.

  Therefore, it is an object of the present invention to provide a smart camera system capable of maintaining continuity of image processing before and after video switching, a smart camera usable in this system, an image processing apparatus, a data transmission method and a program. .

  According to the embodiment, the smart camera can communicate with the image processing apparatus, and the plurality of imaging units, the selection unit, the switching unit, the feature data generation unit, the synchronization processing unit, the multiplexing unit, And a transmitter. The imaging units respectively generate video data. The selection unit selects an imaging unit that generates video data corresponding to image processing in the image processing apparatus. The switching unit switches and outputs the video data from the selected imaging unit in synchronization with each other's frame phase each time another imaging unit is selected by the selection unit. The feature data generation unit generates feature data of the image from the selected imaging unit over a predetermined period including the time point of the switching output. The synchronization processing unit synchronizes the switched and output video data with the feature data. The multiplexing unit multiplexes synchronized video data and feature data into a transport stream. The transmission unit transmits the transmission stream to the image processing apparatus.

FIG. 1 is a system diagram showing an example of a monitoring camera system according to the embodiment. FIG. 2 is a block diagram showing an example of a camera. FIG. 3 is a block diagram showing an example of the image processing apparatus. FIG. 4 is a diagram showing an example of information exchanged between the camera and the image processing apparatus. FIG. 5 is a flowchart illustrating an example of the processing procedure of the camera in the embodiment. FIG. 6 is a diagram showing a TS basic system of a transport stream. FIG. 7 is a diagram showing an example of a transport stream including synchronously multiplexed feature data. FIG. 8 is a diagram showing an example of the feature data parameter. FIG. 9 is a figure for demonstrating the effect | action in embodiment. FIG. 10 is a system diagram showing another example of the monitoring camera system. FIG. 11 is a system diagram showing another example of the monitoring camera system.

Next, embodiments of the present invention will be described with reference to the drawings. In this specification, an image is understood as a still image or an image of one frame constituting an image. On the other hand, a video is a set of a series of images and can be understood as a moving image.
A platform for linking multiple smart cameras with a cloud computing system (cloud) and utilizing video data as big data is being developed. For example, use of video data in fixed point observation for disaster prevention, traffic monitoring, infrastructure monitoring such as roads and bridges, person search and person tracking, and tracking of suspicious persons is being considered.

  FIG. 1 is a system diagram showing an example of a monitoring camera system according to the embodiment. The system illustrated in FIG. 1 includes a plurality of cameras C1 to Cn as smart cameras, and an image processing apparatus 200 provided in the cloud 100.

  The cameras C1 to Cn are installed at different places. For example, the cameras C3 to C5 are disposed in an area A including a building street where high-rise office buildings are lined, the cameras C6 to Cn are disposed in an area B including a suburban residential area, and the cameras C1 and C2 are areas A , And B are arranged. Each of the cameras C1 to Cn has, for example, a lens and an imaging sensor, and generates video data by imaging a real space at each location.

  The image processing apparatus 200 is connected to cameras C1 to Cn, a base station BS of a mobile communication system, a database, or the like via a communication network. As a protocol of the communication network, for example, TCP / IP (Transmission Control Protocol / Internet Protocol) can be used. The relay network 101 may be interposed between the camera and the cloud 100.

  The image processing apparatus 200 collects video data transmitted from each of the cameras C1 to Cn as a transmission stream (transport stream). The image processing apparatus 200 performs image processing such as shading, filtering, or contour extraction on the collected video data.

  Vehicle V1 or cellular phone P1 is also accessible to cloud 100 via base station BS. The on-vehicle camera of the vehicle V1 and the camera of the cellular phone P1 can also operate as a smart camera.

  In the areas A and B, for example, edge servers S1 and S2 are installed, respectively. The edge server S1 requests the cloud 100 for data according to the features of the area A (for example, a large number of people in the daytime), and provides a service according to the acquired data, and a platform for providing the service. Realize the construction. In addition, the edge server S1 may function as a resource such as a high-speed arithmetic processing function and a large-capacity storage for causing the user to use the acquired data. The edge server S2 requests data from the cloud 100 according to the features of the area B (for example, a large number of children and schools, etc.), and provides a service according to the acquired data and provides a service. Realize the construction. In addition, the edge server S2 may function as a resource for causing the user to use the acquired data.

  The usage form of the cloud computing system is SaaS (Software as a Service) that provides an application as a service, PaaS (Platform as a Service) that provides a platform for operating an application as a service, and It is roughly classified into IaaS (Infrastructure as a Service) that provides resources such as high-speed arithmetic processing functions and large-capacity storage as a service. The cloud 100 can be applied in any form.

  FIG. 2 is a block diagram showing an example of the camera C1 shown in FIG. The cameras C2 to Cn also have the same configuration. The camera C1 includes a plurality of imaging units 1a to 1m, a switch unit 10, a processing circuit 15, a memory 16, a sensor unit 17, a transmitting unit 21, a receiving unit 22, a synchronization processing unit 20, and a multiplexing unit (Multiplexer: MUX) 19 Equipped with

  The imaging units 1a to 1m capture video in respective fields of view and individually generate video data. The imaging units 1 a to 1 m each include, for example, a lens 11, an aperture mechanism 12, an image sensor 13, and an encoding unit 14. An image within the field of view of the lens 11 is imaged on the image sensor 13 through the lens 11 and the aperture mechanism 12. The image sensor 13 is an image sensor such as a CMOS (complementary metal oxide semiconductor) sensor, and generates, for example, a video signal at a frame rate of 30 frames per second. The encoding unit 14 encodes the video signal output from the image sensor 13 to generate video data. The video data from the imaging units 1 a to 1 m are transferred to the switch unit 10 and the processing circuit 15 via the internal bus 23.

  The imaging wavelength bands of the imaging units 1a to 1m may be different from one another. For example, imaging wavelength bands such as visible light, near infrared light, far infrared light, and ultraviolet light may be individually assigned to each of the imaging units 1a to 1m. That is, the camera C1 may be a multispectral camera.

  The sensor unit 17 includes, for example, the device type of the imaging units 1a to 1m, the number of pixels, the frame rate, the sensitivity, the focal length of the lens 11, the light amount of the diaphragm mechanism 12, angle of view, absolute time information, camera direction information, zoom magnification information And parameter information such as wavelength characteristics of the filter are acquired via the data bus 24 and transferred to the processing circuit 15 and the memory 16. The sensor unit 17 also has a positioning function using, for example, a GPS (Global Positioning System), and acquires position information of the camera C1 and time information by positioning processing using a positioning signal received from a GPS satellite. The sensor unit 17 transfers the acquired position information and time information to the processing circuit 15 and the memory 16. The position information is important when the camera itself moves, for example, when the camera is mounted on a cellular phone or a car. Further, the sensor unit 17 includes, for example, sensors such as a temperature sensor, a humidity sensor, and an acceleration sensor, and acquires information on the environment in which the camera C1 is installed as sensor information by using these sensors. The sensor unit 17 transfers the acquired sensor information to the processing circuit 15 and the memory 16.

  The switch unit 10 selectively sends the video data output from any of the imaging units 1 a to 1 m to the synchronization processing unit 20. The processing circuit 15 determines which one of the imaging units 1a to 1m the video data is to be selected.

  The synchronization processing unit 20 synchronizes the video data from the switch unit 10 with the feature data including the feature amount generated from the video data. The feature amount is generated by the processing circuit 15 based on the video data. The feature data is generated by the processing circuit 15 based on the feature amount, parameter information transferred from the sensor unit 17, sensor information, position information, time information, and the like.

  The video data precedes the feature data in time, for example, by the time it takes for the feature data to be generated based on the video data. The synchronization processing unit 20 temporarily stores the video data in the buffer memory for the preceding time. The synchronization processing unit 20 synchronizes the video data with the feature data by reading the video data from the buffer memory at the timing when the feature data is created. The synchronized video data and feature data are passed to the multiplexing unit 19.

  The multiplexing unit 19 multiplexes the video data and the feature data synchronized with the video data into, for example, a transport stream of the MPEG-2 (Moving Picture Experts Group-2) system.

  The transmission unit 21 transmits the transmission stream in which the video data and the feature data are multiplexed to the image processing apparatus 200 of the cloud 100 via the line L.

  The receiving unit 22 acquires data transmitted from the cloud 100 or the image processing apparatus 200 via the line L. The data transmitted from the image processing apparatus 200 includes, for example, a message regarding image processing in the image processing apparatus 200. The message includes, for example, a type of image processing method, and information indicating a video parameter (such as contrast value and signal-to-noise ratio) to be prioritized. The acquired data is transferred to the processing circuit 15 and the memory 16.

  The memory 16 is, for example, a semiconductor memory such as SDRAM (Synchronous Dynamic RAM) or a non-volatile memory such as EPROM (Erasable Programmable ROM) and EEPROM (Electrically Erasable Programmable ROM). The memory 16 stores a program 16a for causing the processing circuit 15 to execute various functions according to the embodiment, and feature data 16b.

  The processing circuit 15 controls the operation of the camera C1 based on the program stored in the memory 16. The processing circuit 15 is an LSI (Large Scale Integration) that includes, for example, a multi-core CPU (Central Processing Unit) and is tuned to execute image processing at high speed. The processing circuit 15 can also be configured by an FPGA (Field Programmable Gate Array) or the like. The processing circuit 15 may be configured using an MPU (Micro Processing Unit) instead of the CPU.

  The processing circuit 15 includes an image analysis unit 15a, a selection unit 15b, a switching control unit 15c, and a feature data generation unit 15d as processing functions according to the embodiment. In the image analysis unit 15a, the selection unit 15b, the switching control unit 15c, and the feature data generation unit 15d, the program 16a stored in the memory 16 is loaded into the register of the processing circuit 15, and the processing circuit 15 is processed as the program progresses. Can be understood as a process generated by performing arithmetic processing. That is, the program 16a includes an image analysis program, a selection program, a switching program, and a feature data generation program.

  The image analysis unit 15a performs image analysis and video analysis on the video data transferred from the imaging units 1a to 1m. Thereby, the image analysis unit 15a generates feature amounts for each of the video data transferred from the imaging units 1a to 1m. In the present embodiment, the feature amount is used as, for example, an index indicating a feature of an image and an index indicating a feature of an image. The feature amount includes, for example, information for identifying the nature of an image such as a visible light image, an infrared image, a far infrared image, an ultraviolet image, a color image, or a monochrome image. More specifically, the feature amount includes a histogram of oriented gradients (HOG) feature amount, contrast, resolution, S / N ratio, color tone, and the like. In addition, a luminance gradient direction co-occurrence histogram (Co-occurrence HOG: Co-HOG) feature amount, a Haar-Like feature amount, and the like are also known as feature amounts.

  The selection unit 15 b determines which image data of the imaging units 1 a to 1 m is appropriate for transfer to the image processing apparatus 200 with respect to the image processing being executed in the image processing apparatus 200. That is, the selection unit 15 b selects an imaging unit that generates video data corresponding to the image processing of the image processing apparatus 200. Specifically, the selection unit 15 b selects one of the imaging units 1 a to 1 m using, for example, a predetermined evaluation value. The evaluation value represents the degree to which the video data corresponds to the image processing of the image processing apparatus 200, and is calculated based on the feature amount calculated by the image analysis unit 15a.

  For example, when the contour extraction process is performed by the image processing apparatus 200, the selection unit 15b has a clear or unclear outline of the image for each of the video data transferred from the imaging units 1a to 1m. Calculate the indicator that represents This index can be numerically represented, for example, in the range of 0 to 100 based on the feature amount of the video data, and the value is used as an evaluation value. When attention is focused on the contour extraction processing, the evaluation value of the imaging unit that outputs a high contrast monochrome image is the highest.

  The selection unit 15 b selects an imaging unit that generates video data with the highest evaluation value.

  It is not desirable for the image processing apparatus 200 to switch the imaging unit frequently. Therefore, the selection unit 15b calculates only the evaluation value of the video data generated by the imaging unit currently in use, unless, for example, a message representing a change in the image processing method or the like is transmitted from the image processing apparatus 200. If the calculated evaluation value is equal to or greater than the predetermined threshold value, the selection unit 15b does not calculate an evaluation value for video data generated by another imaging unit. On the other hand, if the calculated evaluation value is less than the predetermined threshold value, the evaluation value of video data generated by another imaging unit is calculated. The details will be described using the flowchart of FIG.

  Note that, for example, when the image processing employed by the image processing apparatus 200 allows frequent switching of the imaging units, the selection unit 15b may, for example, have a fixed cycle (every minute, every ten minutes, every hour, etc. The evaluation value of each imaging unit may be calculated according to. This makes it possible to flexibly cope with changes in the environment (such as weather).

  The switching control unit 15c and the switch unit 10 switch and output the video data from the selected imaging unit in synchronization with each other's frame phase each time another imaging unit is selected by the selection unit 15b. That is, the switching control unit 15c and the switch unit 10 function as a switching unit. When the imaging environment changes significantly with the passage of time or the request of the image processing apparatus 200 changes, an imaging unit other than the imaging units currently used is selected. Then, the switching control unit 15c synchronizes the frame phase of the video data from the imaging unit selected so far and the frame phase of the video data from the newly selected imaging unit according to the synchronization signal of the internal bus 23. Specifically, by matching the phase of the start symbol of the frame of the video data before switching and the phase of the start symbol of the frame of the video data after switching with the synchronization signal from the outside, the frames of the respective video data Synchronize the phase. When frame synchronization is completed, the switching control unit 15 c switches the switch unit 10 and sends the video data from the selected imaging unit to the synchronization processing unit 20.

  The feature data generation unit 15 d generates feature data of the video data from the imaging unit selected by the selection unit 15 b. Specifically, the feature data generation unit 15d selects the selection unit 15b based on, for example, the feature amount generated by the image analysis unit 15a, the sensor information transferred from the sensor unit 17, position information, time information, and the like. Feature data of video data from the selected imaging unit is generated. The generated feature data is temporarily stored in the memory 16 (feature data 16 b) and sent to the synchronization processing unit 20. Note that after the connection is switched by the switching control unit 15c, the feature data generation unit 15d stops generation of feature data when a period sufficient for the image processing of the image processing apparatus 200 to follow has elapsed. I don't care.

  FIG. 3 is a block diagram showing an example of the image processing apparatus 200. As shown in FIG. The image processing apparatus 200 is a computer provided with a processor 250 such as a CPU or an MPU. The image processing apparatus 200 includes a read only memory (ROM) 220, a random access memory (RAM) 230, a hard disk drive (HDD) 240, an optical media drive 260, and a communication unit 270. Furthermore, a GPU (Graphics Processing Unit) 210, which is a processor with an enhanced function for image processing, may be provided. The GPU can execute operation processing such as product-sum operation, convolution operation, 3D (three-dimensional) reconstruction at high speed.

  The ROM 220 stores basic programs such as a BIOS (Basic Input Output System) and a UEFI (Unified Extensible Firmware Interface), various setting data, and the like. The RAM 230 temporarily stores programs and data loaded from the HDD 240. The HDD 240 stores a program 240 a executed by the processor 250, image processing data 240 b, and feature data 240 c.

  The optical media drive 260 reads digital data recorded on a recording medium such as a CD-ROM 280. Various programs executed by the image processing apparatus 200 are, for example, recorded on a CD-ROM 280 and distributed. The program stored in the CD-ROM 280 is read by the optical media drive 260 and installed in the HDD 240. The latest program can be downloaded from the cloud 100 via the communication unit 270, and the already installed program can be updated.

  The communication unit 270 is connected to the cloud 100, and communicates with the cameras C1 to Cn, and other servers and databases of the cloud 100. For example, various programs executed by the image processing apparatus 200 may be downloaded from the cloud 100 via the communication unit 270 and installed in the HDD 240.

  The communication unit 270 includes a receiving unit 270a. The receiving unit 270a receives a transmission stream including video data from the cameras C1 to Cn via the communication network of the cloud 100.

  The processor 250 executes an operating system (OS) and various programs.

  The processor 250 further includes an image processing unit 250a, a separation unit 250b, a decoding unit 250c, a compensation unit 250d, and a notification unit 250e as processing functions according to the embodiment. In the image processing unit 250a, the separation unit 250b, the decoding unit 250c, the compensation unit 250d, and the notification unit 250e, the program 240a stored in the HDD 240 is loaded into the register of the processor 250, and the processor 250 calculates it as the program progresses. It can be understood as a process generated by performing a process. That is, the program 240 a includes an image processing program, a separation program, a decoding program, a compensation program, and a notification program.

  The image processing unit 250 a performs image processing on video data included in the received transmission stream or a video decoded from the video data, and obtains image processing data such as point cloud data and person tracking data. The image processing data is stored in the HDD 240 as the image processing data 240 b.

  The separation unit 250 b separates the video data and the feature data from the received transmission stream. The separated feature data is stored in the HDD 240 as feature data 240 c.

  The decoding unit 250c decodes the separated video data to reproduce a video.

  The compensation unit 250d compensates for the continuity of the reproduced image based on the separated feature data. That is, the compensation unit 250d performs tone conversion processing and the like of each pixel based on the feature data (sensor information / parameter information) so that the video before and after the switching of the imaging unit gradually changes. For example, the compensation unit 250d performs processing so that the color tone of each pixel of the received video gradually changes during a total of 20 seconds of 10 seconds before switching and 10 seconds after switching. Such processing is known as morphing. It is preferable to make the period for changing the image longer than the period necessary for the image processing function of the image processing apparatus 200 to follow switching of the imaging unit.

  The image frame subjected to the processing by the compensation unit 250d is delivered to the image processing unit 250a. The image processing unit 250a can perform image processing on the compensated video even if the received video data includes a switching portion of the video data.

  The notification unit 250e notifies the cameras C1 to Cn of a message including information on image processing of the image processing unit 250a. For example, information indicating whether to prioritize the type of the image processing method, the contrast of the video, or the signal-to-noise ratio of the video is notified to the cameras C1 to Cn by a message.

  FIG. 4 is a diagram showing an example of information exchanged between the camera C1 and the image processing apparatus 200. As shown in FIG. The camera C1 multiplexes the video data generated by the selected imaging unit and the feature data of the video data into a transmission stream and sends it. The image processing apparatus 200 sends a message regarding image processing to the camera C1 via the cloud 100 as necessary. The camera C1 having received the message selects an imaging unit corresponding to the information described in the message from the imaging units 1a to 1d. Then, the camera C1 multiplexes the video data generated by the selected imaging unit and the feature data of the video data in the transmission stream and sends it. Next, the operation of the present invention will be described based on the above configuration.

  FIG. 5 is a flowchart illustrating an example of the processing procedure of the cameras C1 to Cn in the embodiment. Although the camera C1 will be mainly described here, the cameras C2 to Cn operate similarly.

  In FIG. 5, the camera C1 waits for notification of a message from the image processing apparatus 200 (step S1). If a message is received (Yes in step S1), the camera C1 decodes the content (step S2). The message received here includes, for example, the type of the image processing method, or information indicating a video parameter (a contrast value, a signal to noise ratio, etc.) to be prioritized. The camera C1 determines whether the feature value to be calculated, which is recognized by the decryption, needs to be changed from the feature value currently to be calculated (step S3).

  If there is no change in the feature amount to be calculated (No in step S3), the processing procedure returns to step S1, and the camera C1 waits for notification of a message from the image processing apparatus 200. If it is determined in step S3 that there is a change to the feature amount (Yes), the processing procedure proceeds to step S6.

  On the other hand, if no message is received in step S1 (No), the camera C1 calculates the feature amount currently being calculated for the video data from the imaging unit (current imaging unit) selected at that time. Calculation is performed (step S4), and an evaluation value based on the feature amount is calculated (step S14).

  Next, the camera C1 compares the calculated evaluation value with a predetermined threshold (step S5). If the evaluation value is equal to or greater than the threshold (Yes), the current evaluation value of the imaging unit is sufficiently high, so switching of the imaging unit is skipped, and the processing procedure returns to step S1. If it is determined in step S5 that the evaluation value is less than the threshold (No), the camera C1 calculates feature amounts currently being calculated for each of the video data generated by the imaging units 1a to 1m (step S6). ).

  Here, when the processing procedure reaches from step S5 to step S6, the change of the feature amount to be calculated is not requested from the image processing apparatus 200. On the other hand, when the process proceeds from step S3 to step S6, the image processing apparatus 200 is requested to change the feature value to be calculated.

  Next, the camera C1 calculates an evaluation value based on the calculated feature amount (step S7). Based on the evaluation value, the camera C1 selects the imaging unit with the highest evaluation value among the imaging units 1a to 1m (step S8). If the current imaging unit and the imaging unit selected this time are the same (No in step S9), switching of the imaging unit is skipped and the processing procedure returns to step S1.

  If the current imaging unit and the imaging unit selected this time are different, the camera C1 determines that switching of the imaging unit is necessary (Yes in step S9), and starts generation of feature data related to the image of the imaging unit of the switching destination. (Step S10). Next, the camera C1 synchronizes the frames of the video signal between the newly selected imaging unit and the currently selected imaging unit, and executes switching of the imaging units (step S11). Then, when a predetermined period including the time of frame switching has elapsed, the generation of feature data ends (step S7). The feature data generated during that time is synchronously multiplexed on the transmission stream together with the video data (step S13), and transmitted to the image processing apparatus 200.

  FIG. 6 is a diagram showing an example of a TS basic system of a transport stream. The TS basic system conforms to MPEG-2 Systems, and has a hierarchical structure of TS (Transport Stream), PAT (Program Association Table), and PMT (Program Map Table). Under the PMT, PES (Packetized Elementary Stream) packets such as video (Video), audio (Audio), and PCR (Program Clock Reference) are arranged. A PTS (Presentation Time Stamp) / DTS (Decoding Time Stamp) is inserted into the header of the video packet, and a PTS is inserted into the header of the audio packet.

  FIG. 7 is a diagram showing an example of a transport stream including synchronously multiplexed feature data. As shown in FIG. 7, the multiplexing unit 19 of the camera C1 multiplexes feature data by arranging feature data parameters at any position (under the TS, under the PAT, or under the PMT) in the TS basic system. .

  Also, the multiplexing unit 19 may multiplex feature data by arranging feature data elementarys in which PTS / DTS is added to the header under the PMT. The feature data parameters will be included in the feature data elementary. When feature data is stored in a feature data elementary, ITU-T recommendation H.3. 222. You may use the PES header option according to. That is, for example, an identifier representing a stream type (auxiliary stream (0xF9), metadata stream (0xFC), and extended stream ID (0xFD)), an identifier representing an elementary PID, or an identifier of undefined (0xFC), etc. It is good to insert the PMT header including data elementary.

  FIG. 8 is a diagram showing a specific example of parameters of feature data generated by the camera C1. In FIG. 8, the feature data parameters include items such as absolute time information, camera direction information, and parameter information such as zoom magnification information, position information, sensor information, and feature amounts. The sensor information includes, for example, temperature information, humidity information, digital tachometer information (such as an on-vehicle camera), point cloud data of a structure, and the like.

  As described above, in the embodiment, in the camera having a plurality of imaging units, it is determined on the camera side which image from the imaging unit is most suitable for the image processing of the image processing apparatus 200. That is, in the camera, the same processing as the image processing of the image processing apparatus 200 is performed on the image from each imaging unit, and the imaging unit with the highest score (evaluation value) is selected.

  Further, in the embodiment, when switching the image of a camera having a plurality of imaging units, the feature amount over a period sufficient to eliminate the discontinuity of the image processing in the image processing apparatus 200 is calculated on the camera side. The video data is synchronously multiplexed and transmitted to the image processing apparatus 200.

  In the existing remote monitoring system, when the color tone difference between the images (imaging units) is large, the characteristic data becomes discontinuous as shown in FIG. 9A each time the imaging units of the camera are switched, and the image processing apparatus 200 There was a case that the image processing was reset on the side. This is particularly true in hybrid camera systems that use different types of cameras.

  On the other hand, in the embodiment, in the camera that generates a video stream, the selection unit 15b selects an imaging unit that generates a video most suitable for the image processing of the image processing apparatus 200. When the selected imaging unit changes, the frames of the video data are synchronized between the imaging units before and after that, and the video data is switched. Then, the video data and its feature data (sensor information, parameter information, determination results, etc.) are synchronously multiplexed on the transmission frame and sent to the image processing apparatus 200.

  Since this is done, as shown in FIG. 9B, at the time of synchronous switching of a plurality of cameras, feature data can be passed from the camera to the image processing apparatus 200 via the cloud. Thereby, the feature data can be transmitted to the image processing apparatus 200 without break, and the image processing apparatus 200 can compensate for the continuity of the feature data.

  Further, the compensation unit 250d compensates for the continuity of the image sent in synchronization with the feature data based on the feature data acquired via the cloud. That is, at the time of image processing, the compensation unit 250d compensates for the continuity of the image before and after the switching of the imaging unit using the feature data. Thus, the image processing apparatus 200 can perform image processing based on the compensated video data.

  As described above, it is possible to select a camera most suitable for the image processing apparatus 200 and switch the image. In addition, since the video data and the feature data associated with the video data are multiplexed in the same transmission stream, the time series of the video and the feature data as the analysis result does not shift. Therefore, the continuity of the image processing in the image processing apparatus 200 can be maintained. From this, it is possible to achieve both the economy of sharing a plurality of camera images in a single transmission path and maintaining the processing accuracy while continuously performing image processing on the receiving side.

  That is, according to the embodiment, a smart camera system capable of maintaining continuity of image processing before and after video switching, a smart camera usable in this system, an image processing apparatus, a data transmission method, and a program are provided. It becomes possible.

(Example of application to multi-view camera system)
FIG. 10 is a diagram showing an example of a multi-viewpoint camera system. The argument according to the embodiment is also established for a multi-viewpoint camera system. In the case shown in FIG. 10, for example, the functions of the selection unit 15b and the switching control unit 15c may be implemented as a service of the cloud 100.

(Example of application to array camera system)
FIG. 11 is a diagram showing an example of a so-called array camera system including a plurality of cameras arranged in an array. For example, there is an array camera system in which the camera C1 is a visible light camera, the camera C2 is an infrared camera, and an object common to both cameras C1 and C2 is observed. In this type of system, by implementing the selection unit 15b, the switching control unit 15c, and the switch unit 10 shown in FIG. 2 in the image processing apparatus 200, the same argument as that in the above embodiment can be made. That is, when the cameras C1 and C2 are switched according to the image processing of the image processing apparatus 200, if the feature data necessary for the image processing is synchronously multiplexed and transmitted to the video data, the image processing in the image processing apparatus 200 is performed. Continuity can be compensated.

  The present invention is not limited to the above embodiment. For example, the information constituting feature data is not unique to that shown in FIG. In other words, it is not necessary to include in the feature data all of absolute time information, camera direction information, zoom magnification information, position information, sensor information, and feature amounts. Also, for example, sensor information includes temperature information, humidity information, vibration information, acceleration information, rainfall information, water level information, velocity information, digital tachometer information, and point cloud data, or device type of imaging unit, number of pixels, frame rate, It is not necessary to include all the information such as the sensitivity, the focal length of the lens, the amount of light, and the angle of view. Information to be included in feature data to be multiplexed into a transport stream may be determined according to system requirements.

  The same argument holds for a camera of a type that obtains a plurality of images with a single-eye camera by combining different wavelength cut filters with one imaging unit as well as a multispectral camera having a plurality of cameras.

  Further, in the embodiment, feature data is generated at the time of switching of the imaging unit and multiplexed in the video stream. In addition to this, feature data may be constantly calculated, and may be multiplexed into a video stream, if necessary (when the imaging unit is switched).

  In the embodiment, it has been described that the image analysis unit 15a analyzes the image of each of the imaging units 1a to 1m and generates the feature amount of the image for each of the imaging units 1a to 1m. Not only the feature quantities defined for the image, but also the feature quantities calculated for the image. Therefore, the image analysis unit 15a may be configured to calculate the feature amount of the image and execute various processes based on the feature amount of the image.

  Furthermore, the functions of the image analysis unit 15a may be individually implemented in the imaging units 1a to 1m. In this way, it is possible to output together the video data of the photographed video and the feature amount of the video from the imaging units 1a to 1m. The selection unit may obtain an evaluation value using the feature amount associated with the video data, and select one of the imaging units 1a to 1m. By shifting the analysis processing to the imaging units 1a to 1m in this manner, resources of the processing circuit 15 can be saved.

  The term "processor" used in connection with a computer is, for example, a CPU, an MPU, a GPU, or an application specific integrated circuit (ASIC), a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), etc. Can be understood as

  The processor implements a specific function based on the program by reading and executing the program stored in the memory. Instead of the memory, the program can be directly incorporated into the processor circuit. In this case, the processor realizes its function by reading and executing a program embedded in the circuit.

  While several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and the gist of the invention, and are included in the invention described in the claims and the equivalent scope thereof.

  DESCRIPTION OF SYMBOLS 1 ... Imaging part, 1a-1m ... Imaging part, 10 ... Switch part, 11 ... Lens, 12 ... Aperture mechanism, 13 ... Image sensor, 14 ... Encoding part, 15 ... Processing circuit, 15a ... Image analysis part, 15b ... Selection unit 15c Switching control unit 15d Feature data generation unit 16 Memory 16a Program 16b Feature data 17 Sensor unit 19 Multiplexing unit (MUX) 20 Synchronization processing unit 21 ... Transmission unit, 22 ... Reception unit, 23 ... Internal bus, 24 ... Data bus, 42 ... ROM, 100 ... Cloud, 101 ... Relay network, 200 ... Image processing device, 210 ... GPU, 220 ... ROM, 230 ... RAM, 240 ... HDD, 240 a ... program, 240 b ... image processing data, 240 c ... feature data, 250 ... processor, 250 a ... image processing unit, 250 b ... separation unit, 25 c Decoding unit 250d Compensating unit 250e Notification unit 260 Optical media drive 280 CD-ROM 270 Communication unit 270a Receiving unit C1 to Cn Camera P1 Cellular phone S1 Edge Server, S2: Edge server, V1: Vehicle.

Claims (23)

In a smart camera that can communicate with an image processing device,
A plurality of imaging units each generating video data;
A selection unit that selects, from the plurality of imaging units, an imaging unit that generates video data corresponding to image processing in the image processing apparatus;
A switching unit that switches and outputs the video data from the selected imaging unit in synchronization with each other's frame phase each time another imaging unit is selected by the selection unit;
A feature data generation unit that generates feature data of an image from the selected imaging unit over a predetermined period including the time point of the switching output;
A synchronization processing unit that synchronizes the switched and output video data with the feature data;
A multiplexing unit for multiplexing the synchronized video data and feature data into a transport stream;
And a transmitter configured to transmit the transmission stream to the image processing apparatus.
Smart camera. The image processing apparatus further comprises an image analysis unit that analyzes a video of each of the imaging units and generates a feature amount of the video of each of the imaging units.
The smart camera according to claim 1, wherein the selection unit selects an imaging unit that generates image data corresponding to image processing in the image processing apparatus based on a feature amount of the image for each of the imaging units. The selection unit calculates, for each of the imaging units, an evaluation value indicating a degree corresponding to image processing in the image processing apparatus based on the feature amount.
The smart camera according to claim 2, wherein an imaging unit that generates video data corresponding to image processing in the image processing apparatus is selected based on the evaluation value.   The smart camera according to claim 3, wherein the selection unit selects an imaging unit different from the selected imaging unit if the evaluation value of the selected imaging unit is less than a predetermined threshold. And a receiver configured to receive a message including information on the image processing from the image processing apparatus.
The smart camera according to claim 1, wherein the selection unit selects the imaging unit in accordance with information included in the message.   The smart camera according to any one of claims 1 to 5, wherein the plurality of imaging units are individually assigned imaging wavelength bands.   The smart camera according to claim 6, wherein the plurality of imaging units include an infrared camera and a visible light camera.   The smart camera according to any one of claims 1 to 7, wherein the feature data includes at least one of sensor information of the imaging unit and parameter information of the image.   The smart camera according to claim 8, wherein the sensor information includes at least one of a device type, a pixel number, a frame rate, a sensitivity, a focal length of a lens, a light amount, and an angle of view.   The smart camera according to claim 8, wherein the parameter information includes at least one of a color tone of the image and a luminance histogram. In an image processing apparatus capable of communicating with a smart camera having a plurality of imaging units,
A receiving unit that receives a transmission stream including video data from the smart camera;
A separation unit that separates the video data and feature data synchronized with the video data from the received transmission stream;
A decoding unit that decodes the video data and reproduces the video;
A compensation unit that compensates for the continuity of the reproduced image based on the separated feature data;
And an image processing unit that performs image processing based on the compensated image.   The image processing apparatus according to claim 11, further comprising a notification unit configured to notify the smart camera of a message including information on the image processing.   The image processing apparatus according to claim 12, wherein the message includes either information indicating that the video contrast is prioritized or information indicating that the video signal-to-noise ratio is prioritized. A smart camera, and an image processing apparatus communicably connected to the smart camera via a communication network;
The smart camera is
A plurality of imaging units each generating video data;
A selection unit that selects an imaging unit that generates video data corresponding to image processing in the image processing apparatus;
A switching unit that switches and outputs the video data from the selected imaging unit in synchronization with each other's frame phase each time another imaging unit is selected by the selection unit;
A feature data generation unit that generates feature data of an image from the selected imaging unit over a predetermined period including the time point of the switching output;
A synchronization processing unit that synchronizes the switched and output video data with the feature data;
A multiplexing unit for multiplexing the synchronized video data and feature data into a transport stream;
And a transmitter configured to transmit the transmission stream to the image processing apparatus.
The image processing apparatus is
A receiving unit that receives a transmission stream including video data from the smart camera;
A separation unit that separates the video data and feature data synchronized with the video data from the received transmission stream;
A decoding unit that decodes the video data and reproduces the video;
A compensation unit that compensates for the continuity of the reproduced image based on the separated feature data;
And an image processing unit that performs image processing based on the compensated image. A data transmission method applicable to a smart camera comprising a plurality of imaging units and processors that respectively generate video data, comprising:
The processor selecting an imaging unit for generating video data corresponding to image processing in the image processing apparatus;
Each time another imaging unit is selected, the processor switches and outputs the video data from the selected imaging unit with their frame phases synchronized with each other;
The processor generating feature data of an image from the selected imaging unit over a predetermined period including the time point of the switching output;
The processor synchronizing the switched image data with the feature data;
The processor multiplexing the synchronized video data and feature data into a transport stream;
Transmitting, by the processor, the transport stream to the image processing apparatus. Furthermore, the processor comprises a process of analyzing an image of each imaging unit to generate a feature amount of an image of each imaging unit,
The image processing unit according to claim 15, wherein in the selecting step, the processor selects an imaging unit that generates image data corresponding to image processing in the image processing apparatus based on the feature amount of the image for each of the imaging units. Data transmission method. The process of selecting is
The processor calculating, for each of the imaging units, an evaluation value indicating a degree corresponding to image processing in the image processing apparatus based on the feature amount;
17. The data transmission method according to claim 16, further comprising the step of selecting, based on the evaluation value, an imaging unit that generates video data corresponding to image processing in the image processing apparatus.   In the selection process, the processor selects an imaging unit different from the selected imaging unit if the evaluation value of the selected imaging unit is smaller than a predetermined threshold value. Data transmission method described. The processor receiving from the image processing apparatus a message including information regarding the image processing;
The method according to claim 15, further comprising the step of the processor selecting the imaging unit according to the information contained in the message.   The data transmission method according to any one of claims 15 to 19, wherein the feature data includes at least one of sensor information of the imaging unit and parameter information of the video.   The data transmission method according to claim 20, wherein the sensor information includes at least one of a device type, the number of pixels, a frame rate, a sensitivity, a focal length of a lens, a light amount, and an angle of view.   The data transmission method according to claim 20, wherein the parameter information includes at least one of a color tone of the image and a luminance histogram.   A program comprising instructions for causing a computer to perform the method according to any one of claims 15-22.