JP7114387B2 - IMAGING DEVICE, IMAGING DEVICE CONTROL METHOD AND PROGRAM - Google Patents

Description

The present invention relates to techniques for distributing captured infrared light images.

2. Description of the Related Art In recent years, among network cameras used for monitoring purposes, there are models that use infrared light imaging that can capture images sufficient for monitoring purposes even at night or under adverse conditions such as rain or snow.

An infrared camera detects infrared rays (infrared light) emitted from an object with a dedicated sensor, and generates visible images by image processing the detected data.

Here, an index color system is generally used as a method of visualizing and displaying infrared light that is actually invisible. The index color method is an image format in which each pixel of image data does not have color information but has an index value for referring to a color information table. This color information table is for converting index values into pixel values in a display color space such as RGB, and is generally called a color palette.

An image for display is generated by applying color information from the color palette, and compression processing is applied to the generated image before saving. Compression methods include Mpeg4 and H.264 for moving images. For still images such as H.264, a compression method such as Jpeg is generally used.

Japanese Unexamined Patent Application Publication No. 2002-200001 discloses a method of applying different image quality parameters to imaged video data and simultaneously displaying the data.

JP-A-2006-345309

In Patent Document 1, different image quality parameters (brightness, contrast, hue, sharpness, etc.) are applied to video data having RGB or YUV color information. However, the technique of Patent Document 1 does not consider infrared imaging, and it is difficult to adjust image quality by directly applying image quality parameters to index values that have no concept of brightness or contrast.

In order to solve the above problems, an imaging apparatus according to the present invention includes infrared light imaging means, holding means for holding a plurality of color palettes, and an infrared light image obtained by the infrared light imaging means, and output means for associating and outputting a plurality of moving image files generated by referring to each of the plurality of color palettes held by the holding means.

When playing back images captured with infrared light, the user can select images that have been developed with a color palette that suits their purpose from multiple linked recorded images, improving the visibility of the images according to the application. can be made

1 is a schematic diagram showing the appearance of a network camera; FIG. 1 is a schematic diagram showing a network configuration including network cameras; FIG. 1 is a block diagram showing a schematic configuration of a network camera; FIG. FIG. 2 is a schematic diagram showing a file structure of a recording medium in a network camera; FIG. FIG. 4 is a schematic diagram showing a color palette; FIG. 4 is a schematic diagram showing a processing flow and a data flow during image recording; FIG. 5 is a schematic diagram showing a processing flow of a moving image generation unit; 3 is a schematic diagram showing a file structure of a recording medium; FIG. FIG. 4 is a schematic diagram showing a processing flow and a data flow during image reproduction; 4 is a schematic diagram showing color palette management information; FIG. 4 is a schematic diagram showing color palette management information; FIG. FIG. 3 is a schematic diagram showing recording timings in chronological order; It is a block diagram showing an example of a system schematic configuration.

<<Embodiment 1>>
EMBODIMENT OF THE INVENTION Below, the form for implementing this invention is demonstrated, referring drawings.

FIG. 1(a) is a diagram showing the appearance of a network camera that is one embodiment of the present invention. The network camera 100 includes a lens barrel section 101 including a lens and an imaging device (infrared imaging section) for imaging infrared light. FIG. 1B is a block diagram showing an outline of the hardware configuration regarding information processing of the network camera 100. As shown in FIG. A CPU (Central Processing Unit) 102 is a central processing unit that controls the imaging apparatus 100 in an integrated manner. The IF 105 communicates with client devices via a network according to protocols such as the Internet protocol and ONVIF.

The ROM 103 has a storage element such as a flash memory, and stores programs, parameters, and color palettes (details will be described later) for the CPU 102 to cause the network camera 100 to execute processing of flowcharts described later and to realize functional blocks described later. etc. is stored in a non-volatile manner. A RAM (RANDOM ACCESS MEMORY) 104 temporarily stores a part of the computer program executed by the CPU 200 and intermediate values. The RAM 202 may also function as a buffer that temporarily stores data externally acquired via the IF 105 .

FIG. 2 is a system configuration diagram including the network camera 100. As shown in FIG. A client device 110 represents an external device in the present invention. The network camera 100 and the client device 110 are connected via a network 120 so as to be able to communicate with each other. The client device 110 transmits various commands to the network camera 100 . The network camera 100 transmits responses to those commands to the client device 110 .

The configuration and function of each part of the network camera 100 will be described with reference to FIG. The lens barrel section 200 has an infrared light lens 300 and an infrared light sensor 301 . Although the wavelength range corresponding to lenses and sensors differs depending on the materials used, here, the infrared light lens 300 and the infrared light sensor 301 are assumed to correspond to far infrared rays (wavelength 4 to 1000 μm). The infrared light lens 300 may be a detachable lens. The infrared light sensor 301 may be a CMOS or CCD capable of capturing visible light, and in that case, it may be provided with a spectroscopic element for separating visible light.

The video processing unit 201 has a function of receiving information captured by the lens barrel unit 200 and converting it into image data. The data correction unit 302 normalizes and quantizes the information (intensity of far-infrared rays) received from the infrared light sensor 301 to obtain an index value (infrared light image) for referring to the color information table (color palette). Convert to In this embodiment, the index value of one pixel is 1 byte (8 bits), and 256 kinds of colors can be expressed for each pixel. Details of the color palette will be described later.

The image generating unit 304 uses the color palette specified by the function setting unit 303 to develop (convert) the index values into color data represented by RGB (a format in which each component of RGB is represented by 8 bits). to generate a displayable image. The encoder 305 performs compression processing on the moving image or still image generated by the image generating unit 304 . The video transmission unit 306 transmits the compressed video data to the client device 110 via the network.

A still image recording unit 307 receives data converted into a format such as JPEG or RAW data from the encoder 305, generates a still image file, and then records the file in an SD card, internal memory, or the like.

In the moving image recording unit 308, the encoder 305 converts Mpeg4 or H.264. 264, HEVC, etc., generates a video file, and records it in an SD card or built-in memory.

FIG. 4A schematically shows image data (index image data) using index values generated by the data correction unit 302. FIG. In this embodiment, the PNG (Portable Network Graphics) format is used as a representative image format having an index value for referring to the color palette for each pixel. FIG. 4 shows the file format of the PNG format. Area 401 is a PNG file signature indicating that this file is in PNG format. In hexadecimal, "89 50 4E 47 0D 0A 1A 0A" is described. An area 402 is an IHDR chunk indicating an image header, and holds information indicating "use of palette" and information indicating "width of image" and "height of image". An area 403 is a PLTE chunk that stores a color palette body and holds one or more palettes (the structure of the palette will be described later with reference to FIG. 5). An area 404 holds text information, and an area 405 is an IDAT chunk that stores index values for all pixels forming an image. Region 406 indicates the end of the image.

FIG. 4B shows the data structure of a moving image file recorded by the moving image recording unit 308 in a recording medium such as an SD card or storage on a network. A plurality of folders are generated starting with Douga001, and under each folder, a moving image file corresponding to each folder and a metadata storage file (Douga001.xml) corresponding to the moving image file are stored. In the metadata file, the information of the color palette used at the time of image generation (the color palette itself or the address information for reading the color palette, etc.) is described in XML format.

FIG. 5 is a diagram showing the structure of the color palette. A color palette 501 is a color palette for displaying an image in black and white (a color palette for monochrome display), and can express 256 levels of gradation from 0 to 255. FIG. For example, the color palette 501 indicates that when the index value of a pixel in the index image is "252", the pixel value of (R, G, B) is converted to (240, 240, 240). Also, the color palette 501 indicates that when the index value of the pixel of the index image is "3", the pixel value of (R, G, B) is converted to (10, 10, 10).

By inverting the index value of the color palette, black and white can be displayed in reverse. For example, the index value “10” of the pixel of the index image is converted to (R, G, B)=(240, 240, 240), and the index value “252” is converted to (R, G, B)=(10, 10). , 10).

A color palette 502 is a color palette used for color display, and the user can set arbitrary color information to create a color palette. In this example, cool colors such as blue are assigned to areas with low temperatures (areas with small far-infrared component values), and warm colors such as red are assigned to areas with high temperatures (areas with large far-infrared component values). are assigned colors. For example, the color palette 502 indicates that when the index value of a pixel in the index image is "252", the pixel value of (R, G, B) is converted to (240, 0, 0). Also, the color palette 502 indicates that when the index value of the pixel of the index image is "3", the pixel value of (R, G, B) is converted to (0, 0, 240). Of course, a plurality of types of color palettes for color display may be held.

Next, the processing flow and data flow during image recording by the video processing unit 201 will be described with reference to FIG.

The data correction unit 601 generates image data by converting the two-dimensional sensor detection values acquired by the infrared light sensor 301 into index values, and transmits the image data to the image generation unit 602 as index image data (611 ). The function setting unit 603 reads out a plurality of color palette data held in the camera from the ROM 103 (RAM 104 if expanded in the RAM 104) and sends them to the image generation unit 602 (612). The image generation unit 602 generates a plurality of image data (display image data) by converting the image index value into color information by referring to each of the color palettes received from the function setting unit 603 , and transmits the image data to the encoder 604 . (613). Details of the processing of the image generation unit 602 will be described later.

The encoder 604 compresses the received multiple image data and converts them into Mpeg4, H.264, and Mpeg4. N different types of compressed moving image data, such as H.264 or HEVC, are generated and output to the moving image recording unit 605 (614). The moving image recording unit 605 generates individual files for a plurality of (N) pieces of compressed image data and stores them in the recording medium 606 or storage (not shown) on the network (615).

Next, details of the processing of the image generation unit 602 will be described with reference to FIG. The image generation unit 602 acquires index image data from the data correction unit 601 (S701). Next, the number of color palettes held by the camera (the number of color palettes) is obtained from the function setting unit 603 (S702). Next, the number of remaining unreferenced color palettes is confirmed (S704), and if there are remaining color palettes, one of the remaining color palettes is acquired (S705). Next, the index image data is converted into color image data using the color palette values (S706). Then, the converted color image data is transmitted to the encoder unit 604 (S707). 1 is subtracted from the number of color palettes (S708). If the remaining color palette is 0 in step S704, the process ends.

FIG. 8 is a schematic diagram showing how the moving image recording unit 605 stores a plurality of moving image files with different color palettes. A folder is created for each color palette name (palette name) in the Douga001 folder so that the moving image data generated from the same source can be identified. In the folder with this palette name, a moving image file obtained by compressing image data developed with this palette name and a file containing color palette data used for development are saved. This is to allow later retrieval of the color palette data used during generation. In the example of FIG. 8, a folder with a palette name "Douga001_Palette01" contains color palette data "Douga001_Palette01.plt" corresponding to this palette name. Stores moving image data "Douga001_Palette01.mp4" and metadata "Douga001_Palette01.xml" for color palette and encoding. Metadata includes frame rate and resolution. By doing so, the moving image recording unit 605 can output a plurality of developed moving image files in association with each other by referring to each of the plurality of color palettes.

Next, a method of reproducing moving image data stored in a recording medium and distributing it to a client device will be described.

A moving image recording unit 902 manages images recorded on a recording medium 904 . When a video to be played back is specified from the client device, the information of the built-in color palette is presented to the client device from the function setting unit 901, the client device executes the operation of selecting the color palette, and the result is recorded as a moving image. 902 is notified. The moving image recording unit 902 identifies a moving image file based on the selected image and color palette, and reads compressed moving image data from the recording medium 904 (912).

Next, the moving image recording unit 902 transmits the compressed moving image data to the video transmission unit 903 for distribution to the client device (913). A video transmission unit 903 uses a video transmission protocol such as RTP to distribute moving image data to a client device.

As described above, according to the present embodiment, video files corresponding to the number of color palettes are stored in a recording medium from video captured by an infrared camera. can be specified and played back, the visibility of the video can be improved.

<<Embodiment 2>>
In the first embodiment, moving image files for all color palettes built in the infrared camera are generated, but the number of moving image files that can be simultaneously generated may be limited depending on the writing performance of the encoder or recording medium.

In this embodiment, a moving image file generating method when the number of simultaneously generated files is limited will be described. Components and steps having the same functions as those of the first embodiment are denoted by the same reference numerals, and descriptions of components that are structurally and functionally the same are omitted.

FIG. 10 shows an example of color palette management information held in the ROM 103 of the camera. A priority (priority order) is set for each color palette, and a moving image file to be generated is determined according to the priority order. In FIG. 10, a color palette with a smaller priority number indicates a higher priority. If the number of simultaneously generated files is less than the total number of color palettes, no moving image file using a color palette with a lower priority is generated.

The number of simultaneously generated files is determined by the performance of encoding processing executed by hardware or software and the writing speed to recording media such as SD cards. Here, the resolution and frame rate of the video to be encoded are fixed.

Here, if the number of simultaneously generated files is N, a moving image file using a color palette with a priority value of 1 to N shown in FIG. 10 is generated. Instead of the number of simultaneously generated files, a predetermined number of moving image files may be output from the color palette having the highest priority.

As described above, according to this embodiment, even if the number of simultaneously generated files is less than the number of built-in color palettes, the video files using the color palette desired by the user are preferentially saved. becomes possible.

<<Embodiment 3>>
In the second embodiment, there are video files that cannot be generated depending on the order of priority. How to generate a video file for 10 minutes will be explained. Components and steps having the same functions as those of Embodiments 1 and 2 are denoted by the same reference numerals, and descriptions of components that are structurally and functionally the same are omitted.

FIG. 11 is a table in which resolutions and frame rates for encoding are added to the color palette management information of FIG. When encoding using a color palette with priority 1, the video data is generated at the resolution and frame rate desired by the user, and when encoding using a color palette other than priority 1, all Reduce the resolution and frame rate until you can generate a video file using the color palette. The resolution and frame rate in this case can be determined by the encoding processing performance and the writing speed to the recording medium or the like, as in the second embodiment.

As described above, according to this embodiment, it is possible to reduce the load of encoding and the amount of file writing data, so that it is possible to generate moving image files for all color palettes.

<<Embodiment 4>>
Since network cameras for surveillance have an upper limit to the amount of image data that can be recorded in the camera, it is desirable to be able to record image data as efficiently as possible.

In this embodiment, in the video recording method of Embodiment 1, a method of efficiently saving image data by using the moving body detection function of the camera and reproducing it in synchronization with separately saved moving image data will be described. do. Components and steps having the same functions as those of Embodiments 1 to 3 are denoted by the same reference numerals, and descriptions of components that are structurally and functionally the same are omitted.

When saving multiple video files in Embodiment 1, there is a possibility that there is no change in the saved images. By saving a plurality of moving image files, the amount of image data can be reduced. The moving body detection function of the CPU 102 acquires the difference between the preceding and succeeding frame images (infrared light images) and the difference between the previously acquired background image (infrared light image) and the current frame image (infrared light image), If the difference area is equal to or larger than a predetermined size, it is detected as a moving object.

FIG. 12 is a diagram showing in chronological order timings for saving moving image data in this embodiment. A moving image using a color palette with priority 1 in the second embodiment is always recorded, and a moving object detection function detects the occurrence of a recording start event for moving images using color palettes other than priority 1. Start recording as a trigger. The save operation ends when the moving object is no longer detected.

The methods of Embodiments 2 and 3 may be used together as a moving image file recording method based on the priority of the color palette.

As described above, according to the present embodiment, it is possible to reduce the amount of data to be stored in the recording medium, so it is possible to reduce missed shots due to insufficient capacity.

<<other embodiments>>
In a system including a network camera, it is common to reproduce video data using a viewer application (hereinafter referred to as a viewer) that operates on a client device of a delivery destination.
In the present embodiment, a method of realizing reproduction of index image data and switching of color palettes in a client device will be described.

Referring to FIG. 13, the configurations and functions of the imaging device 1301 and the client device 1302 according to this embodiment will be described. Components and steps having the same functions as those of Embodiments 1 to 3 are denoted by the same reference numerals, and descriptions of components that are structurally and functionally the same are omitted.

As in the first embodiment, the data correction unit 1311 converts the information received from the sensor into an index value for referring to the color palette. A data transmission unit 1315 receives image data converted into index values from the data correction unit 1311 and stores the image data in a format such as a PNG file. Table information of a plurality of color palettes that can be set by the function setting unit 1316 is also stored.

The data receiving unit 1324 receives color palette table information from the data transmitting unit 1315 in advance and stores it inside the client device 1302 . A function setting unit 1325 of the client device 1302 allows the user to select a color palette to be used for image display.

Moving image data distributed from the imaging device 1301 to the client device 1302 is processed in the order of the image generation unit 1312, the encoder 1313, and the video transmission unit 1314 of the imaging device 1301, and transmitted to the client device 1302 via the network. The decoder 1321 decompresses the received compressed video data into reproducible video data, and transmits the video data to the video display unit 1322 . A video display unit 1322 displays a video on the viewer screen.

Next, a method for reproducing index image data in the client device 1302 will be described.

A data receiving unit 1324 of the client device 1302 receives index image data stored in a format such as a PNG file from the data transmitting unit 1315 of the imaging device 1301 . Data transmission protocols such as HTTP and FTP are generally used as a data transfer method.

The image generation unit 1323 of the client device 1302 generates image data converted into color information in the color palette using the image index value, and transmits the image data to the video display unit 1322 . A video display unit 1322 displays image data on the viewer screen.

As described above, according to the present embodiment, when reproducing an image captured by an infrared camera, the user can specify an arbitrary color palette and reproduce the image on the client device side. improve sexuality. It is also possible to apply the methods shown in Embodiments 2 and 3 to the configuration shown in this embodiment.

In the above-described embodiment, the image generation unit 602 converts index image data into RGB image data with a bit depth of 8 bits. You may convert into image data of other bit depths, such as.

In the above-described embodiments, a program that implements one or more functions is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. It is feasible. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100 network camera 110 client device 120 network

Claims (19)

an infrared light imaging means;
holding means for holding a plurality of color palettes;
output means for associating and outputting a plurality of moving image files generated by referring to each of the plurality of color palettes held by the holding means for the infrared light image obtained by the infrared light imaging means; An imaging device characterized by comprising: 2. The imaging apparatus according to claim 1, further comprising recording means for recording the moving image output by said output means on a recording medium. 2. The holding means further holds a priority order for each color palette, and the output means outputs up to a predetermined number of moving image files in descending order of priority held by the holding means. 3. The imaging device according to 2. The holding means holds the color palette and the parameters used for encoding in association with each other,
4. The imaging apparatus according to any one of claims 1 to 3, wherein said output means outputs a moving image file encoded using said parameters. Further provided is detection means for detecting a moving object from the infrared light image captured by the infrared light imaging means, and the output means refers to the first color palette even when the detection means does not detect the moving object. and outputting a plurality of moving image files by referring to the first color palette and the second color palette when a moving object is detected by the detecting means. 5. The imaging device according to any one of items 1 to 4. 6. The output means according to any one of claims 1 to 5, wherein said output means comprises conversion means for converting the infrared light image obtained by said infrared light imaging means into image data for display using a color palette. 10. The image pickup device according to claim 1. 7. The imaging apparatus according to claim 1, wherein said plurality of color palettes include a color palette for monochrome display. 8. The imaging apparatus according to claim 1, wherein said plurality of color palettes include a color palette for color display. 9. The imaging device according to any one of claims 1 to 8, wherein the infrared light image is an image in PNG format. A control method for an imaging device comprising infrared light imaging means and holding means for holding a plurality of color palettes,
an output step of correlating and outputting a plurality of moving image files generated by referring to each of the plurality of color palettes held by the holding means for the infrared light image obtained by the infrared light imaging means; A control method for an imaging device, comprising: 11. The control method according to claim 10, further comprising a recording step of recording the moving image output in said output step onto a recording medium. 11. The holding means further holds a priority order for each color palette, and in the output step, up to a predetermined number of moving image files are output in descending order of priority held by the holding means. 11. The control method according to 11. The holding means holds the color palette and the parameters used for encoding in association with each other,
13. The control method according to any one of claims 10 to 12, wherein said output step outputs a moving image file encoded using said parameters. Further comprising a detecting step of detecting a moving object from the infrared light image captured by the infrared light imaging means, wherein the outputting step refers to the first color palette even when the moving object is not detected in the detecting step. and outputting a moving image file, and outputting a plurality of moving image files by referring to the first color palette and the second color palette when a moving object is detected in the detecting step. 14. The control method according to any one of 10 to 13. 15. The outputting step comprises a conversion step of converting the infrared light image obtained by the infrared light imaging means into image data for display using a color palette. The control method described in the item. 16. The control method according to any one of claims 10 to 15, wherein said plurality of color palettes includes a color palette for monochrome display. 17. The control method according to any one of claims 10 to 16, wherein said plurality of color palettes includes a color palette for color display. 18. The control method according to any one of claims 10 to 17, wherein said infrared light image is a PNG format image. a computer having infrared light imaging means and holding means for holding a plurality of color palettes,
As an output means for associating and outputting a plurality of moving image files generated by referring to each of the plurality of color palettes held by the holding means for the infrared light image obtained by the infrared light imaging means A program characterized by functioning.

Images