JP2011199354A - Image processing unit, image reproducing unit, and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing unit that efficiently encodes an image signal captured by a plurality of cameras, and to provide an image reproducing unit for decoding the coded image signal to be displayed on a monitor, or the like.SOLUTION: The image processing unit includes: imaging means 1 to N of generating a plurality of image signals from a plurality of captured moving pictures; feature amount calculation means 21 to 2N of dividing the plurality of image signals into a plurality of local areas, and of calculating an inter-frame difference value for each local area; an image composite means 32 of combining images of the local area based on the calculated inter-frame difference value, and of generating a plurality of composite images; a frame rate control means 33 of controlling a frame rate of the composite image, based on the inter-frame difference value of the local area constituting the composite image; and an image coding means 34 of performing coding using inter-frame prediction to the composite image to generate a bit stream.

Description

  The present invention relates to an image processing apparatus that encodes a plurality of image signals, and an image reproduction apparatus that decodes the encoded image signals and displays them on a monitor or the like.

  In recent years, monitoring using an image recording display device that shoots different monitoring areas from a plurality of cameras, divides and displays the obtained camera images, and sequentially compresses image data and records it on a recording medium. Camera systems (digital CCTV (Closed Circuit TeleVision) systems) are beginning to become widespread. In this system, a system configuration is used in which images captured by a plurality of cameras are input to an encoding device and image data is compressed, and an encoding technique for compressing image data is, for example, an international standard. MPEG2 (Moving Picture Experts Group Part2) and recently H.264. H.264 / AVC or the like is used. These are hybrid coding schemes that combine motion compensation prediction coding, discrete cosine transform coding, and variable length coding.

  In such a surveillance camera system, in order to avoid an increase in circuit scale, it is assumed that images input from a plurality of cameras are sequentially switched and compressed by one encoder. Accordingly, since different images input from a plurality of cameras are periodically switched, it is necessary to perform prediction between frames of images that are not temporally continuous. Therefore, as an encoding technique for compressing image data, MPEG2 or H.264 is used. When a method using inter-frame prediction such as H.264 / AVC is selected, there is a problem that the compression efficiency is poor.

  In a conventional surveillance camera system, in order to solve such a problem, a plurality of image signals are combined to generate one combined image signal, and a combined image signal having a correlation between frames is combined with one encoder. Encoding was performed by using and compressing. (For example, see Patent Document 1)

JP 2002-27393 A (pages 4-7, FIG. 1)

  In a conventional surveillance camera system, image signals from the cameras are simply arranged and combined, or the resolution of each image signal is reduced and combined to generate a combined image signal. Therefore, when the motion of images from some cameras among the image signals from a plurality of cameras is large, if inter-frame prediction is performed according to the images from cameras with small motion, the images with large motion There is a problem that the information amount is not sufficiently allocated and the image quality is deteriorated. On the other hand, when inter-frame prediction is performed according to an image from a camera with a large movement, there is a problem that the amount of information generated in the encoded bit stream increases and the recording time on the recording medium is shortened. Further, when the resolutions are reduced and combined, there is a problem that the image quality at the time of reproduction deteriorates.

  An image processing apparatus according to the present invention divides a plurality of image capturing means for capturing a moving image and a plurality of image signals generated from the moving images captured by the plurality of image capturing means into a plurality of local areas, respectively. Feature amount calculating means for calculating inter-frame difference values for each area; image combining means for generating a plurality of composite images by combining local area images based on the calculated inter-frame difference values; and locals constituting the composite images Frame rate control means for controlling the frame rate of the composite image based on the inter-frame difference value of the area, and image encoding means for performing coding using inter-frame prediction on the composite image and generating a bitstream It is equipped with.

  The image processing apparatus according to the present invention divides a plurality of image capturing means for capturing moving images and a plurality of image signals generated from the moving images captured by the plurality of image capturing means into a plurality of local areas, respectively. A feature amount calculating unit that calculates an inter-frame difference value for each local area; an image combining unit that combines a local area image based on the calculated inter-frame difference value to generate a plurality of combined images; and a combined image Filter means for removing high-frequency components of the composite image based on the inter-frame difference value of the local area, and image encoding that performs inter-frame prediction on a plurality of composite images and generates a bitstream Means.

  In addition, the image reproducing apparatus according to the present invention includes an image extracting unit that extracts a local area image constituting an image signal from a plurality of composite images generated by combining local area images constituting a plurality of image signals; An image re-synthesizing unit that synthesizes the extracted local area images to generate an image signal; and an image display unit that displays the generated image signal on a monitor. The synthesized image is a frame of a plurality of local area images. It is a synthesized image synthesized by combining local area images based on the inter-difference value.

  The image processing apparatus according to the present invention divides a plurality of image capturing means for capturing moving images and a plurality of image signals generated from the moving images captured by the plurality of image capturing means into a plurality of local areas, respectively. Correlation calculating means for calculating the correlation of a plurality of image signals for each local area, coding control means for determining the order and GOP structure for encoding the image signals based on the calculated correlation, and encoding And an image encoding means for encoding an image signal using inter-frame prediction based on the order to perform and the GOP structure and outputting a bit stream.

  An image processing system according to the present invention is an image processing system including an image processing device that encodes an image signal, and an image reproduction device that decodes and reproduces the encoded image signal on a monitor. The apparatus divides a plurality of image capturing means for capturing a moving image and a plurality of image signals generated from the moving images captured by the plurality of image capturing means into a plurality of local areas, and performs inter-frame differences for each local area. A feature amount calculating unit that calculates a value, an image combining unit that combines a local area image based on the calculated inter-frame difference value, and generates a plurality of combined images, and an inter-frame difference value of the local area that forms the combined image Based on the frame rate control means for controlling the frame rate of the composite image and inter-frame prediction for a plurality of composite images An image encoding unit that performs encoding and generates a bitstream; and an image storage unit that stores a combination of the local area constituting the bitstream and the composite image. And a data acquisition means for acquiring a combination of local areas, an image decoding means for decoding the bitstream acquired by the data acquisition means to generate a composite image, and an image based on the combination of local areas from the decoded composite image Image extracting means for extracting an image of a local area constituting a signal, image recombining means for generating an image signal by combining the extracted images of the local area, and image display means for displaying the generated image signal on a monitor And.

  An image processing system according to the present invention is an image processing system including an image processing device that encodes an image signal, and an image reproduction device that decodes and reproduces the encoded image signal on a monitor. The apparatus divides a plurality of image capturing means for capturing a moving image and a plurality of image signals generated from the moving images captured by the plurality of image capturing means into a plurality of local areas, and performs inter-frame differences for each local area. A feature amount calculating unit that calculates a value, an image combining unit that combines a local area image based on the calculated inter-frame difference value, and generates a plurality of combined images, and an inter-frame difference value of the local area that forms the combined image Based on the filter means for removing high-frequency components of the composite image, and performing encoding using inter-frame prediction on a plurality of composite images An image encoding means for generating a bit stream, and an image storage means for storing a combination of local areas constituting the bit stream and the composite image, and the image reproducing apparatus includes a combination of the bit stream and the local area from the image storage means. A data acquisition means for acquiring the image data, an image decoding means for decoding the bitstream acquired by the data acquisition means and generating a composite image, and a local signal that constitutes an image signal from the decoded composite image based on a combination of local areas Image extraction means for extracting an image of the area, image recomposition means for generating an image signal by combining the extracted images of the local area, and image display means for displaying the generated image signal on a monitor Yes.

  According to the present invention, image signals photographed by a plurality of cameras can be efficiently encoded.

It is a block diagram showing the whole structure of the image processing apparatus and image reproduction apparatus which are shown in Embodiment 1 of this invention. It is a flowchart of the difference value calculation process between frames shown in Embodiment 1 of this invention. It is a conceptual diagram which shows the example of calculation of the difference value between frames shown in Embodiment 1 of this invention. It is a flowchart of the image composition process shown in Embodiment 1 of this invention. It is a conceptual diagram which shows the example of calculation of the combination of the local area shown in Embodiment 1 of this invention. It is a conceptual diagram which shows the example of a production | generation of the synthesized image shown in Embodiment 1 of this invention. It is a conceptual diagram which shows the example of calculation of the frame rate of the synthesized image shown in Embodiment 1 of this invention. It is a conceptual diagram which shows the GOP structure in the encoding process shown in Embodiment 1 of this invention. It is a block diagram showing the whole structure of the image processing apparatus and image reproduction apparatus which are shown in Embodiment 3 of this invention. It is a block diagram showing the whole structure of the image processing apparatus and image reproduction apparatus which are shown in Embodiment 4 of this invention. It is a conceptual diagram which shows the example of calculation of the correlation between the several images shown in Embodiment 4 of this invention. It is a conceptual diagram which shows the encoding order of the synthesized image shown in Embodiment 4 of this invention.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing the overall configuration of a surveillance camera system using an image processing apparatus and an image reproduction apparatus according to Embodiment 1 of the present invention. First, the configuration of the image processing apparatus will be described. In FIG. 1, cameras 1 to N are, for example, an image sensor that captures a moving image, such as a CCD image sensor, and image signals 101 to 10N and image signals 111 to 1 before one frame by converting the captured images into digital signals. It is comprised from the A / D converter which produces | generates 11N. The frame memory units 11 to 1N store the output image signals 101 to 10N and the image signals 111 to 11N one frame before in units of frames, and read control signals 131 to 13N received from the stored image signals. The image reading unit is configured to read an image signal based on the above. The local area feature amount calculation units 21 to 2N include a local area dividing unit that divides the image signals 101 to 10N and the image signals 111 to 11N of the previous frame into regions of a predetermined size (hereinafter referred to as local areas), Inter-frame difference values 141 to 14N between the image signals 101 to 10N and the image signals 111 to 11N of the previous frame are calculated for each local area, and time information 151 to 15N when the image signals 101 to 10N are photographed are calculated. It consists of an inter-frame difference value calculation unit for output. The local area image composition control unit 31a refers to the inter-frame difference values 141 to 14N and the time information 151 to 15N to control a combination for compositing images from a plurality of cameras in units of local areas, and generates the image composition information 161 and the composite image. A suitable frame rate 162 is output. The local area image composition processing unit 32a outputs read control signals 131 to 13N for the frame memory units 11 to 1N based on the image composition information 161 and the frame rate 162, and the image signals 121 to 12N read from the frame memory units 11 to 1N. Are combined to generate a composite image signal 164. The encoding control unit 33 determines a GOP (Group Of Pictures) structure 163 when performing image encoding based on the image synthesis information 161 and the frame rate 162. The image encoder 34 encodes the composite image signal 164 by an encoding unit using inter-frame prediction according to the GOP structure 163, and outputs a variable-length bit stream 165. The recording medium 35 is an HDD (Hard Disk Drive) device that records image composition information 161, a frame rate 162, and a bit stream 165.

  Next, the configuration of the image reproduction device will be described. In FIG. 1, a user interface unit 36 includes an input unit for inputting a camera and time that the user wants to reproduce, and outputs time information 166 and camera information 167 of a specified moving image. The stream read control unit 37 outputs a read control signal 168 for reading the necessary bit stream 169 and the corresponding image composition information 170 from the recording medium 35 based on the time information 166. The image decoder 38 receives the bit stream 169 and outputs a composite image signal 171 decoded by the image decoding means. The local area image recombination control unit 39 refers to the image composite information 170 corresponding to the decoded composite image signal 171 and the camera information 167 to be reproduced, and selects the necessary composite image signal 171 and the original control signal 172. The image recombination information 174 for recombining the camera images is output, and the frame memory unit 40 includes a memory for storing the composite image signal 171. The local area image recombination processing unit 41 generates a decoded image signal 175 from the original camera by combining images in units of local areas based on the image recombination information 174 from the composite image signal 173 read from the frame memory unit 40. To do. The monitor 42 is a monitor device that displays a decoded image from the original camera.

  Next, the operation will be described. First, the cameras 1 to N capture a subject such as an object to be monitored, convert the captured moving image into a digital signal by an A / D converter, and output image signals 101 to 10N. At this time, each camera outputs the image signals 101 to 10N at an arbitrary frame rate in accordance with the importance and state of the shooting target.

  The output image signals 101 to 10N are written in the memory in units of frames in the frame memory units 11 to 1N and are input to the local area feature amount calculation units 21 to 2N. FIG. 2 shows a flowchart of the inter-frame difference value calculation processing of the local area feature amount calculation units 21 to 2N. The local area feature amount calculation units 21 to 2N read the image signals 101 to 10N (step S1-1), and the image signals 111 that have been captured and converted into digital signals by the cameras 1 to N one frame before and written to the memory are completed. To 11N are read from the frame memory units 11 to 1N (step S1-2), and the image signals 101 to 10N and the image signals 111 to 11N of the previous frame are each divided into a plurality of local areas (step S1-3). FIG. 3 shows an image signal in the case of dividing into 5 parts in the horizontal direction and 4 parts in the vertical direction. These image signals are compared for each camera, and inter-frame difference values 141 to 14N are calculated in units of local areas (step S1-4). Here, the inter-frame difference value is a value obtained by accumulating corresponding pixel difference values in a predetermined area (here, a local area) in image signals taken by the same camera and having different times. Note that the inter-frame difference value may be the sum of squares of the difference values of the corresponding pixels, or may be the difference value of the average value or the deviation value of the pixels in the predetermined area.

  FIG. 3 shows an example of calculating the inter-frame difference value in each local area when one frame is divided into five in the horizontal direction and four in the vertical direction. A pixel value at a position corresponding to each local area is compared between the image of the previous frame and the current frame, and the difference value is accumulated for each local area. In the surveillance camera system, when a fixed camera is used, the inter-frame difference value is large only in the local area including the moving body, and the inter-frame difference value is small or close to 0 in the background area where there is no moving body. It becomes. Further, time information 151 to 15N when the image signals 101 to 10N for which the inter-frame difference values are calculated are also output at the same time (step S1-5).

  Based on the inter-frame difference values 141 to 14N, the local area image composition control unit 31a generates image composition information 161 for creating a composite image such as a combination of image signals 121 to 12N from all cameras in units of local areas. Further, a frame rate 162 suitable for encoding the composite image is determined, and these are output. The local area image composition processing unit 32a synthesizes the image signals 101 to 10N based on the image composition information 161. These image composition processes will be described with reference to the flowchart shown in FIG.

  First, a method in which the local area image composition control unit 31a determines the image composition information 161 will be described. FIG. 5 shows a method of determining the image composition information 161 when creating a composite image from images taken from three cameras with N = 3. In FIG. 5, the camera 1 is an image of the elevator hall, the camera 2 is an image of the building taken from the outside, and the camera 3 is an image of the entrance / exit gate to the building. An inter-frame difference value is calculated in units of local areas for two frames in the time direction of images from each camera, and the inter-frame difference value is input to the local area image composition control unit 31a (step S2-1).

  In the local area image composition control unit 31a, images corresponding to the number of local areas in one frame are selected from the local areas of all the camera images in descending order of inter-frame difference values (in this example, 5 × 4 = 20), and the rest These local area unit images are also combined in descending order of the inter-frame difference value, and are classified into a total of three (step S2-2). By synthesizing these images, a synthesized image for three frames can be generated. In the example of FIG. 5, 20 local areas having large inter-frame difference values are represented by squares with a white background, and numbers 1 to 20 are assigned. Next, 20 local areas with the largest inter-frame difference values are represented by a square with a background, and numbers 1 to 20 are assigned. The remaining 20 local areas are represented by hatched squares and numbered from 1 to 20. At this time, it is not necessary to assign a number of 1 to 20 according to the size of the difference value in each classification, and by assigning a number so that images in local area units within the same frame are as continuous as possible. The adjacent local areas in the camera image can be arranged close to each other in the composite image. Thereby, even when the moving body moves between adjacent local areas, it is possible to efficiently perform the encoding in the encoding process using inter-frame prediction described later. As described above, combinations for generating three composite images are determined from the images of local area units classified into three.

  FIG. 6 shows a synthesized image when the camera image in FIG. 5 is synthesized based on the determined combination. Among the composite images on the right side of FIG. 6, composite image 1 is an image in units of local areas having a large inter-frame difference value (in FIG. 5, a square with a white background) in the order of left to right and top to bottom in the order of 1-20. It is a combined image. The composite image 2 is an image obtained by combining local area unit images (in FIG. 5, squares with dots in the background) in the order of 1 to 20 with an inter-frame difference value. The composite image 3 is an image obtained by combining local area unit images with small difference values between frames (in FIG. 5, squares with diagonal lines in the background) in the order of 1 to 20. As described above, the local area image composition control unit 31a determines the image composition information 161 relating to the combination of local areas at certain predetermined time intervals (for example, at intervals of one second) and outputs them to the local area image composition processing unit 32a. (Step S2-3). The image composition information 161 includes, in addition to the combination of local areas constituting the composite image, information on image signals constituting the composite image, for example, the level of the difference value between frames in units of local areas constituting each composite image ( Size), camera number and position information, and camera time information.

  Next, a method in which the local area image composition control unit 31a determines the frame rate 162 of each composite image will be described. In the local area image composition control unit 31a, the frame rate of the composite image is determined based on the size of the inter-frame difference value of the local area constituting each composite image (step S2-4). That is, a high frame rate is set for a composite image composed of a local area having a large inter-frame difference value, and a low frame rate is set for a composite image composed of a local area having a small inter-frame difference value. FIG. 7 shows an example in which all the image signals from the cameras 1 to 3 shown in FIG. 6 are input at 12 frames / second. Since the composite image 1 shown in FIG. 6 is a combination of local area unit images having a large inter-frame difference value, all frames having a frame rate (here, 12 frames / second) input from the camera are set. Since the composite image 2 is a combination of local area unit images with intermediate frame difference values, the frame rate is set to be reduced (here, 6 frames / second). Since the composite image 3 is a combination of local area unit images with small inter-frame difference values, a frame rate (here, 3 frames / second) obtained by further thinning out frames is set when the background image has little motion. . As a result, a local area with a large movement including a moving body can be made a smooth image, and an allocation of information amount can be reduced for a local area with a small movement. The local area image composition control unit 31a outputs the set frame rate to the local area image composition processing unit 32a, the encoding control unit 33, and the recording medium 35, respectively.

  In the local area image composition processing unit 32a, based on the image composition information 161 and the frame rate 162 instructed from the local area image composition control unit 31a, a read control signal that instructs the frame memory units 11 to 1N to read out image signals. 131 to 13N are output (step S2-5). After the image signals 101 to 10N from the respective cameras are temporarily stored in the memories in the frame memory units 11 to 1N, the desired image signals are delayed by the frame memory units 11 to 1N that have received the read control signals 131 to 13N. (Step S2-6). The local area image composition processing unit 32a generates a composite image signal 164 from the image signals 121 to 12N based on the image composition information 161 and the frame rate 162 as shown in FIGS. 6 and 7 (step S2-7). Each composite image is output in turn at predetermined time intervals (step S2-8). When the predetermined time is 1 second, frames 1 to 12 of composite image 1 are output, then frames 1 to 6 of composite image 2 are output, and finally frames 1 to 3 of composite image 3 are output in order. .

  The encoding control unit 33 determines a GOP structure 163 for encoding the composite image signal 164. The encoding control unit 33 receives the image composition information 161 and the frame rate 162, and determines the GOP structure based on these information. A synthesized image with a high frame rate has a short frame interval, so that it can be expected that coding efficiency by inter-frame prediction is high, and a GOP structure is determined so as to perform coding using a P picture or a B picture. On the other hand, since a frame interval is long in a composite image with a low frame rate, if it is a composite image composed of local area images including moving objects, encoding is performed when inter-frame prediction is performed. Efficiency may be reduced. Therefore, based on the composite image information 161 and the frame rate, the GOP structure is determined so that the capacity of the bit stream generated by encoding is minimized. At this time, encoding is performed a plurality of times by changing the M value (interval until the next I or P picture is generated from the I or P picture) and the N value (interval until the next I picture is generated from the I picture). A GOP structure that performs processing and encodes a GOP structure having the smallest bitstream capacity may be used.

  FIG. 8 shows an example of picture types with respect to the frame rates of the composite images 1 to 3 shown in FIG. The composite image 1 with a high frame rate is encoded with a GOP structure using a B picture with M value = 3 and N value = 12. In the composite image 2 in which the images of local area units having intermediate frame difference values are combined, the frame interval is doubled by thinning out the frames from the composite image 1, and therefore, for example, M value = 2, N value = Encoding is performed with 6 GOP structures. The composite image 3 in which the local area unit images having a small inter-frame difference value are combined is further thinned out from the composite image 2, but the local area image constituting the composite image 3 is a background image. Use predictive coding. For example, encoding is performed with a GOP structure of M value = 1 and N value = 3. In addition, when the local area image which comprises the synthetic | combination image 3 contains a moving body and cannot be encoded efficiently using inter-frame prediction, all are comprised by I picture. In addition, since the predetermined time is set to 1 second here, the GOP structure is determined so that the I picture is encoded once per second in each synthesized image.

  The image encoder 34 performs rearrangement of the input composite image signal 164 and encoding using inter-frame prediction according to the GOP structure 163 determined by the encoding control unit 33, and outputs a variable-length bit stream 165 And recorded on the recording medium 35. In addition to the bit stream 165, the corresponding image composition information 161 and the frame rate 162 are recorded in the recording medium 35. The above is the operation of the image processing apparatus.

  Next, the operation of the image playback device will be described. The user interface unit 36 outputs time information 166 and camera information 167 of an image to be reproduced among image signals photographed and recorded by the cameras 1 to N in accordance with an instruction from the user. The stream read control unit 37 outputs a read control signal 168 based on the time information 166, and the recording medium 35 uses bits necessary for generating an image signal to be reproduced from recorded data based on the read control signal 168. The bit stream 169 and the image composition information 170 for reading the stream 169 and the image composition information 170 are read.

  The image decoder 38 receives the bit stream 169 from the recording medium 35 based on the read control signal 168 and performs image decoding. Further, the generated composite image signal 171 is output and written in the frame memory unit 40. Here, based on the camera information 167 and the image composition information 170, the local area image recomposition control unit 39 reads a read control signal for reading a composite image signal 173 including an image signal captured and recorded by the camera to be reproduced. 172 is output to the frame memory unit 40. Further, image recombination information 174 for recombining the original image signal is output. Note that the image recombination information 174 is information including the camera number and position information in units of local areas constituting the read composite image signal 173 and the time information of the camera.

  The local area image recombination processing unit 41 extracts images in units of local areas from the composite image signal 173 based on the image recombination information 174, and combines the extracted images to obtain a desired camera among the original cameras 1 to N. A decoded image 175 photographed and recorded from is generated and output to the monitor 42 for reproduction display.

  As described above, in the image processing apparatus and the image reproduction apparatus according to Embodiment 1, a composite image is created by combining images from a plurality of cameras in order of increasing inter-frame difference values in units of local areas. Since the frame rate is determined according to the size of the inter-frame difference value of the local area to be used, a high frame rate is set for a portion with a large movement, so that the image can be recorded and reproduced without degrading the image. it can. On the other hand, in a portion with a small amount of motion such as the background, it is possible to store the information amount with less allocation by lowering the frame rate. In addition, by optimizing the M and N values of the GOP structure according to the frame rate and inter-frame difference values that make up each composite image, an image in which a part of the screen moves, such as a surveillance camera system, is encoded In this case, efficient encoding can be performed, and the recording time can be extended.

  Also, the case where the number of image signals captured by the camera is the same as the number of composite image signals has been described, but the same effect can be obtained even when the number of image signals captured by the camera differs from the number of composite image signals. .

Embodiment 2. FIG.
In the first embodiment, the local area feature value calculation units 21 to 2N calculate inter-frame difference values 141 to 14N between the image signals 101 to 10N from each camera and the image signals 111 to 11N of the previous frame. In the second embodiment, a combination for generating a composite image is determined based on a value obtained by accumulating a difference value between frames for each camera for a predetermined time. An example of determination will be described. In the image processing device and the image reproduction device according to the second embodiment, the local area image composition control unit 31a compares the interframe difference from each camera with respect to the image processing device and the image reproduction device according to the first embodiment shown in FIG. The values 141 to 14N are accumulated in units of local areas over a plurality of frames, and the image composition information 161 is output based on the values. Also in this case, the same effect as in the first embodiment can be obtained. Compared to the case shown in the first embodiment, more efficient encoding is possible even when the movement of the monitoring object takes a long time.

Embodiment 3 FIG.
In Embodiments 1 and 2, the local area image composition control unit 31a generates image composition information 161 so that images are combined in descending order of inter-frame difference values from each camera, and is suitable for each composite image. Although the frame rate 162 has been determined, Embodiment 3 shows a case where the encoding process is made more efficient by removing the high-frequency component of the composite image without reducing the frame rate.

  FIG. 9 is a block diagram illustrating the overall configuration of the image processing apparatus and the image reproduction apparatus according to the third embodiment. 9, the same reference numerals as those in the image processing apparatus shown in FIG. 1 denote the same or corresponding parts. The local area image composition control unit 31b outputs filter information 176 instead of the frame rate as an output parameter, and the local area image composition processing unit 32b performs filter processing on a part of the synthesized image according to the filter information 176. . Also, the encoding control unit 33 outputs the GOP structure 163 and the quantization information 177 as encoding parameters based on the image synthesis information 161 and the filter information 176.

  Next, the operation will be described. Similarly to the case shown in the first embodiment, the image signals 101 to 10N captured by the cameras 1 to N and the image signals 111 to 11N one frame before are each divided into a plurality of local areas, and each frame is divided in units of local areas. Difference values 141 to 14N are calculated. The local area image composition control unit 31b determines image composition information 161 such as a combination of local areas composing the composite image based on the inter-frame difference values 141 to 14N in units of local areas, and determines the local area composing the composite image. For a composite image whose inter-frame difference value is always smaller than a threshold within a predetermined time, this composite image is determined to be a background image, and filter information 176 is output so as to remove a high-frequency component by applying a low-pass filter. . The filter information 176 is information including, for example, a flag indicating ON / OFF of the filter, a level indicating the strength of the filter, and a filter coefficient.

  The local area image composition processing unit 32b performs filter processing on a part of the composite image according to the filter information 176 when generating the composite image. The encoding control unit 33 does not change the frame rate according to the composite image, and therefore, for all composite images, for example, encoding is performed with the GOP structure 163 using B pictures with M value = 3 and N value = 12. Give instructions to do. In addition, the local area unit image with the high-frequency component dropped has a flatter pattern than the original image, so the quantization information is set to be lower than the quantization width determined by normal information amount control. 177 is output to the image encoder 34.

  The image encoder 34 performs rearrangement of the input composite image signal 164 and encoding using inter-frame prediction according to the GOP structure 163 and the quantization information 177, and outputs a variable-length bit stream 165 for recording. Recording on the medium 35. In addition to the bit stream 165, the corresponding image composition information 161 and the frame rate 162 are recorded in the recording medium 35.

  In the image reproducing apparatus, as in the case of the first embodiment, the decoded image obtained by decoding the bit stream 169, combining the images in units of local areas, and taking and recording from a desired camera among the original cameras 1 to N. 175 is generated and output to the monitor 42 for reproduction display.

  As described above, in the image processing apparatus and the image reproduction apparatus according to the third embodiment, a synthesized image with a small inter-frame difference value is determined as a background image and a high-frequency component is removed by applying a low-pass filter. Therefore, it is possible to reduce the allocation of the information amount to the background portion, and it is possible to improve the image quality of the portion (moving body) with a large movement that is a point of interest as the surveillance camera system, or to extend the recording time.

  Although the case where the frame rate of the composite image is not changed is shown here, although the degradation of the decoded image increases, the frame rate may be controlled according to the inter-frame difference value of the local area constituting the composite image. good.

Embodiment 4 FIG.
In the first to third embodiments, a composite image is created and encoded by combining image signals from a plurality of cameras in units of local areas. However, in the image processing device and the image reproduction device shown in the fourth embodiment, a plurality of images are generated. Efficient encoding is performed by controlling the encoding order in accordance with the correlation between images from the camera.

  FIG. 10 is a block diagram illustrating the overall configuration of the image processing apparatus and the image reproduction apparatus according to the fourth embodiment. 10, the same reference numerals as those of the image processing apparatus and the image reproduction apparatus shown in FIG. 1 denote the same or corresponding parts. In FIG. 10, the local area feature amount calculation units 21 to 2N output pixel average values 181 to 18N and pixel deviation values 191 to 19N in units of local areas of the image signals 101 to 10N. The image correlation detection unit 43 refers to the average pixel values 181 to 18N, the pixel deviation values 191 to 19N, and the time information 151 to 15N in units of local areas from the respective cameras, and detects the correlation between the plurality of cameras. Correlation information 178 is output. Based on the image correlation information 178, the image encoding order control unit 44 determines the order in which the image signals 121 to 12N to be read from the frame memory units 11 to 1N are encoded in a predetermined time unit, and selects the selected image. An image correlation level 179 with the previous frame and camera selection information 180 are output together with the signal 200. The encoding control unit 33 determines the GOP structure 163 for performing image encoding based on the image correlation level 179 and the camera selection information 180. The recording medium 35 is a device such as an HDD that records the camera selection information 180 and the bit stream 165.

  Next, the configuration of the image playback device will be described. In FIG. 10, the image decoder 38 inputs the bit stream 169 and outputs the image signal 202 decoded by the image decoding means, and the reproduction image read control unit 45 performs camera selection information 201 corresponding to the decoded image signal 202. Then, referring to the camera information 167 to be reproduced, a read control signal 172 for selecting a necessary image signal 175 is output.

  Next, the operation will be described. Similar to the first embodiment, the digitally converted image signals 101 to 10N photographed by the cameras 1 to N are written into the memories in the frame memory units 11 to 1N in units of frames. The local area feature amount calculation units 21 to 2N divide the image signals 101 to 10N into a plurality of local areas, and obtain the pixel average values 181 to 18N and the pixel deviation values 191 to 19N of the image signals 101 to 10N for each camera. Calculate in units and extract image features from each camera.

  The image correlation detection unit 43 determines the correlation between images of any two frames based on the pixel average value and pixel deviation value in local area units from any two cameras. For example, FIG. 11 shows an example of determining the correlation between images taken from three cameras. FIG. 11 shows the correlation between two camera images among the images of the cameras 1 to 3. First, a difference between two frames of a pixel average value and a pixel deviation value in local area units is calculated, and each difference value is classified into a plurality of levels with a predetermined threshold value. Here, an example is shown in which the level when the difference value is small is set to a small value and the level is divided into 10 levels from 0 to 9. Next, the level value of each local area is accumulated for one frame, and two frames having a small accumulated value are determined as images having high correlation. In the example of FIG. 11, the cumulative value of the difference value between the pixel average value and the pixel deviation value of the cameras 1 and 3 is 73, the correlation is the highest (the accumulated value is small), and the pixel average value of the cameras 1 and 2 The accumulated value of the difference values of the pixel deviation values is 96, and it is determined that the correlation is the smallest (the accumulated value is large). The image correlation detection unit 43 determines the correlation between any two frames in a predetermined time unit, and outputs the determination result as image correlation information 178.

  Based on the image correlation information 178, the image encoding order control unit 44 selects all the camera image signals 121 to 12N within a predetermined time in order of high correlation, and determines the encoding order. . In the image signal shown in FIG. 11, the correlation between the images of the camera 1 and the camera 3 is the highest, and the images of the cameras 2 and 3 are higher in correlation than the cameras 1 and 2. It decides to perform encoding in the order of camera 1 → camera 3 → camera 2. The read control signals 131 to 13N are generated so as to read the image signals 121 to 12N from the frame memory units 11 to 1N according to the encoding order, and the image correlation level 179 with the previous frame and the camera selection are selected together with the selected image signal 200. Information 180 is output. The camera selection information 180 includes a camera number and camera time information.

  The encoding control unit 33 determines the GOP structure 163 for encoding the image signal 200 based on the image correlation level 179. Instructs to encode using P or B picture for inter-frame prediction when the image is highly correlated with the previous frame, and to encode using I picture for intra-frame prediction for images with low correlation To do. In the image signal shown in FIG. 11, since the correlation between both images is high at the portion where the camera 1 is switched to the image of the camera 3, encoding using inter-frame prediction is performed, and at the portion where the camera 3 is switched to the camera 2, Since the correlation between the two images is low, encoding without using inter-frame prediction is performed. As a result, the encoding can be performed more efficiently than when the encoding is simply performed in the order of camera 1 → camera 2 → camera 3. In addition to the bit stream 165 encoded by the image encoder 34, camera selection information 180 of the corresponding image signal is also recorded on the recording medium 35. The above is the operation of the image processing apparatus.

  Next, the operation of the image playback device will be described. As in the case of the first embodiment, the stream read control unit 37 performs the necessary bit stream 169 among the data recorded on the recording medium 35 based on the time information 166 according to the instruction from the user, and the camera selection information. A read control signal 168 for reading 201 is output, and corresponding data is read from the recording medium 35. The image decoder 38 performs image decoding on the bit stream 169 and writes the image signal 202 in the frame memory unit 40. The playback image read control unit 45 refers to the camera selection information 201 corresponding to the decoded image signal 202 and the camera information 167 to be played back, and outputs a read control signal 172 to the frame memory unit 40, so that the original camera 1 The decoded image 175 photographed and recorded from a desired camera among -N is output to the monitor 42 and reproduced and displayed.

  As described above, the image processing apparatus and the image reproduction apparatus according to the fourth embodiment control to encode images from a plurality of cameras in a highly correlated order within a predetermined time, and the magnitude of the correlation. Since the GOP structure is determined according to the encoding, more efficient encoding can be performed and the recording time can be extended compared to the case where encoding is performed according to the input order from a plurality of cameras. .

  Here, the pixel average value and the pixel deviation value are used as indices for obtaining the correlation between image signals from different cameras. However, a cumulative value or sum of squares of pixel difference values for each local area may be used. .

1-N camera, 11-1N frame memory unit, 21-2N local area feature amount calculation unit, 31a, 31b local area image composition control unit, 32a, 32b local area image composition processing unit, 33 encoding control unit, 34 image Encoder, 35 recording medium, 36 user interface unit, 37 stream read control unit, 38 image decoder, 39 local area image recombination control unit, 40 frame memory unit, 41 local area image recombination processing unit, 42 monitor, 43 image correlation detection unit, 44 image coding order control unit, 45 reproduction image read control unit, 101 to 10N image signal, 111 to 11N one frame previous image signal, 121 to 12N image signal, 131 to 13N read control signal, 141-14N Inter-frame difference value, 151-15N Time information, 161 image composition information, 162 frame rate, 163 GOP structure, 164 composite image signal, 165 bit stream, 166 time information, 167 camera information, 168 read control signal, 169 bit stream, 170 image composition information, 171 composite image Signal, 172 Read control signal, 173 Composite image signal, 174 Image recombination information, 175 Decoded image signal, 176 Filter information, 177 Quantization information, 178 Image correlation information, 179 Image correlation level, 180 Camera selection information, 181-18N Pixel average value, 191 to 19N pixel deviation value, 200 image signal, 201 camera selection information, 202 image signal.

Claims (13)

A plurality of imaging means for capturing moving images;
A feature amount calculating unit that divides a plurality of image signals generated from moving images captured by the plurality of imaging units into a plurality of local areas, and calculates inter-frame difference values for each local area;
Image combining means for combining the images of the local area based on the calculated inter-frame difference value and generating a plurality of combined images;
Frame rate control means for controlling the frame rate of the composite image based on the inter-frame difference value of the local area constituting the composite image;
Image encoding means for performing encoding using inter-frame prediction on the synthesized image and generating a bitstream;
An image processing apparatus comprising: Coding control means for determining a GOP (Group Of Pictures) structure in image coding of the synthesized image based on the frame rate and the inter-frame difference value of the local area constituting the synthesized image;
The image encoding means performs image encoding based on the GOP structure;
The image processing apparatus according to claim 1.   The image synthesizing unit classifies the images of the local area in order of the magnitude of the inter-frame difference values of the plurality of local areas, and generates the plurality of synthesized images by combining the images of the local area for each classification. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The frame rate control means is different from the composite image in a frame rate of another composite image that is different from the composite image in that the inter-frame difference value of the local area to be configured is larger than the composite image. The image processing apparatus according to claim 1, wherein the frame rate is determined.   The said image synthetic | combination means combines the image of the said local area based on the magnitude | size of the value which accumulated the said frame difference value for several frames, The any one of Claims 1-4 characterized by the above-mentioned. Image processing apparatus. A plurality of imaging means for capturing moving images;
A feature amount calculating unit that divides a plurality of image signals generated from moving images captured by the plurality of imaging units into a plurality of local areas, and calculates inter-frame difference values for each local area;
Image combining means for combining the images of the local area based on the calculated inter-frame difference value and generating a plurality of combined images;
Filter means for removing high-frequency components of the composite image based on inter-frame difference values of the local area constituting the composite image;
Image encoding means for performing encoding using inter-frame prediction on the plurality of synthesized images and generating a bitstream;
An image processing apparatus comprising:   The image processing apparatus according to claim 6, wherein the filter unit removes a high frequency component from the composite image in which the inter-frame difference value of the local area constituting the composite image is smaller than a predetermined value. .   The image according to claim 6 or 7, wherein the image encoding means controls a quantization width of the composite image based on the inter-frame difference value of the local area constituting the composite image. Processing equipment.   The image processing apparatus according to claim 8, wherein the image encoding unit lowers a quantization width for the synthesized image in which the inter-frame difference value of the local area to be configured is equal to or less than a predetermined value. Image extracting means for extracting the image of the local area constituting the image signal from a plurality of synthesized images generated by combining the images of the local area constituting the plurality of image signals;
Image recombining means for generating the image signal by combining the extracted images of the local area;
Image display means for displaying the generated image signal on a monitor;
The composite image is a composite image synthesized by combining the local area images based on inter-frame difference values of the images of the plurality of local areas;
An image reproducing apparatus characterized by the above. A plurality of imaging means for capturing moving images;
A correlation calculating unit that divides a plurality of image signals generated from moving images captured by the plurality of imaging units into a plurality of local areas, respectively, and calculates a correlation of the plurality of image signals for each local area; ,
An encoding control means for determining an order and a GOP structure for encoding the image signals based on the calculated correlation;
Image encoding means for encoding the image signal using inter-frame prediction based on the encoding order and the GOP structure, and outputting a bitstream;
An image processing apparatus comprising: An image processing system comprising: an image processing device that encodes an image signal; and an image reproduction device that decodes and reproduces the encoded image signal on a monitor,
The image processing apparatus includes:
A plurality of imaging means for capturing moving images;
A feature amount calculating unit that divides the plurality of image signals generated from the moving images captured by the plurality of imaging units into a plurality of local areas, and calculates inter-frame difference values for each local area;
Image combining means for combining the images of the local area based on the calculated inter-frame difference value and generating a plurality of combined images;
Frame rate control means for controlling the frame rate of the composite image based on the inter-frame difference value of the local area constituting the composite image;
Image encoding means for performing encoding using inter-frame prediction on the plurality of synthesized images and generating a bitstream;
Image storage means for storing a combination of the bit stream and the local area constituting the composite image;
With
The image reproduction device includes:
Data acquisition means for acquiring a combination of the bitstream and the local area from the image storage means;
Image decoding means for decoding the bitstream acquired by the data acquisition means and generating a composite image;
Image extracting means for extracting an image of the local area constituting the image signal from the decoded composite image based on a combination of the local areas;
Image recombining means for generating the image signal by combining the extracted images of the local area;
Image display means for displaying the generated image signal on a monitor;
An image processing system comprising: An image processing system comprising: an image processing device that encodes an image signal; and an image reproduction device that decodes and reproduces the encoded image signal on a monitor,
The image processing apparatus includes:
A plurality of imaging means for capturing moving images;
A feature amount calculating unit that divides the plurality of image signals generated from the moving images captured by the plurality of imaging units into a plurality of local areas, and calculates inter-frame difference values for each local area;
Image combining means for combining the images of the local area based on the calculated inter-frame difference value and generating a plurality of combined images;
Filter means for removing high-frequency components of the composite image based on inter-frame difference values of the local area constituting the composite image;
Image encoding means for performing encoding using inter-frame prediction on the plurality of synthesized images and generating a bitstream;
Image storage means for storing a combination of the bit stream and the local area constituting the composite image;
With
The image reproduction device includes:
Data acquisition means for acquiring a combination of the bitstream and the local area from the image storage means;
Image decoding means for decoding the bitstream acquired by the data acquisition means and generating a composite image;
Image extracting means for extracting an image of the local area constituting the image signal from the decoded composite image based on a combination of the local areas;
Image recombining means for generating the image signal by combining the extracted images of the local area;
Image display means for displaying the generated image signal on a monitor;
An image processing system comprising:

Images