JP2006197098A - Image transmission system of network camera - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce the amount of communication data and to lighten an encoding/decoding processing load in a system comprising a network camera simultaneously distributing two video streams of video with a view angle for photography and wide-angle video and a viewer where those video streams are superposed and displayed one over the other. <P>SOLUTION: When superposition display is performed, encoding processing is carried out while a region of a wide-angle video region which is hidden behind photographic view-angle video is replaced with image data painted out in one color and the resulting data are outputted as wide-angle video data. The replacing monochromatic image data has values closest to the luminance and color difference data of a peripheral image in the object region. Further, the monochromatic image data replacing the region according to peripheral image data are updated each time for each frame of the video. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a transmission method of image data in a network camera capable of monitoring or photographing a video of a camera connected to a network from a remote place.

  Conventionally, when a camera connected to a network is operated and photographed from a remote location, the location where the camera is installed is far away and the situation around the subject cannot be seen directly. It was difficult to find and track. In order to solve this, a camera that simultaneously distributes a wide-angle video for viewing surrounding video in addition to a video angle of view for photographing has been realized (for example, Patent Documents 1 to 3).

  On the other hand, in a viewer for remote control of a camera, a so-called picture-in-picture display in which a shooting angle-of-view video window is displayed superimposed on a wide-angle video is often used as a display mode for simultaneously displaying two video streams. ing. In this picture-in-picture display, a non-displayed portion is cut out from overlapping image data, and only necessary data is encoded and transmitted to shorten the time required for encoding and decoding processing (for example, patents) Document 4) has been proposed.

JP-A-8-275043 JP-A-8-275044 Japanese Patent Laid-Open No. 9-037135 JP-A-8-032944

  In a system consisting of a network camera that delivers two streams of shooting angle-of-view video and wide-angle video, if two streams are delivered at the same time, the amount of data flowing over the network increases. There is a problem that it becomes difficult to transmit.

  On the other hand, in order to further widen the communication band, it is necessary to introduce new equipment, and when the compression rate in encoding each video data is increased, the quality of the video is reduced accordingly. There is also a problem.

  In the method shown in Patent Document 1, in order to cut out in an arbitrary shape, it is necessary to perform an encoding process including shape information, and the encoded data increases accordingly, and further an encoding / decoding process is performed. There is a problem that it becomes complicated. Furthermore, since the encoding unit is normally performed for each macroblock (for example, 16 × 16 dots), there is a problem that the accuracy of clipping is limited.

  The present invention has been made in view of the above problems, and the object of the present invention is a network camera that simultaneously delivers two video streams of a shooting angle of view video and a wide-angle video, and a viewer that superimposes and displays each video stream. The purpose of this system is to reduce the amount of communication data and the encoding / decoding processing load.

  In order to solve the above-described problem, the present invention performs encoding processing by replacing an area hidden in a photographic angle-of-view image in a wide-angle video area with image data filled with a single color when superimposed display is performed. The configuration is such that it is transmitted as video data.

  The single-color image data to be replaced is set to a value closest to the luminance and color difference data of the peripheral image of the target area.

  The monochrome image data to be replaced in accordance with the peripheral image data is updated every frame of the video.

  Further, the monochromatic image data to be replaced in accordance with the peripheral image data is configured to be updated only when the position, size, or shape of the overlapping area is changed.

  Further, the monochrome image data to be replaced in accordance with the peripheral image data is configured to be updated for each intra frame in encoding.

  By the above means, the area filled with a single color has increased spatial and temporal redundancy, so that the coding efficiency is improved and the coding amount of the entire moving image can be reduced. In addition, by setting the fill color to the value closest to the data value of the surrounding image, the discontinuity at the boundary between the area to be filled and the original image is reduced, so that the coding efficiency of the entire image can be further increased. .

According to the present invention, when overlay display is performed in a system including a network camera that simultaneously delivers two video streams of a shooting angle of view video and a wide-angle video and a viewer that overlays and displays each video stream. In addition, the encoding process is performed by replacing the area hidden in the shooting angle-of-view image in the wide-angle video area with the image data filled with a single color, and transmitting as wide-angle video data, the encoding efficiency is improved, and the entire moving image is Since the amount of encoding can be reduced, an effect that high-quality video data can be transmitted and received even on a communication path with a narrow bandwidth can be obtained. In addition, compression with a general coding system,
Since the decompression process can be performed, there is an effect that an extra processing load is not applied to the system.

  Furthermore, during the period from when the overlapping area is changed to when the wide-angle video is updated, a good wide-angle video can be displayed even if there is a network delay by performing a process that complements the video without filling. Therefore, the operability can be improved.

  Hereinafter, an embodiment of a system constituted by a network camera and a viewer according to the present invention will be described with reference to the drawings.

<Embodiment 1>
FIG. 1 is a block diagram showing a schematic configuration of a network camera according to Embodiment 1 of the present invention.

  In FIG. 1, reference numeral 101 denotes a wide-angle camera unit that is a camera unit for taking a wide-angle image. The wide-angle camera unit 101 includes a lens group 1 (102), a CCD 1 (103), and a CCD control unit 1 ( 104) and an image processing unit 1 (105). The lens group 102 is composed of a plurality of lenses for optically projecting a subject image onto the CCD 102, and has a characteristic of capturing a subject at a wide angle.

  A CCD (photoelectric conversion element) 103 is an element for converting a captured image projected by the lens group 102 into an analog electric signal, has a VGA size (640 × 480 dots) pixel number, and has 30 frames / frame. Data can be read at a rate of seconds. The CCD control unit 104 is a timing generator for supplying a transfer clock signal and a shutter signal to the CCD 103, a circuit for performing noise removal and gain processing of the CCD output signal, and for converting an analog signal into a 10-bit digital signal. A / D conversion circuit and the like.

  The image processing unit 105 performs image processing such as gamma conversion, color space conversion, white balance, AE, and flash correction on the 10-bit digital signal output from the CCD control unit 104, and performs YUV (4: 2: 2) An 8-bit digital signal is output in the format. This wide-angle video image is output at a cycle of 30 frames / second with the number of pixels of VGA size (640 × 480 dots).

  Reference numeral 106 denotes a shooting angle of view camera unit which is a camera unit for shooting at a desired angle of view. As with the wide-angle camera unit 101, the shooting angle-of-view camera unit 106 includes a lens group 2 (107), a CCD 2 (108), a CCD control unit 2 (109), and an image processing unit 2 (110), as well as a shooting direction. A control motor 117 for panning and tilting movement is included. The lens group 107 is configured as a zoom lens having characteristics that allow photographing at a normal angle of view.

  The CCD 108 has a pixel number of UXGA size (1600 × 1200), and can read out the UXGA size and read out the VGA size at 30 frames / second. The CCD control unit 109 and the image processing unit 110 perform the same operation as each block of the wide-angle camera unit 101. Similarly, the shooting angle-of-view video is normally output in a cycle of 30 frames / second in VGA size, but when still image capture control is performed, one frame of UXGA size image data is output in response to this. The pan / tilt control motor 117 includes two motors, that is, a motor for rotating the camera unit in the horizontal direction and a motor for rotating the camera unit in the vertical direction, and respective drive circuits.

  As described above, the network camera of the present invention is equipped with two camera units that can shoot at different angles of view. These camera units are connected to a bus 118, and image data captured by each camera is transmitted via the bus 118 and transferred to other processing blocks. Connected to the bus 118 are a still image encoding unit 111, a moving image encoding unit 112, a CPU (Central Processing Unit) 113, a communication interface unit 114, a ROM (Read Only Memory) 115, and a RAM (Random Access Memory) 116, respectively. Has been.

  The still image encoding unit 111 is a processing block for performing JPEG encoding. The UXGA size YUV format data passed from the shooting angle-of-view camera unit 106 via the bus 118 by still image capture control, Compress and encode into a JPEG format as a still image. Similarly, the encoded still image data is temporarily stored in a later-described encoded data temporary storage area 116c in the RAM 116 via the bus 118.

  The moving image encoding unit 112 is a processing block for performing MPEG4 encoding. The VGA size YUV format data transferred from the wide-angle camera unit 101 via the bus 118 is converted into a frame of 30 frames / second. Perform compression encoding to MPEG4 format at the rate. Similarly, compression encoding is performed on VGA size data transferred from the shooting angle of view camera unit 106. These encoded data are temporarily stored in the encoded data temporary storage area 116c in the RAM 116 for each frame.

  The communication interface unit 114 is a processing block for transmitting and receiving data through a network, and mainly manages processing of a network layer, a data link layer, and a physical layer in a wired LAN. Video data shot and encoded by the camera unit is transmitted to a remote viewer through the communication interface unit 114.

  The CPU 113 performs various controls based on the control program in the ROM 115. Among these controls are processing for enlarging / reducing the captured angle of view image captured by the captured angle of view camera unit 106 according to the size of the display area of the viewer, and wide angle image captured by the wide angle camera unit 101. Among them, processing to fill the area to be superimposed with the shooting angle of view video with a single color, processing to divide the encoded video data into communication packets, network protocol processing such as TCP, UDP, information exchange protocol processing with viewer, command It includes processing, pan / tilt control motor drive processing according to pan and tilt operations, overall sequence control processing such as timing control of each block and data transfer control in the shooting operation.

  The RAM 116 includes an image development area 116a, an encoding frame memory 116b, an encoded data temporary storage area 116c, a work area 116d, and a temporary save area 116e. The image development area 116a serves as a temporary buffer for temporarily storing captured images (YUV digital data) sent from the image processing units 105 and 110, and enlargement / reduction processing of image data, single color painting processing, etc. Used as a dedicated work area for images.

  The encoding frame memory 116b is used as a frame memory for temporarily storing the reference frame data before and after the inter-frame prediction in the MPEG4 compression encoding process by the moving image encoding unit 112. The encoded data temporary storage area 116c is used as a work memory for temporarily storing the data compressed and encoded by the still image encoding unit 111 and the moving image encoding unit 112 until the data is transmitted from the communication interface unit 114. Is done. The work area 116d is a work area for various programs, and the temporary save area 116e is an area for temporarily saving various data.

  Next, a block diagram showing a schematic configuration of the viewer according to Embodiment 1 of the present invention in FIG. 2 will be described.

  In FIG. 2, each processing block of the viewer exchanges data through a bus 212. A bus 212 includes a CPU (Central Processing Unit) 201, a still image decoding unit 202, a moving image decoding unit 203, a display control unit 204, a communication interface unit 207, a ROM (Read Only Memory) 208, and a RAM (Random Access Memory). 209 and data storage means 210 are connected to each other. A control key 211 is connected to the CPU 201, a display drive unit 205 is connected to the display control unit 204, and a display unit 206 is connected to the display drive unit 205.

  The CPU 201 performs various controls based on the control program in the ROM 208. These controls include operating system and file system processing, processing to restore received communication packets into encoded video data for one frame, network protocol processing such as TCP and UDP, and information exchange protocol processing with cameras. , Command processing, application processing corresponding to the operation of the control key 211, and overall sequence control processing such as timing control of each block and data transfer control in the display reproduction operation.

  The still image decoding unit 202 is a processing block for performing JPEG decoding. In this block, a process of decoding the UXGA size JPEG encoded data received from the network camera and stored in the data storage unit 210 into YUV format digital data according to the user operation in the application is performed. The decoded YUV still image data is temporarily stored in a display image development memory 209b (to be described later) in the RAM 209 via the bus 212.

  The moving picture decoding unit 203 is a processing block for performing MPEG4 decoding. In this block, the MPEG4 encoded data of the wide-angle video and the shooting angle-of-view video received from the network camera is decoded into YUV format digital data at a frame rate of 30 frames / second. The decoded YUV moving image data is temporarily stored in a display image expansion memory 209b (to be described later) in the RAM 209 via the bus 212 for each frame.

  The display control unit 204 reads the YUV digital image data stored in the display image development memory 209b, converts it into an RGB digital signal, and then outputs it to the display drive unit 205, or stores it in the frame buffer 209c. Similarly, a process of outputting the RGB digital data to the display driving unit 205 is performed. The display driving unit 205 performs control for driving the display unit 206. The display unit 206 is a display for displaying an image, and is a VGA size (640 × 480 dots) TFT liquid crystal display device.

  The communication interface unit 207 is a processing block for transmitting and receiving data through a network, and mainly controls processing of a network layer, a data link layer, and a physical layer in a wired LAN. The encoded wide-angle video data, shooting angle-of-view video data, and shooting still image data are received from a remote network camera through the communication interface unit 207.

  The RAM 116 includes an encoding frame memory 209a, a display image development memory 209b, a frame buffer 209c, a work area 209d, and a temporary save area 209e. The encoding frame memory 209a is used as a frame memory for temporarily storing the reference frame data before and after the inter-frame prediction in the MPEG4 decoding process by the moving picture decoding unit 203.

  The display image development memory 209b is a temporary buffer for temporarily storing the YUV digital data decoded by the still image decoding unit or the YUV digital data decoded by the moving image decoding unit, or by the CPU 201. It is used as an image-dedicated work area for image data enlargement and reduction processing. The frame buffer 209c is used as a VRAM that stores display data created by an application executed by the CPU 201, and is stored as RGB 8-bit digital data. The work area 209d is a work area for various programs, and the temporary save area 209e is an area for temporarily saving various data.

  The data storage unit 210 is used as an image data storage memory for storing JPEG encoded data of a still image received from a network camera as a file. A file system is constructed in this memory.

  Hereinafter, a control processing procedure of the system configured by the network camera and the viewer according to the present embodiment will be described based on flowcharts shown in FIG.

  FIG. 3 is a flowchart showing main processing of the network camera body.

  First, in step S301 in FIG. 3, processing for establishing a communication connection with the viewer using the TCP / IP protocol is performed. Next, in step S302, processing for acquiring overlay information from the viewer is performed. Here, an information request command is transmitted to the viewer, and information indicating the shape, size, and position of the captured field-of-view video superimposed on the wide-angle video data transmitted by the viewer is received in response. Is called.

  Hereinafter, the process proceeds to shooting and transmission of shooting angle-of-view video and wide-angle video.

  In step S303, shooting angle-of-view video processing for one frame is performed. Details will be described with reference to FIG. In step S304, wide-angle video processing for one frame is performed. Details will be described with reference to FIG.

  When these processes for one frame are completed, in step S305, it is checked whether or not a command has been received from the viewer. If no command has been received, the process returns to the photographing angle-of-view video process in S303. If a command is received in step S305, first, in step S306, it is checked whether it is a shooting end command. If it is a shooting end command, a network camera end process and a power-off process are executed in step S307. The process ends.

  If it is not a shooting end command in S306, it is checked in step S308 whether it is a shutter command. If it is a shutter command, in step S309, a still image capture process, that is, a UXGA size still image in the shooting angle of view camera unit 106 is checked. As described above, a process for extracting data for one frame is executed.

  Next, in step S310, a still image encoding process is executed, and JPEG encoded data is created in the still image encoding unit 111 from the still image data in the YUV format extracted in S309. The JPEG encoded data is transmitted to the viewer via the communication interface unit 114 in step S311. When the transmission process is completed, the process returns to the photographing angle-of-view video process in S303 again.

  If it is not a shutter command in S308, it is checked in step S312 whether it is a superposition condition change command. If it is a change command, in step S313, superimposition information is acquired in the same procedure as in S302. Execute. Then, the process returns to the shooting angle-of-view video processing in S303. If the command is not a change command in S312, other command processing is executed in Step S314, and the process returns to S303 shooting angle-of-view video processing. Other commands include, for example, pan / tilt control commands, commands for changing shooting conditions (image correction conditions, compression rate, frame rate, etc.), commands linked to other applications, and the like.

  As described above, by repeatedly executing the processes from S303 to S314 at intervals of 1/30 seconds, it is possible to transmit the shooting angle-of-view video and wide-angle video as a moving image stream.

  Next, FIG. 4 shows a flowchart showing details of the processing angle-of-view video processing for one frame described in FIG. 3 and S303.

  In FIG. 4, step S <b> 401 is processing for reading one frame of the shooting angle of view video. In this process, one frame of video shot at a frame rate of 30 frames / second by the shooting angle-of-view camera unit 106 is read as YUV digital data into the image development area 116 a of the RAM 116.

  Next, in step S402, referring to the overlay information acquired in FIG. 3 and S302, enlargement or reduction processing is performed to the size of the shooting angle-of-view video displayed on the viewer. The image data for which the resizing process has been completed is similarly stored in the image development area 116a, and in step S403, the MPEG4 encoding process is performed by the moving image encoding unit 112 on the image data, and the encoded data is temporarily stored in the RAM 116. Stored in area 116c. The stored encoded data is divided into transmission packet sizes in step S404 and a transmission header is added. Then, in step S405, the communication interface unit 114 transmits data to the viewer.

  FIG. 5 is a flowchart showing details of the wide-angle video processing for one frame described in FIG. 3 and S304.

  In FIG. 5, step S501 is a process of reading a wide-angle video for one frame. In this process, similarly to the reading process of the shot angle of view video, the video shot at the frame rate of 30 frames / second in the wide-angle camera unit 101 is read as one frame of YUV digital data to the image development area 116a of the RAM 116. .

  Then, in step S502, referring to the overlay information acquired in FIG. 3 and S302, a region that is to be superimposed on the shooting angle-of-view image, that is, a region that is not displayed, is filled with a single color among the wide-angle images displayed on the viewer. Process. Here, the YUV data is replaced with (0, 128, 128) as black data. The image data for which the filling process has been completed is similarly stored in the image development area 116a. In step S503, the MPEG4 encoding process by the moving image encoding unit 112 is performed on the image data, and the encoded data is temporarily stored in the RAM 116. Stored in area 116c.

  Thereafter, the stored encoded data is divided into transmission packet sizes in step S504 and a transmission header is added in step S504, and the viewer is then viewed by the communication interface unit 114 in step S505. Data transmission is performed.

  The processing of the shooting operation in the network camera has been described above with reference to FIGS. 3 to 5. Next, the display processing in the viewer will be described.

  FIG. 6 is a flowchart showing the main processing of the viewer body.

  First, in step S601 of FIG. 6, processing for establishing a communication connection with the network camera using the TCP / IP protocol is performed in the same manner as the processing of S301 in the network camera. Next, in step S602, processing for transmitting overlay information is performed in accordance with the request command transmitted from the viewer. In other words, this is a process of transmitting information representing the shape, size, and position of the captured field-of-view video to be superimposed on the wide-angle video data, determined by the user's operation, to the network camera.

  Next, in step S603, processing for checking whether or not there has been an operation by the user is performed. If no operation event is detected, the received image data display process described in steps S604 to S617 is executed.

  In step S604, the communication interface unit 207 performs processing for receiving image data for one frame. Next, in step S605, if the received image is wide-angle frame data, in step S606, the received frame data is subjected to MPEG4 decoding processing by the moving picture decoding unit 203.

  The wide-angle frame data that has been decoded and expanded into YUV digital data is written in the display image expansion memory 209b of the RAM 209 in step S607, and then proceeds to step S611.

  On the other hand, if it is determined in step S605 that the received image is not wide-angle frame data, it is checked in step S608 if the received image is photographed view angle frame data. If the received image is photographed view angle frame data, step S609 is performed in the same manner as in steps S606 and S607. The moving picture decoding unit 203 executes MPEG4 decoding processing on the received frame data.

  The captured field angle frame data that has been decoded and expanded into YUV digital data is written in the display image expansion memory 209b of the RAM 209 in step S610, and then proceeds to step S611. In step S611, the superposition processing of the wide-angle video and the shooting angle-of-view video, that is, the shooting angle-of-view frame data similarly written in S610 is added to the wide-angle frame data written in the display image development memory 209b in S607. A process of overwriting and creating composite video data is executed. Here, the area to be overwritten coincides with the area that is painted in a single color by the network camera. In step S612, the composite video data is converted into RGB digital data by the display control unit 204, and then the display drive unit 205 performs display processing on the display unit 206.

  If it is determined in step S608 that the frame is not shooting angle of view frame data, it is checked in step S613 if the received image is still image frame data. If it is still image frame data, the still image frame data is converted into a file in step S614. And stored as a JPEG file in the image data storage memory of the data storage unit 210.

  In step S615, the still image decoding unit 202 executes JPEG decoding processing on the received frame data. The still image frame data that has been decoded and developed into YUV digital data is subjected to processing for reducing the UXGA size to the display resolution in step S615, and then converted into RGB digital data in step S617 as in S612. Thereafter, a still image display process is executed as confirmation of the received image. Then, the process returns to the check of user operation in S603 again.

  If an event of an operation by the user is detected in step S603, it is checked in step S618 whether the event is a shooting end operation event. If shooting is ended, a shooting end command is sent to the network camera in step S619. Process to send. Further, in step S620, a process for turning off the power of the viewer is executed, and the process ends.

  If it is not a shooting end event in S618, it is checked in step S621 whether the event is a shutter operation, and if it is a shutter operation, in step S622, a shutter command for instructing still image shooting is sent to the network camera. Then, the process returns to the user operation check process in step S603.

  If it is not a shutter operation event in S621, it is checked in step S623 whether or not an operation for changing the overlay state has been performed. The change of the superposition state is an operation for changing the position, size, shape, and the like of the shooting angle-of-view image superimposed on the wide-angle image, and is performed by an operation using the control key 211 attached to the viewer. Here, if a change operation is performed, in step S624, a process of transmitting overlay information that matches the change condition to the network camera is performed, and then the process returns to the check process of the user operation in S603 again.

  If the event is not a change operation event in step S623, in step S625, processing for other operations, for example, pan / tilt operation, operation for changing shooting conditions (image correction conditions, compression rate, frame rate, etc.), and other applications The operation linked to is executed, and the process returns to S603 in the same manner.

  Next, FIG. 7 illustrates video captured by the network camera and video displayed by the viewer.

  In FIG. 7, reference numeral 701 denotes a wide-angle image of one frame taken by the wide-angle camera unit 101 in the network camera. A region 702 indicated by a broken line in the video indicates a region where the shooting angle-of-view video corresponding to the superposition information transmitted from the viewer is superimposed.

  Reference numeral 703 denotes an image obtained by performing monochrome painting processing on the area indicated by 702 with respect to the wide-angle image. The image data of 703 is encoded and transmitted to the viewer as a wide-angle image. An area 704 is filled with a single color.

  Reference numeral 705 denotes a one-frame shooting angle-of-view image shot by the shooting angle-of-view camera unit 106 in the network camera.

  Reference numeral 706 denotes a display image displayed on the display unit 206 of the viewer. The wide-angle video 703 and the shooting angle-of-view video 705 transmitted from the network camera are combined and displayed. Among these, 707 is a wide-angle video and 708 is a shooting angle-of-view video.

<Embodiment 2>
In the first embodiment, an embodiment has been described in which an area of a wide-angle video that is to be overlaid with a captured video is transmitted as a video that is filled with fixed single-color data, and transmitted from the network camera to the viewer. However, the present invention is not limited to this. Is not to be done.

  In the fixed monochromatic data, since the boundary between the filled monochromatic data area and the surrounding video area becomes discontinuous, the spatial frequency of the image becomes high and the coding efficiency is lowered. Therefore, in the present embodiment, an embodiment will be described in which the overlay area fill data is dynamically switched in accordance with the surrounding pixels.

  FIG. 8 is a flowchart showing the processing in the second embodiment regarding the details of the wide-angle video processing for one frame explained in FIG. 3 and S304 in the first embodiment.

  In FIG. 8, step S801 is a process of reading a wide-angle video for one frame as in FIG. 5 and S501. In this process, similarly to the reading process of the shooting angle of view video, the video shot at the frame rate of 30 frames / second in the wide-angle camera unit 101 is read as YUV digital data for one frame into the image development area 116a of the RAM 116. .

Next, in step S802, the boundary of the region where the captured image is superimposed on the wide-angle image (the portion indicated by 702 in FIG. 7) matches the boundary of the pixel block (shown in FIG. 9), or It is checked whether or not the boundary is a position for dividing the block (shown in FIG. 10). This pixel block represents a block composed of pixels of 8 × 8 dots. Usually, JPEG, which is a still image compression method, MPEG4, which is a moving image compression method, is the upper and lower of the 8 × 8 pixel block. Coding units are made using four blocks (MCU and macroblock) adjacent to the left and right.

  In step S802, if the overlapping boundary coincides with the block boundary, in step S803, a process of obtaining an average value of luminance and color difference data in a peripheral block one block outside the overlapping region is executed, and further in step S804. In step S807, the overlapping area is filled with the obtained average value data, and the process advances to step S807.

  If the overlap boundary does not coincide with the block boundary in S802, a process for obtaining the average value of the luminance and color difference data of the block including the boundary of the overlap area is executed in Step S805, and further in Step S806. The area surrounded by the block one block inside the block including the boundary is filled with the obtained average value data, and the process proceeds to S807.

  The encoding process in step S807, the dividing process into transmission packets in S808, and the transmission process to the viewer in S809 are the same processes as in FIG. 5, S503, S504, and S505, respectively.

  FIG. 9 is a diagram showing a block configuration in a case where the boundary of the overlapping region coincides with the boundary of the pixel block.

  In FIG. 9, reference numeral 901 denotes a line representing the boundary of the overlapping region, and the grayed out portion is the overlapping region. Reference numeral 902 denotes a pixel block that is not filled, and reference numeral 903 denotes a pixel block that is filled with a single color. Each pixel block is composed of 8 × 8 pixels.

  FIG. 10 is a diagram illustrating a pixel configuration when the boundary of the overlapping region is a position where the pixel block is divided.

  In FIG. 10, reference numeral 1001 denotes a line representing the boundary of the overlapping area as in FIGS. 9 and 901, and reference numeral 1002 denotes a line representing the boundary of the area to be filled with a single color. Similarly, the grayed-out portion is an overlap region, and the dark gray portion of the region is a region filled with a single color. Reference numeral 1003 denotes a pixel block that is not filled, and reference numeral 1004 denotes a pixel block that is filled with a single color.

  In the present embodiment, as described with reference to FIG. 8, the processing procedure is such that the average value is calculated and the value of the filled image data is determined each time each frame of the wide-angle video is read, but the processing load is reduced. Therefore, the embodiment may be such that the filled image data is fixedly updated at intervals of once every several frames.

Needless to say, the present invention can be applied as an embodiment in which means for detecting the motion of a wide-angle video is provided and the filled image data is updated when there is a certain amount of motion. Similarly, an embodiment in which the filled image data is updated only when it is detected that the overlay position has been changed by the operation of the viewer is also conceivable. Even in these cases, in a normal shooting composition, the wide-angle image often occupies the majority of the background,
Since there is almost no intense movement in a short time, the coding efficiency is not greatly impaired.

<Embodiment 3>
In the second embodiment, an embodiment has been described in which a region to be overlapped with a captured image in a wide-angle image is determined according to the pixel data of the surrounding image, and all pixels in the region are filled with the same data. However, the present invention is not limited to this.

  In the second embodiment, the fill data value is determined by the average value of the peripheral video data in the overlap area. However, if the difference in the peripheral video data value is large depending on the position of the peripheral video, the fill data in the overlap area is determined. Because the number of pixels in which the difference from the above increases becomes large, the encoding efficiency may not increase.

  Therefore, in the present embodiment, the data for one block inside the boundary between the overlapping area and the peripheral video area is continuously variable between the numerical value of the peripheral image data and the numerical value of the filled data. A process is performed that is composed of pixel data that is numerical and increases data redundancy.

  FIG. 11 is a diagram showing the configuration of blocks near the boundary between the overlapping region and the peripheral video in the present embodiment.

  In FIG. 11, reference numeral 1101 denotes a line representing the boundary of the overlapping area, 1102 denotes an area composed of unfilled pixel blocks, and 1103 denotes an area composed of variable data pixel blocks in the overlapping area, Reference numeral 1104 denotes an area composed of pixel blocks of a single data obtained from the average value of surrounding pixels in the overlapping area.

  The pixels of the variable data pixel block 1103 are equally increased (or decreased) based on the value of the pixel data 1105 of the neighboring peripheral image and the value of the filled pixel data 1106 obtained as an average value. As an appropriate value, 8 pixels 1107 in the Y direction are determined.

<Embodiment 4>
In the first embodiment, as a camera unit constituting the network camera, the wide-angle camera unit 101 and the shooting angle-of-view camera unit 106 each have an independent lens group, a CCD, a CCD control unit, and an image processing unit. However, the present invention is not limited to this.

  In the present embodiment, a camera unit is configured by a set of imaging units (CCD, CCD control unit, image processing unit), and each image of a captured image is created by cutting out the angle of view to be captured from a high-resolution wide-angle image. An embodiment having such a configuration will be described.

  FIG. 12 is a block diagram showing a schematic configuration of the network camera according to the present embodiment.

  In FIG. 12, reference numeral 1201 denotes a wide-angle camera unit that is a camera unit for taking a wide-angle image. The wide-angle camera unit 1201 includes a lens group (1202), a CCD (1203), and a CCD control unit (1204). The image processing unit (1205) is included. These perform the same processing as the blocks described in FIGS. 1 and 102 to 105, but each block has a performance capable of high-resolution (UXGA) wide-angle imaging of wide-angle video, and an image processing unit. From 1205, YUV digital data is output at a frame rate of 30 frames / second with the number of pixels of UXGA size.

  A wide-angle distortion correction processing unit 1206 digitally corrects image distortion caused by the wide-angle lens of YUV digital data output from the image processing unit 1205, and the corrected image is used as a wide-angle image.

Reference numeral 1207 denotes a shooting angle-of-view cutout unit that performs a trimming process to cut out a part of each frame of the wide-angle video whose distortion has been corrected in 1206, and outputs the cut-out video as YUV digital data. Is. The clipped video is used as each video shot. The position, size, and shape to be cut out by the shooting angle-of-view cutout unit 1207 are panned remotely from the viewer side,
It is determined by tilt and zoom operations, and operation information is transmitted. In accordance with the received information, the CPU 1210 sets the extraction conditions of the shooting angle of view extraction unit.

  A still image encoding unit 1208, a moving image encoding unit 1209, a CPU 1210, a communication interface unit 1211, a ROM 1212, and a RAM 1213 connected to the bus 1214 are blocks that perform the same control as in FIGS.

<Embodiment 5>
In the fifth embodiment, when the position or size of the overlapping area is changed by the viewer, the updated overlapping information is transmitted to the network camera, and the network camera fills in the wide-angle video according to the received information. Although the control is performed in the sequence of updating the area and transmitting it to the viewer, the present invention is not limited to this.

  Usually, in remote operation using a network, a response to an operation request is often delayed due to a communication delay on the network. Also in the first embodiment, for example, when the superimposed position of each captured image is moved on the viewer side, the window of each captured image displayed on the viewer side is immediately moved in response to a user operation. However, since the update image from the network camera arrives with delay for the wide-angle video, the position before moving the window is displayed with the single-color filled wide-angle video remaining, The video becomes unsightly.

  Therefore, in the present embodiment, when a wide-angle video that has not been subjected to painting processing is transmitted in advance from the network camera and the overlay condition is changed by the viewer, until the updated wide-angle video is received. An embodiment that complements with a wide-angle video that has not been subjected to the painting process that has been transmitted first will be described.

  FIG. 13 is a flowchart showing details of wide-angle video processing for one frame in the network camera.

  In FIG. 13, step S1301 is a process for reading out one frame of wide-angle video. Similar to FIGS. 5 and 501, one frame of video at a frame rate of 30 frames / second is converted into YUV digital data in the image development area 116a of the RAM 116. read out.

  Next, in step S1302, it is checked whether or not the wide-angle image has a certain amount of movement. If there is no movement, in step S1303, the wide-angle image displayed on the viewer is displayed as in FIG. 5 and S502. A process of filling a region to be overlaid on the shooting angle-of-view image with a single color is performed. In step S1304, the MPEG4 encoding process by the moving image encoding unit 112 is executed and stored in the encoded data temporary storage area 116c of the RAM 116.

  If a motion is detected in S1302, the paint process in S1303 is not performed, and the process proceeds to the encoding process in S1304. Hereinafter, similarly to FIG. 5, S504, and S505, the stored encoded data is divided into transmission packet sizes in step S1305, a transmission header is added, and then in step S1306, the communication interface unit 114 sends the data to the viewer. Data transmission is performed.

  FIG. 14 is a flowchart showing details of the processing in the fifth embodiment in the viewer-side wide-angle frame processing described in the first embodiment, FIG. 6, S605, S606, and S607.

  In FIG. 16, A represents the connection from FIG. 6, S604, B represents the connection to S608, and C represents the connection to S611.

  In step S1401, it is checked whether the packet for one frame received from the communication interface in FIG. 6 or S604 is a wide-angle frame. If it is not a wide-angle frame, the process proceeds to B. If it is a wide-angle frame, in step S1402, the wide-angle frame data is decoded in the same manner as in FIG. 6 and S606.

Next, in step S1403, it is checked whether the received wide-angle frame is an unfilled video frame,
If the frame has not been filled, in step S1404, the frame image is stored in the display image development memory 209b, and the process proceeds to S1405. If there is the same frame data stored before, the update is overwritten. In the case of a frame for which the filling process has been performed in S1403, the process proceeds to S1405 without performing the storage process.

  In step S1405, it is checked whether or not the overlay change process is in progress, that is, whether or not the delay time from when the overlay change request is transmitted from the viewer to the network camera until the updated wide-angle frame is received is checked. If the change processing is not in progress, in step S1406, similarly to FIG. 6 and S607, a process of writing the wide-angle video data subjected to the fill processing to the display image development memory 209b of the RAM 209 is executed, and the C overlay composition processing is performed. Connected to.

  If the change process is being performed in step S1405, a process for complementing the filled area that has come to the front by the moving operation of the shooting angle-of-view video window with the pixel data in the wide-angle image stored in step S1404 in step S1407. Execute. After that, it is connected to the process of C.

  Next, FIG. 15 shows a processing sequence when the overlay change is performed between the network camera and the viewer in the present embodiment.

  In FIG. 15, 1501 represents processing on the network camera side, and 1502 represents processing on the viewer side. Here, wide-angle video data is transmitted from the network camera to the viewer at a period of 1/30 seconds. Here, reference numeral 1503 denotes a video frame that is being filled, and this frame data is normally transmitted continuously. Reference numeral 1504 denotes video frame data that has not been filled, and reference numeral 1505 denotes an updated video frame whose fill position has been changed.

  In FIG. 15, first, in the network camera, when a certain amount of motion is detected in the wide-angle video in 1506, switching from the filling process to the non-filling process is performed in 1507, and the next transmission frame is an unfilled video frame (1504. ) The viewer detects this frame and issues an update request for saved data (1508), and overwrites and saves the already saved wide-angle video data without update (1509).

Reference numeral 1510 denotes transmission of change information when the overlay condition is changed on the viewer side. In response to this, on the network camera side, 1511 corresponds to the transmitted overlay information.
A process of changing the fill position, size, etc. is executed, and from the next shooting frame that has been completed, it is transmitted as an overlay change frame (1505).

On the other hand, when the change information is transmitted in 1510, the overlay change event occurs in 1512 on the viewer side,
In response, for the wide-angle frame arrived while the overlay change frame (1505) arrives from the network camera, the unfilled wide-angle video data stored in 1509 is read out and brought to the front by the change operation. The process of complementing the filled area is executed (1513). In this figure, the time indicated by 1514 is the delay time on the network from the viewer operation until the change frame is received.

  In the present embodiment, the timing for transmitting unfilled video frame data is described in the embodiment in which the video frame data is transmitted only when there is a certain level of motion in the wide-angle video, but the motion detection on the network camera side is described. In order to reduce this processing load, the same effect can be obtained even if frame data transmission without filling is performed at a fixed interval of a fixed frame interval.

Further, although the non-filled frame data has been described in the embodiment in which it is transmitted by replacing it with normal filled frame data,
The unfilled frame data may be transmitted independently in a separate stream.

<Other embodiments>
The viewer described in the first to fifth embodiments has been described as a dedicated device including a liquid crystal display and a control key. However, in the present invention, an application of a PC connected to a network may be used as the viewer. Needless to say.

  In addition, MPEG4 has been described as an example of the moving image encoding method, but the present invention is not limited to this, and can be applied to all encoding methods using spatial or temporal redundancy such as motion JPEG, MPEG1, and MPEG2. is there.

Furthermore, in the embodiment, the wide-angle video data and each image data of the captured image are transmitted from the network camera and described as an application to a system in which the video is superimposed and displayed on the viewer.
For example, in a system such as a videophone or a video conference, a video transmission method from a communication partner is also used in a system that receives a video captured by the other party and displays the video captured by the other party on the screen. The present invention can be applied. Similarly, it goes without saying that the present invention can also be applied to an application that displays a graphic image (such as an icon or menu) created by its own system on a moving image transmitted from a communication partner.

It is a block diagram which shows schematic structure of the network camera by Embodiment 1 of this invention. It is a block diagram which shows schematic structure of the viewer by Embodiment 1 of this invention. It is a flowchart showing the main process of the network camera main body by Embodiment 1 of this invention. 5 is a flowchart showing details of processing for photographing angle-of-view video for one frame according to Embodiment 1 of the present invention. It is a flowchart showing the detail of the wide-angle video processing for 1 frame by Embodiment 1 of this invention. It is a flowchart showing the main process of the viewer main body by Embodiment 1 of this invention. It is a figure which shows the example of an image | video displayed with the image | video and the viewer image | photographed with the network camera by Embodiment 1 of this invention. It is a flowchart showing the detail of the wide-angle video process for 1 frame by Embodiment 2 of this invention. It is a figure showing the structure of a block in case the boundary of the superimposition area | region by Embodiment 2 of this invention corresponds with the boundary of a pixel block. It is a figure showing the structure of a pixel in case the boundary of the superimposition area | region by Embodiment 2 of this invention is a position which divides | segments a pixel block. It is a figure showing the structure of the block near the boundary of the superimposition area | region and periphery image | video by Embodiment 3 of this invention. It is a block diagram which shows schematic structure of the network camera by Embodiment 4 of this invention. It is a flowchart showing the detail of the wide-angle image processing for 1 frame by Embodiment 5 of this invention. It is a flowchart showing the detail of the wide angle frame process by the side of the viewer by Embodiment 5 of this invention. It is a processing sequence diagram when the overlay change is performed between the network camera and the viewer according to the fifth embodiment of the present invention.

Explanation of symbols

101 Wide-angle camera unit 102 Lens group 1
103 CCD1
104 CCD control unit 1
105 Image processing unit 1
106 Shooting angle of view camera unit 107 Lens group 2
108 CCD2
109 CCD control unit 2
110 Image processing unit 2
111 Still image encoding unit 112 Moving image encoding unit 113 CPU
114 Communication interface 115 ROM
116 RAM
117 Pan / tilt control motor 118 Bus

Claims (7)

  A camera capable of encoding captured image data and transmitting it to a remote location, and a viewer capable of decoding the encoded image data received from the camera and displaying it superimposed on other image data A network characterized in that the image data captured by the camera is encoded by overwriting the same area as the area overlaid with the other image data by the viewer with monochromatic data, and transmitted. Camera image transmission method. A first photographing means capable of photographing a wide-angle video and a second photographing means capable of photographing a part of the angle of view of the wide-angle video;
A camera capable of encoding and transmitting image data captured by each imaging unit, and a video captured by the first imaging unit among the encoded image data received from the camera, and captured by the second imaging unit And a viewer capable of displaying the recorded video in a superimposed manner, and the camera superimposes the image data captured by the second imaging unit on the image data captured by the first imaging unit. An image transmission method for a network camera, wherein the same region as the region to be combined is overwritten with monochromatic data, encoded, and transmitted.   2. The network camera image transmission method according to claim 1, wherein the monochrome data to be overwritten comprises a numerical value closest to the numerical value of the peripheral pixel data of the overwritten area.   4. The network camera image transmission method according to claim 3, wherein the numerical value closest to the peripheral pixel data is an average value of the numerical values of the peripheral pixel data.   Of the area to be overwritten, the block data closest to the boundary with the video data is composed of a numerical value that continuously increases or decreases between the numerical value to be overwritten and the numerical value of the pixel adjacent to the overwritten area. 2. The network camera image transmission system according to claim 1, wherein the image transmission system is a block. In control such that image data is continuously transmitted from the camera as a frame of a moving image, some frames of the image data are transmitted as frames in which monochrome data is not overwritten and are displayed in the viewer. When conditions such as the shape, position, size, etc. of the image to be superimposed have changed with respect to the transmitted video, the change information is transmitted from the viewer to the camera,
The camera performs control such that image data updated in the area to be overwritten in response to the change information is generated and transmitted to the viewer, and the viewer receives the updated image data from the time when the condition changes. 2. The network camera image transmission method according to claim 1, wherein control is performed so that the image data is complemented with image data of a frame in which monochrome data is not overwritten.   7. The network camera image transmission system according to claim 6, wherein when a certain amount of motion is detected in the captured video, the frame is transmitted as a frame in which monochrome data is not overwritten.

Images