JP6479062B2 - Data distribution system and data distribution method - Google Patents

Description

  The present invention relates to a data distribution system and a data distribution method for distributing data to a terminal device.

  A data distribution system that distributes image data from a transmission device to a terminal device is used. For example, in the image monitoring system disclosed in Patent Document 1, image data captured by a camera is directly transmitted from the camera to a terminal device (monitoring terminal), or image data is transmitted to a recorder or server for storage, and the server or the like is stored. The image data is displayed on the terminal device by transmitting the data to the terminal device.

JP 2012-209727 A

  By the way, in recent years, various image processing such as not only displaying image data received at a terminal device but also authenticating a person or detecting a suspicious person or a suspicious object has been widely performed. Depending on the purpose of image processing, it is not always necessary to have image data for the entire area of the captured image, and it may be sufficient to receive a specific image area. For example, when face authentication image processing is performed in a terminal device, it may be sufficient to receive even face area image data. In addition, when image processing for suspicious person detection is performed by a terminal device, it may be sufficient to receive even image data of a person area. However, since the conventional technology transmits image data of the entire area regardless of the purpose of image processing of the terminal device, the amount of communication on the camera side increases as the number of cameras and the number of terminal devices as distribution destinations increase. There is a problem that the communication line on the camera side is tight.

  In order to reduce the amount of communication on the camera side when transmitting image data to a plurality of terminal devices, the image data from the camera is temporarily transmitted to a server installed in a data center or the like and stored from the server. Image data can also be distributed to the terminal device. However, even in this case, since the image data of the entire area is transmitted from the server, another problem arises that the communication amount on the server side (data center side) increases and the communication line becomes tight.

  An object of the present invention is to provide a data distribution system and a data distribution method capable of reducing the amount of communication by distributing only packets including image data of an image area required by a distribution destination terminal device. .

  In order to solve this problem, the present invention provides a data distribution system including a data transmission device, an image analysis device, and a distribution control device. In such a data distribution system, the data transmission device identifies the position in the captured image of the partial image from the information of the payload portion of the packet related to the partial image in the captured image, and the position information regarding the identified position in the header portion of the packet In addition to generating an extended packet by including the extended packet, the extended packet is transmitted to the distribution control device, and the image analysis device performs image analysis on the captured image to indicate the position of interest in the captured image. And the position of interest information is transmitted to the distribution control device. The distribution control device receives the extension packet received from the data transmission device and the position of interest information transmitted from the image analysis device. The received position information receiving part and the position related to the position information included in the header part of the received extended packet are received. It has been a specifying unit for specifying an extended packet corresponding to the position of the point of interest information, and the packet distribution part for distributing packets based on specified extension packet to the terminal device.

  In the data distribution system according to the present invention, the position-of-interest information further indicates an object of interest shown in the position indicated by the position-of-interest information, and the distribution control device includes a plurality of pieces of identification information of a plurality of terminal devices. A storage unit that stores interest information associated with any one of the interests of interest, and the specifying unit receives the position of interest received by the interest target related to the stored interest information. Preferably, the terminal device corresponding to the object of interest related to the information is specified, and the packet distribution unit distributes the packet based on the specified extended packet to the specified terminal device.

  Further, in the data distribution system according to the present invention, the position of the partial image is described in the payload portion of the packet, and the data transmitting device reads the position described in the payload portion of the packet, thereby It is preferable to specify the position in the captured image.

  In the data distribution system according to the present invention, it is preferable that the data transmission device restores a captured image from a plurality of packets and specifies a position in the captured image of the partial image based on the restored captured image.

  In the data distribution system according to the present invention, it is preferable that the data transmitting apparatus encrypts the payload portion in the extension packet by an encryption method that can be decrypted by the terminal apparatus.

  In the data distribution system according to the present invention, the data is data related to a monitoring target guarded by the security device, and the distribution control device is in a security state indicating whether the security device is guarding the monitoring target. It is preferable that the information processing apparatus further includes a storage unit that stores information, and the packet distribution unit distributes a packet based on the specified extended packet only when the security device is guarding the monitoring target.

  In order to solve the problem, the present invention provides a data distribution method in a data distribution system including a data transmission device, an image analysis device, and a distribution control device. In such a data distribution method, the data transmission device specifies the position in the captured image of the partial image from the information of the payload portion of the packet related to the partial image in the captured image, and the position information related to the specified position in the header portion of the packet. In addition to generating an extended packet by including it, the extended packet is transmitted to the distribution control device, and the image analysis device performs image analysis of the captured image, thereby indicating the position of interest in the captured image. The position of interest information is transmitted to the distribution control device, the distribution control device receives the extended packet transmitted from the data transmission device, and receives the position of interest information transmitted from the image analysis device Among the extended packets, the position related to the position information included in the header portion identifies the extended packet corresponding to the position related to the received position information of interest. Comprises delivering packets based on specified extended packet to the terminal device.

  The data distribution system and data distribution method according to the present invention can reduce the amount of communication.

It is a schematic structure figure of data distribution system 1 by this embodiment. 2 is a diagram illustrating a schematic configuration of a data transmission device 30. FIG. 2 is a diagram showing a schematic configuration of a distribution control device 40. FIG. 2 is a diagram illustrating a schematic configuration of an image analysis device 60. FIG. It is a whole operation | movement sequence diagram of the data delivery system 1 which concerns on this embodiment. (A)-(c) is a schematic diagram which shows an example of the data structure of an image packet, (d) is a schematic diagram which shows an example of the data structure of image sequence format information, (e) is image format information It is a schematic diagram which shows an example of a data structure. It is a flowchart which shows the example of operation | movement of an extended packet production | generation process. (A) is a schematic diagram which shows an example of the data structure of the extended packet production | generation information 331, (b)-(d) is a schematic diagram which shows an example of the data structure of an extended packet, (e) is a partial image. It is a schematic diagram which shows an example of the data structure of area | region information. It is a flowchart which shows the example of operation | movement of an image analysis process. It is a schematic diagram which shows an example of the data structure of a region-of-interest packet. (A) is a schematic diagram which shows an example of the data structure of the delivery destination request | requirement information 421, (b) is a schematic diagram which shows an example of the data structure of the delivery destination information 422, (c)-(e) is delivery It is a schematic diagram which shows an example of the data structure of a use packet. It is a flowchart which shows the example of operation | movement of a terminal device specific process.

  The data distribution system of the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing a schematic configuration of a data distribution system 1 according to the present embodiment. As shown in FIG. 1, the data distribution system 1 includes an image transmission device 10, a security device 20, a data transmission device 30, a distribution control device 40, a plurality of terminal devices 50, an image analysis device 60, and the like. The image transmission device 10 and the security device 20 are arranged in the security area 2. The image transmission device 10 and the data transmission device 30 are connected to the network 3 for communication and communicate via the network 3. The network 3 is a wired network such as the Internet. The network 3 may be a wireless network such as a wireless LAN (Local Area Network), WiMAX (Worldwide Interoperability for Microwave Access), or PHS (Personal Handy-phone System).
The distribution control device 40 is an in-network device of the communication carrier network 4. The security device 20, the data transmission device 30, the distribution control device 40, the plurality of terminal devices 50, and the image analysis device 60 are each connected to the communication carrier network 4 and distributed in the communication carrier network 4. It communicates with the control device 40. The communication carrier network 4 is a network according to a communication method such as W-CDMA (Wideband Code Division Multiple Access), CDMA2000, or LTE (Long Term Evolution). The communication carrier network 4 may be a wired network such as an intranet.

The image transmission device 10 is mounted on a network camera or the like, for example, and packetizes a captured image captured by the network camera. For example, each image transmission device 10 is arranged in the security area 2 so that the network camera captures an image in the security area 2, and captures an image of a monitoring target such as an intruder or a suspicious object in the security area 2. Get an image. The image transmission apparatus 10 uses the acquired captured image as an H.264 image. H.264 / MPEG (Moving Picture Experts Group) -4 Image encoded according to a predetermined encoding standard such as AVC (Advanced Video Coding) standard, and encoded into image data having attributes such as a predetermined resolution, image quality, and frame rate An image packet related to the data is generated. The image transmitting apparatus 10 divides the captured image into a plurality of partial images for each predetermined unit, and encodes each partial image to generate an image packet. For example, in the H.264 / MPEG-4 AVC standard, the predetermined unit is a slice. The image transmission device 10 includes a communication unit, and transmits the generated image packet to the data transmission device 30 via the communication unit.
The data distribution system 1 may include a plurality of image transmission devices 10, and each image transmission device 10 may transmit an image packet to the data transmission device 30.

  The security device 20 is installed in the security area 2 such as a house or a dwelling unit of a condominium, and monitors and guards monitoring targets such as an intruder entering the security area 2 and a suspicious object existing in the security area 2. One or more sensors for detecting an abnormality in the security area 2 are connected to the security device 20. The sensor is, for example, an infrared sensor that detects the presence or absence of a monitoring target in the detection range based on a change in the amount of received infrared light, and a sensor in the imaging range based on a difference signal between a captured image and a normal image stored in advance. An image sensor or the like that detects the presence or absence of a monitoring target. In the security device 20, “security” or “non-security” is set. The security device 20 monitors whether or not there is a monitoring target in the security area 2 while “security” is set. When the monitoring target is detected in the security area 2, the security device 20 passes through the distribution control device 40. An abnormality report indicating that the monitoring target has been detected is sent to the terminal device 50.

  The terminal device 50 is a server arranged in a data center for long-term storage that stores a captured image transmitted by the image transmission device 10 and a monitoring control for monitoring a captured image transmitted by the image transmission device 10 by a controller. It is a personal computer arranged at the center, or an authentication device for authenticating a person based on the face of the person shown in the captured image. Each terminal device 50 includes a communication unit and a display unit, receives a packet based on the image packet transmitted by the image transmission device 10 from the distribution control device 40 via the communication unit, and captures a captured image from the received packet. Restore and display on the display.

  FIG. 2 is a diagram illustrating a schematic configuration of the data transmission device 30. The data transmission device 30 receives the image packet from the image transmission device 10, generates an extension packet from the received image packet, and transmits the extended packet to the distribution control device 40. The data transmission device 30 is mounted on, for example, a set top box that can execute data processing according to a predetermined application program. The data transmission device 30 and the image transmission device 10 may be implemented as one device. As shown in FIG. 2, the data transmission device 30 includes a first communication unit 31, a second communication unit 32, a storage unit 33, a control unit 34, and the like.

  The first communication unit 31 includes a communication interface circuit for allowing the data transmission device 30 to communicate with the network 3 according to a communication method such as TCP / IP. The first communication unit 31 sends data received from the image transmission device 10 via the network 3 to the control unit 34. Further, the first communication unit 31 transmits the data received from the control unit 34 to the image transmission device 10 via the network 3.

  The second communication unit 32 includes a communication interface circuit for allowing the data transmission device 30 to perform wireless communication with the communication carrier network 4 and a mobile phone in accordance with a communication method such as W-CDMA, CDMA2000, or LTE. The second communication unit 32 sends data received from the distribution control device 40 via the communication carrier network 4 to the control unit 34. Further, the second communication unit 32 transmits the data received from the control unit 34 to the distribution control device 40 via the communication carrier network 4.

  The storage unit 33 includes a semiconductor memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory), or a magnetic recording medium and its access device or an optical recording medium and its access device. The storage unit 33 stores a computer program and various data for controlling the data transmission device 30, and inputs / outputs such information to / from the control unit 34. The computer program is installed in the storage unit 33 from a computer-readable portable recording medium such as a CD-ROM (compact disk read only memory) or a DVD-ROM (digital versatile disk read only memory) using a known setup program. May be. The storage unit 33 stores extended packet generation information 331 and the like. Details of the extended packet generation information 331 will be described later.

The control unit 34 includes a CPU (Central Processing Unit), an MPU (Micro Processing Unit), peripheral circuits, and the like. The control unit 34 includes an extended packet generation unit 341, an extended packet transmission unit 342, and the like.
The extended packet generation unit 341 determines the position of the partial image in the captured image from the information of the payload portion of the image packet related to the partial image in the captured image received from the image transmission device 10 via the first communication unit 31. Identify. The extended packet generator 341 generates an extended packet by including partial image position information, which is position information regarding the specified position, in the header portion of the image packet.
The extended packet transmitter 342 transmits the extended packet generated by the extended packet generator 341 to the distribution control device 40 via the second communication unit 32.

  FIG. 3 is a diagram illustrating a schematic configuration of the distribution control device 40. The distribution control device 40 receives the extension packet from the data transmission device 30, and generates a distribution packet based on the received extension packet. Also, the distribution control device 40 receives from the image analysis device 60 a position in which the user of each terminal device 50 is interested in the captured image, and the object of interest reflected in the position. The region-of-interest packet including the position-of-interest information indicating is received. The object of interest is, for example, a person or a part of an object (such as a person's face) or the entire object. Note that the target of interest may be a part or all of a specific person or object such as an intruder or suspicious object that enters the security area 2. The distribution control device 40 identifies the terminal device 50 to which the distribution packet is to be transmitted based on the interest position information included in the received region-of-interest packet, and transmits the distribution packet to the identified terminal device 50. The distribution control device 40 is mounted on, for example, a router that can execute data processing according to a predetermined application program. As illustrated in FIG. 3, the distribution control device 40 includes a communication unit 41, a storage unit 42, a control unit 43, and the like.

  The communication unit 41 includes a communication interface circuit for the distribution control device 40 to communicate with the security device 20, the data transmission device 30, the terminal device 50, and the image analysis device 60 via the communication carrier network 4. The communication unit 41 sends data received from the security device 20, the data transmission device 30, the terminal device 50, or the image analysis device 60 to the control unit 43. The communication unit 41 transmits the data received from the control unit 43 to the security device 20, the data transmission device 30, the terminal device 50, or the image analysis device 60.

The storage unit 42 includes a semiconductor memory such as a ROM and a RAM, or a magnetic recording medium and its access device or an optical recording medium and its access device. The storage unit 42 stores a computer program and various data for controlling the distribution control device 40, and inputs / outputs such information to / from the control unit 43. The computer program may be installed in the storage unit 42 using a known setup program or the like from a computer-readable portable recording medium such as a CD-ROM or DVD-ROM.
The storage unit 42 stores distribution destination request information 421, distribution destination information 422, and the like. In the distribution destination request information 421, identification information of each terminal device 50 and request target information indicating a target of interest that a user of each terminal device 50 requests to browse among a plurality of targets of interest are stored in association with each other. The The identification information of each terminal device 50 is, for example, the IP address and port number of each terminal device 50 (hereinafter may be referred to as a distribution destination terminal address). The request target information is an example of interest target information. The delivery destination information 422 stores the delivery destination terminal address of the terminal device 50 to which the extended packet is to be delivered. Details of the distribution destination request information 421 and the distribution destination information 422 will be described later.

The control unit 43 includes a CPU, an MPU, a peripheral circuit, and the like. The control unit 43 includes an extended packet reception unit 431, a position of interest information reception unit 432, a distribution destination information update unit 433, a specification unit 434, a distribution packet generation unit 435, a packet distribution unit 436, and the like.
The extension packet receiving unit 431 receives the extension packet transmitted from the data transmission device 30 via the communication unit 41.
The region-of-interest information receiving unit 432 receives the region-of-interest packet including the region-of-interest information transmitted from the image analysis device 60 via the communication unit 41.
The distribution destination information update unit 433 updates the distribution destination information 422 based on the distribution destination request information 421 and the interest position information included in the region-of-interest packet transmitted from the image analysis device 60.
The specifying unit 434 has a position related to the partial image position information included in the header portion of the extended packet received by the extended packet receiving unit 431 corresponding to the position related to the interested position information received by the interested position information receiving unit 432. Identify extended packets. Further, the specifying unit 434 specifies the terminal device 50 in which the interest target related to the request target information stored in the distribution destination request information 421 corresponds to the target of interest related to the interested position information received by the interested position information receiving unit 432. To do.
The distribution packet generator 435 generates a distribution packet based on the extended packet received by the extended packet receiver 431.
The packet distribution unit 436 uses the communication unit 41 to identify the distribution packet based on the extended packet identified by the identification unit 434 among the distribution packets generated by the distribution packet generation unit 435. Delivered to the terminal device 50.

  FIG. 4 is a diagram illustrating a schematic configuration of the image analysis device 60. The image analysis device 60 performs image analysis on the captured image to detect a position where the object of interest appears in the captured image, and generates interest position information indicating the detected position and the object of interest reflected in the position. Then, the region-of-interest packet including the generated position-of-interest information is transmitted to the distribution control device 40. The distribution control device 40 and the image analysis device 60 may be implemented as one device. As shown in FIG. 4, the image analysis device 60 includes a communication unit 61, a storage unit 62, a control unit 63, and the like.

  The communication unit 61 includes a communication interface circuit that allows the image analysis device 60 to perform wireless communication with a communication carrier network 4 and a mobile phone in accordance with a communication method such as W-CDMA, CDMA2000, or LTE. The communication unit 61 sends data received from the distribution control device 40 via the communication carrier network 4 to the control unit 63. In addition, the communication unit 61 transmits the data received from the control unit 63 to the distribution control device 40 via the communication carrier network 4.

  The storage unit 62 includes a semiconductor memory such as a ROM and a RAM, or a magnetic recording medium and its access device or an optical recording medium and its access device. The storage unit 62 stores a computer program and various data for controlling the image analysis device 60, and inputs / outputs such information to / from the control unit 63. The computer program may be installed in the storage unit 62 using a known setup program or the like from a computer-readable portable recording medium such as a CD-ROM or DVD-ROM.

The control unit 63 includes a CPU, an MPU, a peripheral circuit, and the like. The control unit 63 includes a position-of-interest information generation unit 631, a position-of-interest information transmission unit 632, and the like.
The position-of-interest information generation unit 631 receives the extension packet from the distribution control device 40, and restores the captured image from the received extension packet. The position-of-interest information generation unit 631 detects a target of interest from the restored captured image using a known image processing technique, and identifies a position where the target of interest is shown in the captured image. Then, the position-of-interest information generation unit 631 generates position-of-interest information indicating the identified position (coordinates indicating the range) and the identifier of the object of interest reflected in the position, and generates a region-of-interest packet including the position-of-interest information. Generate.
The region-of-interest information transmission unit 632 transmits a region-of-interest packet including the region-of-interest information generated by the region-of-interest information generation unit 631 to the distribution control device 40 via the communication unit 61.

FIG. 5 is an overall operation sequence diagram of the data distribution system 1 according to the present embodiment. The operation sequence described below is executed mainly by the control unit of each device in cooperation with each element of each device based on a program stored in advance in the storage unit of each device included in the data distribution system 1. Is done.
First, each image transmission device 10 acquires a captured image captured by a network camera, encodes a partial image in the acquired captured image, and generates an image packet related to the encoded partial image (step S101).

6A to 6C are schematic diagrams illustrating an example of the data structure of an image packet. As shown in FIGS. 6A to 6C, the image packet has a header portion and a payload portion.
The header part has an image transmission device ID, a data transmission device address, and the like. The image transmission device ID is a unique value that can uniquely identify the image transmission device 10. The data transmission device address is an address of the data transmission device 30, for example, an IP address and a port number of the data transmission device 30. The configuration of the image packet is classified into three types according to the configuration of the payload portion.

FIG. 6A is an example of an image packet in which image sequence format information indicating common format information such as resolution, image quality, or frame rate is included in the payload portion for time-sequential image sequences.
FIG. 6D is a schematic diagram illustrating an example of the data structure of the image sequence format information. As shown in FIG. 6D, the image sequence format information includes resolution information indicating the resolution of the image, image quality information indicating the encoding quality (quantization parameter, etc.) of the image, or the number of images included in one second. And frame rate information indicating (number of frames).
The image sequence format information is common format information necessary for decoding and displaying an image sequence (images of a plurality of frames) continuous in time series. For example, in the H.264 / MPEG-4 AVC standard, SPS (Sequence Parameter Set), PPS (Picture Parameter Set), etc. are set as image sequence format information.
Note that SPS frame_mbs_only_flag, pic_width_in_mbs_minus1, pic_height_in_map_units_minus1, etc. are set as resolution information. The frame_mbs_only_flag is set to 1 when only a frame / macroblock is included in a picture, that is, not interlaced, and set to 0 otherwise. pic_width_in_mbs_minus1 is (the number of macroblocks in the horizontal direction of the picture−1), and pic_height_in_map_units_minus1 is (the number of macroblocks in the vertical direction of the picture−1). When the number of pixels in the horizontal and vertical directions of the macroblock is 16, the number of pixels in the horizontal direction and the number of pixels in the vertical direction are calculated by the following equations.
Number of pixels in the horizontal direction = (pic_width_in_mbs_minus1 + 1) x 16
Number of pixels in the vertical direction = (2−frame_mbs_only_flag) × (pic_height_in_map_units_minus1 + 1) × 16
The image sequence format information may include information other than the information described above.

FIG. 6B shows, for the partial image data in a predetermined unit that is a part of the image data, the image format information indicating the format information such as the encoding type and a part of the partial image data are included in the payload portion. FIG.
FIG. 6E is a schematic diagram illustrating an example of the data structure of the image format information. As shown in FIG. 6E, the image format information includes a frame number, a head macroblock number, and the like. The frame number is the number of the frame, and the head macroblock number is the number of the head macroblock included in the frame. The image format information is format information necessary for decoding and displaying a predetermined unit of image data such as the format of an encoded image. For example, in the H.264 / MPEG-4 AVC standard, a slice header or the like is set as image format information. Also, frame_num of a slice header (slice header) is set as the frame number, and first_mb_in_slice of the slice header is set as the top macroblock number. The image format information may include information other than the information described above.
When the partial image data of a predetermined unit exceeds the size of one packet, the partial image data is further divided into a plurality of image data and stored in a plurality of packets. The image format information is stored at the beginning of the partial image data.

  FIG. 6C is an example of an image packet in which only a part of the partial image data is included in the payload portion.

  Next, in step S102 of FIG. 5, the image transmission apparatus 10 transmits the generated image packet to the data transmission apparatus 30 via the communication unit (step S102). Next, the extended packet generation unit 341 of the data transmission device 30 receives an image packet from each image transmission device 10 via the first communication unit 31 (step S103). Next, the extended packet generation unit 341 executes extended packet generation processing for each received image packet (step S104).

  FIG. 7 is a flowchart illustrating an example of the operation of the extended packet generation process. The extended packet generation process shown in FIG. 7 is executed every time an image packet is received. Further, the extended packet generation unit 341 does not execute the extended packet generation process for the image packet received from each image transmission device 10 until the image packet including the image sequence format information is first received. That is, the extended packet generation process shown in FIG. 7 is executed from an image packet including image sequence format information.

  First, the extended packet generation unit 341 determines whether image sequence format information is included in the received image packet (step S201). When the received image packet includes image sequence format information (Yes in step S201), the extended packet generation unit 341 stores (initializes) the extended packet generation information 331 based on the image sequence format information (step S201). S202).

  FIG. 8A is a schematic diagram illustrating an example of the data structure of the extended packet generation information 331. As shown in FIG. 8A, as the extended packet generation information 331, the image transmission device ID, the image sequence format information, and the image format information are received from each image transmission device 10 for each continuous image sequence in time series. Each information such as a frame ID and partial image position information is stored in association with each other.

The image transmission device ID is an image transmission device ID included in the image packet.
The image sequence format information is image sequence format information included in the image packet.
The image format information is image format information included in the image packet.
The frame ID is a unique value for each frame that can uniquely identify each frame. As the frame ID, for example, a value that increases by one for each frame is assigned. Alternatively, the time when the corresponding image is captured may be assigned as the frame ID.

The partial image position information is position information indicating the position (coordinates) in the captured image of the partial image related to the partial image data included in the payload portion of the image packet. For example, when the partial image is a rectangular region, the X and Y coordinates of the upper left vertex and the X and Y coordinates of the lower right vertex of the partial image are set as the partial image position information. Further, when the partial image corresponds to a plurality of macroblocks (fixed size rectangular pixel set) of the H.264 / MPEG-4 AVC standard, the above coordinates are set for each macroblock as partial image position information. The
The partial image position information is calculated using, for example, the first macroblock number (first_mb_in_slice) in the image format information of the image packet. The extended packet generation unit 341 stores the first macroblock number in the image format information of the received image packet. The extended packet generator 341 subtracts the difference obtained by subtracting the head macroblock number in the image format information of the image packet received last time from the head macroblock number in the image format information of the image packet for one slice received this time as partial image data. Calculated as the total number of macroblocks included (macroblock_count_in_slice). The extended packet generation unit 341 may accumulate image packets for one slice, scan the accumulated image packets, and count the total number of macroblocks included in one slice.
The extended packet generation unit 341 includes the X coordinate left_top_x_i and the Y coordinate left_top_y_i of the upper left vertex and the X coordinate right_bottom_x_i of the lower right vertex of each macroblock whose macroblock number is i {i = first_mb_in_slice, ..., first_mb_in_slice + macroblock_count_in_slice-1} And Y coordinate right_bottom_y_i is calculated by the following formula.
left_top_x_i = (i% (pic_width_in_mbs_minus1 + 1)) × 16
left_top_y_i = [(i ÷ (pic_width_in_mbs_minus1 + 1))] × 16
right_bottom_x_i = left_top_x_i + 16
right_bottom_y_i = left_top_y_i + 16
In the above formula, the horizontal and vertical sizes of the macroblock are each assumed to be 16 pixels. [%] Is a calculation symbol for calculating the remainder. pic_width_in_mbs_minus1 is included in the resolution information of the image sequence format information and indicates the number of macroblocks in the horizontal direction of the picture minus one.
Further, the number of each macro block may be set as the partial image position information.

As described above, the position in the captured image of the partial image included in the image packet is described in the payload portion of the image packet, and the extended packet generation unit 341 includes the partial image described in the payload portion of the image packet. The position in the captured image of the partial image is specified by reading the position in the captured image.
Note that the extended packet generation unit 341 may specify the position of the partial image in the captured image by analyzing the payload portion of the image packet. For example, the image data relating to the captured image is RAW data in which each pixel based on RGB values is arranged so that the vertical resolution with respect to the horizontal resolution has a predetermined ratio, and the start data at the head of each frame Will be described, and stop data is included at the end of each frame. The extended packet generator 341 detects start data and stop data from a plurality of image packets, specifies the resolution of the captured image from the number of pixels between the start data and stop data, and restores the captured image. Then, the extended packet generation unit 341 specifies a position corresponding to each partial image included in the payload portion in the restored captured image.
Thereby, the extended packet generation unit 341 can specify the position in the captured image of the partial image included in the image packet even for the image packet in which the position in the captured image of the partial image is not described in the payload portion. .

  In step S202, when the received image packet includes image sequence format information, the extended packet generation unit 341 sets the image transmission device ID and the image sequence format information included in the image packet in the extended packet generation information 331. Then, initialize.

  Next, in step S203 of FIG. 7, the extended packet generator 341 generates an extended packet based on the extended packet generation information 331 (step S203), and the series of steps is terminated.

8B to 8D are schematic diagrams illustrating an example of the data structure of the extension packet. As shown in FIGS. 8B to 8D, the extension packet has the same header part and payload part as the image packet.
However, the header portion has a distribution control device address instead of the data transmission device address, and further has partial image area information. The distribution control device address is an address of the distribution control device 40, for example, an IP address and a port number of the distribution control device 40. The configuration of the extended packet is classified into three types according to the configuration of the payload portion, similarly to the configuration of the image packet.
FIG. 8E is a schematic diagram illustrating an example of the data structure of the partial image area information. As shown in FIG. 8E, the partial image area information includes information such as a frame ID and partial image position information. As the frame ID and partial image position information, the frame ID and partial image position information stored in the extended packet generation information 331 are set, respectively.

  When the extended packet generation unit 341 receives an image packet including image sequence format information (FIG. 6A), the extended packet generation unit 341 generates an extended packet including image sequence format information (FIG. 8B). Further, when receiving the image packet including the image format information (FIG. 6B), the extended packet generation unit 341 generates the extended packet including the image format information (FIG. 8C). On the other hand, when the extended packet generation unit 341 receives the image sequence format information and the image packet not including the image format information (FIG. 6C), the extended packet generation unit 341 does not include the image sequence format information and the image format information (FIG. 8). (D)) is generated.

  The extended packet generation unit 341 may encrypt the payload part in the extended packet by an encryption method that can be decrypted by the terminal device 50 or the image analysis device 60. Even if the distribution control device 40 cannot refer to the payload portion of the extension packet, the distribution control device 40 can specify the distribution destination terminal device 50 based only on the header portion. On the other hand, by encrypting the payload portion of the extension packet, the distribution control device 40 cannot access the image data included in the extension packet, and the administrator of the distribution control device 40 can download the image data included in the extension packet. Since the contents cannot be browsed, it is possible to improve security and protect privacy.

  In step S201 in FIG. 7, when the received image packet does not include image sequence format information (No in step S201), the extended packet generation unit 341 determines whether or not the received image packet includes image format information. Determination is made (step S204). When the received image packet includes image format information (Yes in step S204), the extended packet generation unit 341 updates the extended packet generation information 331 based on the image format information (step S205).

In step S205, the extended packet generation unit 341 determines whether or not the received image packet is an image packet corresponding to the head of the frame. If the received image packet is an image packet corresponding to the head of the frame, the extended packet generation information The frame number 331 is incremented (increased by 1). The extended packet generation unit 341 specifies the frame number from the frame type included in the image format information, and receives the received packet depending on whether the specified frame number is different from the frame number included in the previously received image format information. It is determined whether the image packet is an image packet corresponding to the head of the frame.
The extended packet generation unit 341 sets the image format information included in the image packet in the extended packet generation information 331. Also, the extended packet generation unit 341 calculates partial image position information from the top macroblock number included in the image format information, and sets the information in the extended packet generation information 331.

  Next, the extended packet generation unit 341 generates an extended packet based on the extended packet generation information 331 (step S203) and ends a series of steps.

  If no image format information is included in the received image packet in step S204 (No in step S204), the extended packet generator 341 generates an extended packet based on the extended packet generation information 331 (step S203). , To end a series of steps.

  Next, in step S105 of FIG. 5, the extended packet transmitter 342 transmits the extended packet generated by the extended packet generator 341 to the distribution control device 40 via the second communication unit 32 (step S105). Next, the extended packet receiving unit 431 of the distribution control device 40 receives the extended packet from the data transmitting device 30 via the communication unit 41 (step S106). Next, the extended packet receiving unit 431 transmits the received extended packet to the image analysis device 60 via the communication unit 41 (step S107). Next, the position-of-interest information generation unit 631 of the image analysis device 60 receives the extended packet from the distribution control device 40 via the communication unit 61 (step S108). Next, the position-of-interest information generation unit 631 executes image analysis processing of the captured image based on the received extension packet (step S109).

  FIG. 9 is a flowchart illustrating an example of the operation of the image analysis process. The interested position information generation unit 631 can store the received extension packet in the storage unit 62 in association with each extension packet having the same frame ID as the image transmission device ID, and can decode it as one image. At that time, the image analysis processing shown in FIG. 9 is executed.

First, the position-of-interest information generation unit 631 restores the captured image by decoding the payload portion of the received extension packet. The interested position information generation unit 631 detects a target of interest such as an intruding person or a suspicious object from the restored captured image using a known image processing technique (step S301).
For example, the position-of-interest information generation unit 631 detects an object of interest using machine learning technology. The position-of-interest information generation unit 631 uses a plurality of sample images in which the object of interest is captured and a plurality of sample images in which the object of interest is not captured, and the object of interest is captured with respect to the image in which the object of interest is captured. The discriminator is trained in advance to output an identification result indicating that the object of interest is not captured for an image in which the object of interest is not captured. The position-of-interest information generation unit 631 cuts out a region of a predetermined size in the reconstructed captured image while shifting the position, inputs the region to the classifier, and identifies whether the region of interest is captured in the region By obtaining the result, the object of interest is detected.
Alternatively, the position-of-interest information generation unit 631 may detect the object of interest using a pattern matching technique. The position-of-interest information generation unit 631 stores a plurality of image patterns in which the object of interest is captured in the storage unit 62 in advance. The position-of-interest information generation unit 631 cuts out a region of a predetermined size in the reconstructed captured image while shifting the position, and obtains the degree of similarity of the pattern of the image stored in the storage unit 62. The object of interest is detected based on whether the degree of similarity is equal to or greater than a threshold value. The degree of similarity can be a normalized cross-correlation value, for example.

  Next, when the interested position information generating unit 631 detects the target of interest from the captured image, the interested position information generating unit 631 generates the interested position information for each detected target of interest and generates a region of interest packet including the interested position information (step S302). ), A series of steps is completed.

FIG. 10 is a schematic diagram illustrating an example of a data structure of a region-of-interest packet. As shown in FIG. 10, the region-of-interest packet has a distribution control device address, an image transmission device ID, a frame ID, and interest position information.
As the distribution control device address, the image transmission device ID, and the frame ID, the distribution control device address, the image transmission device ID, and the frame ID included in the extended packet in which the object of interest is detected are set.
The position-of-interest information includes corresponding position information and corresponding object information. The corresponding position information indicates the position (coordinates) of an area (area having a specific meaning) in which the object of interest is shown in the captured image. Correspondence target information is an identifier (ID) indicating an object of interest shown in the position indicated by the corresponding position information. For example, when the object of interest is the entire image, 1 is set as the corresponding target information, when the target of interest is a person's face, 2 is set as the corresponding target information, and when the target of interest is a suspicious object, the corresponding target 3 is set as information.

  In step S110 of FIG. 5, the position-of-interest information transmission unit 632 transmits a region-of-interest packet including the position-of-interest information generated by the position-of-interest information generation unit 631 to the distribution control device 40 via the communication unit 61 (step S110). . Next, the position-of-interest information reception unit 432 of the distribution control device 40 receives a region-of-interest packet including the position-of-interest information from the image analysis device 60 via the communication unit 41 (step S111). Next, the distribution destination information update unit 433 includes a storage unit based on the position of interest information included in the region of interest packet received by the position of interest information reception unit 432 and the distribution destination request information 421 stored in the storage unit 42. The delivery destination information 422 stored in 42 is updated (step S112).

  FIG. 11A is a schematic diagram illustrating an example of the data structure of the distribution destination request information 421. As shown in FIG. 11A, as the delivery destination request information 421, for each item corresponding to the terminal device 50 that is the delivery destination of the delivery packet, the image transmission device ID, the delivery destination terminal address, the request target information, etc. Each information is stored in association with each other. The distribution destination request information 421 is set in advance by the administrator using an input device (not shown). Alternatively, the distribution destination request information 421 may be set according to a request received from each terminal device 50 from a user of each image transmission device 10.

The image transmission device ID is an image transmission device ID of the image transmission device 10 that transmits an image packet corresponding to a distribution packet to be received by the terminal device 50.
The distribution destination terminal address is the address of the terminal device 50 that is the distribution destination of the distribution packet, and is, for example, the IP address and port number of the terminal device 50.
The request target information is an identifier (ID) indicating the target of interest for which the user of each terminal device 50 is interested and for which a captured image is requested to be received and displayed at each terminal device 50. The request target information is set according to the same definition as the corresponding target information.

  FIG. 11B is a schematic diagram illustrating an example of a data structure of the distribution destination information 422. As shown in FIG. 11B, as the delivery destination information 422, for each item corresponding to the terminal device 50 that is the delivery destination of the delivery packet, the image transmission device ID, the delivery destination terminal address, the frame ID, and the position of interest information. Are stored in association with each other.

As the image transmission device ID, the frame ID, and the position of interest information, the image transmission device ID, the frame ID, and the position of interest information included in the region of interest packet are set.
The distribution destination terminal address is the address of the terminal device 50 that is the distribution destination of the distribution packet, and is, for example, the IP address and port number of the terminal device 50.

  In step S112, the distribution destination information update unit 433 first extracts the image transmission device ID and the correspondence target information from the region-of-interest packet received by the position-of-interest information reception unit 432. Then, it is determined whether or not an item that is a combination of the image transmission device ID and the request target information that corresponds to (matches) the combination of the extracted image transmission device ID and the corresponding target information is stored in the distribution destination request information 421. . If the corresponding item is stored, a new item is added to the delivery destination information 422. Here, the image transmission device ID, the frame ID, and the position of interest information that are newly added to the distribution destination information 422 are the image transmission device ID, the frame ID, and the position of interest information included in the region of interest packet, respectively. Further, the distribution destination terminal address of the item newly added to the distribution destination information 422 is the distribution destination terminal address of the item corresponding to the image transmission apparatus ID extracted from the region of interest packet and the corresponding target information in the distribution destination request information 421. is there. If an item that is a combination of the image transmission device ID and the request target information corresponding to the combination of the image transmission device ID and the corresponding target information extracted from the region-of-interest packet is not stored in the distribution destination request information 421, the distribution destination The information update unit 433 considers that there is no terminal device 40 that requires the region of interest indicated by the corresponding target information, and does not update the distribution destination information 422.

  Next, in step S113 of FIG. 5, the specifying unit 434 executes terminal device specifying processing in order to specify a packet delivery destination based on the received extended packet (step S113).

  FIG. 12 is a flowchart illustrating an example of the operation of the terminal device specifying process. The terminal device specifying process shown in FIG. 12 is executed every time an extended packet is received.

First, the specifying unit 434 has an item in which the image transmission device ID included in the received extension packet corresponds to (matches) any of the image transmission device IDs stored as the distribution destination information 422 in the storage unit 42. It is determined whether or not it exists (step S401).
When the image transmission device ID included in the extended packet does not correspond to any image transmission device ID in the item stored as the distribution destination information 422, the specifying unit 434 displays the corresponding terminal device 40 (distribution of distribution packet). It is assumed that there is no previous terminal device 40) (No in step S401), and a series of steps are terminated without executing any particular processing. In this case, a distribution packet based on the received extension packet is not distributed to any terminal device 50.
On the other hand, when the image transmission device ID included in the extension packet corresponds to any of the image transmission device IDs of the items stored as the distribution destination information 422, the specifying unit 434 displays the corresponding terminal device 40 (distribution The terminal device 40 serving as the distribution destination of the packet for use is present (Yes in step S401), and the item is extracted from the distribution destination information 422. Next, the specifying unit 434 determines whether or not the frame ID and partial image position information in the partial image area information included in the received extension packet correspond to the frame ID and corresponding position information in each item extracted from the distribution destination information 422. Is determined (step S402).
Here, the identification unit 434 matches the frame IDs, and at least a part of the area indicated by the partial image position information of the extension packet and the area indicated by the corresponding position information of the distribution destination information 422 overlap. It is determined that each information corresponds. Note that when the area indicated by the partial image position information of the extension packet is included in the area indicated by the corresponding position information of the delivery destination information 422, the specifying unit 434 may determine that each piece of information corresponds. In addition, when the area indicated by the corresponding position information of the distribution destination information 422 is included in the area indicated by the partial image position information of the extension packet, it may be determined that each piece of information corresponds. Further, only when the area indicated by the partial image position information of the extension packet matches the area indicated by the corresponding position information of the delivery destination information 422, it may be determined that each piece of information corresponds.
As described above, the specifying unit 434 includes, in the extended packet received by the extended packet receiving unit 431, the partial image position information included in the header portion of the extended packet is the corresponding position information (interesting position information receiving unit 432) of the distribution destination information 422. Specifies an extension packet corresponding to the received position information of interest). Then, a distribution packet is generated based on the identified extension packet.
The distribution destination terminal address in the item determined to correspond to each information in the distribution destination information 422 is the address of the terminal device 50 corresponding to the target of interest related to the position of interest information received by the position of interest information receiving unit 432. That is, the specifying unit 434 specifies the item to which each piece of information corresponds in the delivery destination information 422, so that the target of interest related to the request target information stored in the delivery destination request information 421 is obtained by the interested position information receiving unit 432. The terminal device 50 corresponding to the object of interest related to the received position-of-interest information is specified.

When each information does not correspond (No of step S402), the specific | specification part 434 complete | finishes a series of steps, without performing a process in particular. In this case, the distribution packet based on the extension packet is not distributed to any terminal device 50.
On the other hand, when each information corresponds (Yes in step S402), the distribution packet generation unit 435 generates a distribution packet based on the extension packet (step S403), and ends a series of steps. In step S403, it is assumed that a distribution packet is generated not only for the specified extended packet but also for the extended packet in FIG. 8B having the same image transmission apparatus ID as the specified extended packet.

FIGS. 11C to 11E are schematic diagrams illustrating an example of the data structure of the distribution packet. As shown in FIGS. 11C to 11E, the distribution packet has the same header part and payload part as the extension packet.
However, the header part has a delivery destination terminal address instead of the delivery control device address, and the partial image area information is removed from the header part. The distribution destination terminal address is an item that matches the image transmission device ID and the frame ID of the extended packet received by the image transmission device ID and the frame ID in the distribution destination information 422, and is a partial image of the extended packet received by the corresponding position information It is a delivery destination terminal address in an item determined to correspond to position information. When the image transmission device ID and the frame ID match and there are a plurality of items determined to correspond to the corresponding position information (only the distribution destination terminal address is different), a plurality of distribution packets with different distribution destination terminal addresses are generated. . In this manner, the specifying unit 434 performs processing for specifying the terminal device 50 serving as a distribution destination by specifying one or a plurality of terminal devices 50 associated with the correspondence target information of the distribution destination information 422. Here, the association between the correspondence target information in the distribution destination information 422 and the terminal device 50 is generated based on the position-of-interest information transmitted from the image analysis device 60. Therefore, the image analysis apparatus 60 transmits the corresponding target information (indicating the identifier of the target of interest) and the corresponding position information indicating the position of the target of interest as the position of interest information (by the region of interest packet), thereby It is possible to appropriately distribute the partial image distribution packet corresponding to the object of interest also to the plurality of terminal devices 50 that request the distribution of the above.
Note that the types of distribution packets are also classified into three types according to the configuration of the payload portion, similar to the types of extension packets.

  When receiving the extended packet (FIG. 8B) including the image string format information, the distribution packet generator 435 generates a distribution packet (FIG. 11C) including the image string format information. Further, when receiving the extended packet (FIG. 8C) including the image format information, the distribution packet generation unit 435 generates a distribution packet (FIG. 11D) including the image format information. On the other hand, when the distribution packet generation unit 435 receives the image sequence format information and the extended packet not including the image format information (FIG. 8D), the distribution packet generation unit 435 does not include the image sequence format information and the image format information ( FIG. 11E is generated.

  Next, in step S114 of FIG. 5, the packet distribution unit 436 uses the communication unit 41 to transmit the distribution packet generated by the distribution packet generation unit 435 to each terminal device of the distribution destination terminal address included in the distribution packet. 50 (step S114). Next, each terminal device 50 receives a distribution packet from the distribution control device 40 via the communication unit (step S115). Next, each terminal device 50 decodes the payload portion of the received distribution packet, restores the partial image, and displays it on the display portion (step S115). In the present embodiment, image data in units of slices is used as partial image data included in the payload portion. Therefore, in each terminal device 50, only the slice area included in the received distribution packet is displayed in the captured image, and the other slice areas are not displayed (displayed in black). Thereby, the entire operation sequence of the data distribution system 1 is completed.

The packet delivery unit 436 of the delivery control device 40 may deliver each delivery packet only when a predetermined condition is satisfied.
For example, the packet delivery unit 436 may deliver each delivery packet only when the security device 20 is guarding the monitoring target. In that case, the distribution control device 40 determines whether “security” or “non-security” is set in the security device 20 in the storage unit 42, that is, whether the security device 20 is monitoring the monitoring target. The security state information which shows is memorize | stored. As described above, the captured image included in the image packet is an image obtained by capturing the monitoring target to be guarded by the security device 20. When the security device 20 is not guarding the monitoring target, the packet delivery unit 436 delivers each delivery packet only when the security device 20 is guarding the monitoring target without delivering each delivery packet. To do.
Thereby, when the security device 20 is not guarding the monitoring target, the security device 20 is guarding the monitoring target while reducing the data load of the packet received by each terminal device 50 and reducing the network load. Can display an image on each terminal device 50.

  Further, the image analysis device 60 may acquire the captured image by another method instead of acquiring the captured image by receiving the extension packet from the distribution control device 40. For example, the image analysis device 60 may acquire a captured image by receiving an extension packet from the data transmission device 30, or acquire a captured image by receiving an image packet from the image transmission device 10. Also good.

  In addition, the distribution control device 40 specifies the terminal device 50 in which the interest object related to the interest object information stored in the distribution destination request information 421 corresponds to the interest object related to the interest position information received from the image analysis device 60. Instead, the distribution packet may be distributed to a predetermined terminal device 50 that is determined in advance.

As described above, in the data distribution system 1 according to the embodiment of the present invention, the distribution control device 40 identifies a packet in which a target object is shown in the partial image, for the packet related to the partial image in the captured image. Then, only the identified packet is distributed to the terminal device 50. Thereby, the data distribution system 1 can distribute only the packet of the partial image in which the object of interest is reflected to the terminal device 50, and the network load is reduced by reducing the data amount of the packet received by each terminal device 50. Can be reduced.
Further, since the distribution control device 40 immediately identifies and distributes the terminal device 50 to which the packet is to be distributed, it is not necessary to hold the received packet for a long time. Therefore, it is possible to reduce the storage capacity in the distribution control device 40.
In addition, the distribution control device 40 allocates the packet related to the predetermined data to each terminal device 50 by selecting in units of packets, so that the terminal device 50 needs to reconstruct a captured image from the packet. It is not necessary to perform image conversion that cuts out an image, and the processing load can be reduced.
Further, since the distribution control device 40 only needs to access the header portion of the data and does not need to access the payload portion (data itself), it is possible to make it difficult for the administrator of the distribution control device 40 to view the data. Thus, it is possible to improve security and protect privacy. In particular, even when the data in the payload part is encrypted and transmitted, the data can be selected in units of packets based only on the header part, so that it is possible to make the data browsing in the distribution control device 40 even more difficult.

DESCRIPTION OF SYMBOLS 1 Data distribution system 10 Image transmission apparatus 20 Security apparatus 30 Data transmission apparatus 40 Distribution control apparatus 50 Terminal apparatus 60 Image analysis apparatus 42 Memory | storage part 431 Extended packet receiving part 432 Interesting-position information receiving part 433 Delivery destination information update part 434 Identification | designated part 435 Delivery packet generator 436 Packet delivery unit

Claims (7)

A data distribution system having a data transmission device, an image analysis device, and a distribution control device,
The data transmission device identifies a position in the captured image of the partial image from information of a payload portion of a packet related to the partial image in the captured image, and includes position information regarding the identified position in a header portion of the packet Generating an extended packet, and transmitting the extended packet to the distribution control device,
The image analysis device performs image analysis on the captured image to generate interest position information indicating a position where an object of interest is reflected in the captured image, and transmits the interest position information to the distribution control device.
The delivery control device
An extended packet receiver for receiving an extended packet transmitted from the data transmitting device;
A position-of-interest information receiving unit for receiving position-of-interest information transmitted from the image analysis device;
Among the received extended packets, the position related to the position information included in the header part is a specifying unit for specifying an extended packet corresponding to the position related to the received position of interest information;
A packet delivery unit for delivering a packet based on the identified extended packet to the terminal device;
A data distribution system comprising: The position of interest information further indicates an object of interest shown in the position indicated by the position of interest information,
The distribution control device further includes a storage unit that stores the identification information of the plurality of terminal devices in association with the interest target information indicating any one of the plurality of interest targets.
The identifying unit identifies a terminal device corresponding to the object of interest related to the received position of interest information, the object of interest related to the stored object of interest information;
The data distribution system according to claim 1, wherein the packet distribution unit distributes a packet based on the identified extension packet to the identified terminal device. In the payload portion of the packet, the position of the partial image is described,
3. The data distribution system according to claim 1, wherein the data transmission device identifies the position of the partial image in the captured image by reading the position described in the payload portion of the packet.   The data according to claim 1, wherein the data transmission device restores the captured image from a plurality of the packets, and specifies a position in the captured image of the partial image based on the restored captured image. Distribution system.   The data distribution system according to any one of claims 1 to 4, wherein the data transmission device encrypts the payload portion in the extension packet by an encryption method that the terminal device can decrypt. The captured image is an image relating to a monitoring target to be guarded by a security device,
The delivery control device further includes a storage unit that stores security state information indicating whether or not the security device is guarding the monitoring target,
The data according to any one of claims 1 to 4, wherein the packet distribution unit distributes a packet based on the specified extended packet only when the security device is guarding the monitoring target. Distribution system. A data distribution method in a data distribution system having a data transmission device, an image analysis device, and a distribution control device,
The data transmission device specifies a position of the partial image in the captured image from information of a payload portion of the packet related to the partial image in the captured image, and includes position information regarding the specified position in the header portion of the packet Generating an extended packet, and transmitting the extended packet to the distribution control device,
The image analysis device performs image analysis on the captured image to generate interest position information indicating a position where the object of interest is reflected in the captured image, and transmits the interest position information to the distribution control device.
The delivery control device is
Receiving an extended packet transmitted from the data transmitting device;
Receiving the position of interest information transmitted from the image analysis device;
Among the received extended packets, the position related to the position information included in the header part specifies an extended packet corresponding to the position related to the received position of interest information,
Delivering a packet based on the identified extended packet to the terminal device;
A data distribution method characterized by comprising:

Images