JP2016072686A - Image transmission/reception system and method for performing data reduction processing based on region request - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image transmission/reception system capable of reliably transmitting a captured image within a limitation of a transmission rate of a transmission line in response to a region request to the captured image.SOLUTION: An image reception system in the image transmission/reception system includes means for generating a region request to an image. On the other hand, an image transmission system in the image transmission/reception system includes: an imaging unit; means which generates a mask for passing data on an image portion other than a requested region at a lower rate on the basis of a received region request; and means which performs mask processing for reducing an image data amount by using the generated mask on image information that is generated on the basis of an image signal outputted from the imaging unit. Further, the image reception system includes: a display unit for displaying at least a part of the captured image; and means for distributing or outputting captured image information generated on the basis of received image data after the mask processing to the display unit in response to the region request.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for transmitting image information such as moving images and still images via a transmission path such as a wireless network.

  Nowadays, with the spread of mobile terminals with cameras, digital cameras with communication functions, and remote cameras, there are more opportunities to acquire and use images such as moving images and still images via wireless communication means in various situations. ing. Various types of robots that can respond to disaster sites and that can relay conferences and exhibitions have been proposed. These robots are provided with a camera, and can provide information of captured images to viewers via a wireless local area network (LAN) or the like.

  In general, when an image is transmitted from a remotely operated robot or remote camera, a wireless transmission path such as a wireless LAN is often used. Here, for the purpose of performing safe operation and detailed observation, there are many scenes where both an image range corresponding to a wide field of view and high image quality are required. However, since the upper limit of the transmission rate is not so high in the wireless transmission path, in the end, one or both of a wide field of view and high image quality must be sacrificed.

  To deal with this problem, for example, in Patent Document 1, when video data is transmitted from a remote control robot, an important area in the screen is determined according to a moving state such as a moving direction and a moving speed, and data on areas other than the important area is determined. Has proposed a technique for reducing the amount of transmitted data by increasing the compression ratio.

  Furthermore, in Patent Document 2, an encoding parameter in which the quantization parameter value is set to be smaller in a region having a larger number of votes is generated in the aggregation result for the region where image quality is to be improved, which is transmitted as improvement request data from the video decoding device. A technique for controlling the encoding process has been proposed.

JP 2009-118072 A JP 2014-57128 A

  However, in the conventional techniques as described in Patent Documents 1 and 2, it may still be difficult to provide an image having an image range or image quality requested or desired by a viewer of the image. .

  For example, in the technique described in Patent Document 1, the amount of transmitted data is reduced by detecting an important region in the operation of the robot with reference to the movement state of the robot and lowering the image quality for the other region. . However, viewing an image acquired from a robot is not limited to one robot operator. For example, in addition to the operator, there are many observers (viewers of images), each carrying or wearing his own smartphone or wearable terminal, and different in images such as moving images and still images transmitted from the robot A situation where the area is observed is also envisaged.

  In this situation, considering the application of the technique described in Patent Document 1, even if a simple sum of image areas viewed by a plurality of observers including the operator is obtained, an appropriate value that matches each person's wishes is obtained. It is obviously difficult to reduce the amount of transmission data after determining the important area.

  On the other hand, the technique described in Patent Document 2 assumes that one image is transmitted to a plurality of viewers (observers). However, the transmission here is a broadcast directed to an unspecified number. For this reason, image quality improvement requests from all viewers are handled equally, and the area to be encoded with high image quality is ultimately determined by majority vote.

  Therefore, if the technique described in Patent Document 2 is applied to a remote control robot, for example, in a situation where a large number of viewers desire to see a specific target in a direction orthogonal to the traveling direction of the robot, There is a possibility that the pilot may not be able to sufficiently see the image in the traveling direction, or that a viewer who wants to see an object with little observation desire cannot observe the object.

  Therefore, the present invention can reliably transmit the information of the captured image within the limit of the transmission rate in the transmission path for transmitting the image in a form corresponding to the request for at least a part of the region of the captured image. It is an object to provide a possible system and method.

According to the present invention, there is provided an image transmission / reception system including an image transmission system capable of transmitting information of a captured image and an image reception system capable of receiving the information of the image,
This image receiving system
An area request management means for generating one or more area requests that are requests for at least a part of an area of a captured image;
First communication control means for transmitting the area request to the image transmission system;
This image transmission system
One or more imaging units;
Based on the received area request, a mask generating means for generating a mask that allows data of an image portion other than the requested area to pass at a lower rate;
Mask processing means for performing mask processing for reducing the amount of image data using the generated mask for image information generated based on the image signal output from the imaging unit;
Second communication control means for transmitting image data generated based on the masked image information to the image receiving system;
This image receiving system
One or more display units for displaying at least part of the captured image;
There is provided an image transmission / reception system further comprising captured image distribution means for distributing or outputting captured image information generated based on the received image data to the display unit in response to the area request.

As one embodiment of the image transmission / reception system of the present invention,
The image transmission system further includes a drive unit that changes one or both of the shooting position and the shooting direction of the shooting unit,
The image receiving system receives an operation that can be performed while visually recognizing at least one of the one or more display units and controls the driving of the driving unit, and transmits an image based on the operation. An operation control unit that generates an operation control signal to be transmitted to the system;
The area request management means generates a viewing area request for requesting acquisition of a viewing area in an image to be captured related to the operation of the drive unit,
It is also preferable that the mask generating means generates a mask that allows the data of the image portion of the visual field requested by the received visual field request to be passed at a higher rate or with the highest priority.

As another embodiment of the image transmission / reception system of the present invention,
The area request management means gives priority information indicating the priority adopted in generating the mask to the area request to be generated,
The mask generation means determines whether to accept the received area request in generating the mask based on the assigned priority, or requested by a higher priority assigned area request. It is also preferable to generate a mask that passes the data of the image portion of the region at a higher rate.

Furthermore, as still another embodiment of the image transmission / reception system of the present invention,
The image transmission system further includes a drive unit that changes one or both of the shooting position and the shooting direction of the shooting unit,
The image receiving system receives an operation that can be performed while visually recognizing at least one of the one or more display units and controls the driving of the driving unit, and transmits an image based on the operation. An operation control unit that generates an operation control signal to be transmitted to the system;
Based on the received area request, the mask generation means generates a mask that passes the data of the image portion of the request superimposed area, which is an area where the request is superimposed more than a predetermined degree, at a higher rate,
The captured image distribution means preferably distributes or outputs at least the information of the image portion of the request superimposition area in the generated captured image information to a display unit that is visually recognized when operating the drive unit.

As still another embodiment of the image transmission / reception system of the present invention,
The image transmission system further includes a drive unit that changes one or both of the shooting position and the shooting direction of the shooting unit,
The image receiving system receives an operation that can be performed while visually recognizing at least one of the one or more display units and controls the driving of the driving unit, and transmits an image based on the operation. An operation control unit that generates an operation control signal to be transmitted to the system;
The captured image distribution unit determines an image part of the request superimposition region that is a region where the request is superimposed more than a predetermined degree based on the region request generated by the region request management unit, and the generated captured image information It is also preferable to distribute or output information in which at least the image portion is distinguished, explicitly, or emphasized to a display unit that is visually recognized when operating the drive unit.

Furthermore, as still another embodiment of the image transmission / reception system of the present invention,
The image transmission system further includes a bit rate measuring unit that measures a bit rate in the transmission of the image data by the second communication control unit,
The mask generation means preferably scales the mask value in the generated mask so that the bit rate falls within a predetermined value or a predetermined range based on the measured bit rate.

  As still another embodiment of the image transmission / reception system of the present invention, the mask processing means performs low-pass filtering in the spatial direction according to the mask value of the generated mask as mask processing for the image information. It is also preferable to perform a combination of one or more of low-pass filtering and color reduction processing in the time direction.

  Further, as the image transmission / reception system of the present invention, the image transmission system is preferably a drivable robot, and the image reception system preferably includes a remote control terminal capable of controlling the drive of the robot. The image transmission system is preferably a camera system including a plurality of cameras as imaging units connected to a network, and the image reception system is preferably a monitor system including a plurality of monitor displays as display units. .

According to the present invention, there is also provided an image transmission / reception method in an image transmission / reception system including an image transmission system capable of transmitting information of a captured image and an image reception system capable of receiving the information of the image,
A first step in which the image receiving system generates one or more area requests that are requests for at least a portion of an area of the image to be captured, and transmits the area request to the image transmission system;
A second step in which the image transmission system generates a mask that passes data of an image portion other than the requested area at a lower rate based on the received area request;
A mask for reducing the amount of image data by using the generated mask for image information generated based on an image signal output from one or a plurality of imaging units provided in the system by the image transmission system A third step of processing;
A fourth step in which the image transmission system transmits image data generated based on the masked image information to the image reception system;
A fifth step in which the image receiving system distributes or outputs the captured image information generated based on the received image data to a display unit provided in the system in response to the area request; An image transmission / reception method is provided.

  According to the image transmission / reception system and method of the present invention, information on a photographed image in a form corresponding to a request for at least a partial region of the photographed image is within the limit of the transmission rate in the transmission path for transmitting the image. It can be transmitted reliably.

It is the schematic which shows one Embodiment of the image transmission / reception system by this invention, and the functional block diagram which shows the function structure in one Embodiment of the image transmission system by this invention, and an image reception system. FIG. 6 is a schematic diagram of a mask illustrating one embodiment of a method for determining a mask value based on a region request. FIG. 6 is a schematic diagram of a mask illustrating another embodiment of a method for determining a mask value based on a region request. It is a flowchart which shows one Embodiment of the mask generation and mask process in the image transmission / reception method by this invention. It is a schematic diagram which shows other embodiment of the image transmission / reception system by this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Image transmission / reception system]
FIG. 1 is a schematic diagram showing an embodiment of an image transmission / reception system according to the present invention, and a functional block diagram showing a functional configuration in an embodiment of an image transmission system and an image reception system according to the present invention.

The image transmission / reception system of the embodiment shown in FIG.
(A) a photographing robot 1 as an image transmission system capable of transmitting information of a photographed image via a wireless network;
(B) An image receiving system 2 capable of receiving information of the captured image via the wireless network. Here, the image receiving system 2
(B1) the image distribution device 20;
(B2) a plurality of terminals 21 capable of wireless communication connection with the image distribution device 20;
(B3) One control terminal 22 capable of wireless communication connection with the image distribution device 20 is provided.

  A wireless network, which is a transmission path responsible for communication connection between the imaging robot (image transmission system) 1 and the image distribution apparatus 20 (image reception system 2), is a wireless LAN (Local Area Network) such as Wi-Fi (registered trademark), for example. It can be. As a modification, the radio network is connected to the imaging robot 1 and the image distribution device via the Internet via a wireless access network such as LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), or 3G (3rd Generation). 20 may be connected for communication.

  In any case, the wireless network is a transmission path with a limited amount of transmission data having an upper limit on the transmission rate (communication speed). In the present embodiment, when moving image data captured by a plurality of cameras 101 from the imaging robot 1 is simply transmitted by simply performing a normal encoding process, the upper limit of the transmission rate becomes an obstacle, and real time Suppose that it is difficult to provide the image receiving system 2 with a good video with secured characteristics.

  Further, the terminal 21 and the control terminal 22 can be user interface devices such as a smartphone, a tablet computer, a wearable terminal, a notebook computer, or a personal computer (PC). The terminal 21 (control terminal 22) can acquire at least a part of the information of the image captured by the imaging robot 1 via the image distribution device 20, and can display the information on the display unit 201 (display unit 203). Further, it is possible to input information for generating an area request described later using the input interface 202 (input interface 204). The display unit 201 (display unit 203) and the input interface 202 (input interface 204) may be, for example, a touch panel display.

  Further, the control terminal 22 has an operation control unit 221 and can control the drive of the drive unit 102 of the imaging robot 1. Specifically, a control signal for instructing at least one of the position, orientation, moving speed and traveling direction of the photographing robot 1, and the camera orientation, height, photographing start, photographing adjustment, photographing end, etc. It can be transmitted to the imaging robot 1 via the distribution device 20. Here, the control signal is generated by the operation control unit 221 in response to a command input operation performed via the input interface 204. Further, it is also preferable that an image of the robot visual field range in the traveling direction of the imaging robot 1 is displayed on the display unit 203 of the control terminal 22 as a default.

  Further, the communication between the terminal 21 and the control terminal 22 and the image distribution device 20 can also be wireless communication via a wireless LAN such as Wi-Fi (registered trademark). Further, short-range wireless communication such as Bluetooth (registered trademark) is also possible. However, as a matter of course, the terminal 21 and the control terminal 22 and the image distribution device 20 may be connected by wire to communicate. There are no restrictions on the number of terminals 21 and control terminals 22 in this method.

The image transmission / reception system according to this embodiment including the image transmission system and the image reception system described above transmits a plurality of areas to be watched in one image simultaneously via a wireless network to view the image ( This is a simultaneous viewing system for multiple gaze areas that can be distributed to an observer or a viewer. In particular,
The image receiving system 2 (image distribution device 20)
(A) One or a plurality of “region requests” that are requests for at least a part of the image to be captured are generated and transmitted to the imaging robot (image transmission system) 1.
The imaging robot (image transmission system) 1
(B) having a camera 101 as one or a plurality of photographing units;
(C) Based on the received “region request”, generate a “mask” that passes data of the image portion other than the requested region at a lower rate (lower passage rate);
(D) A mask process for reducing the amount of image data using the generated “mask” is performed on the image information generated based on the image signal output from the imaging unit;
(E) The image data generated based on the masked image information is transmitted to the image receiving system.

  Here, the “area request” is request information for designating the position and size of at least a partial area of a captured image and requesting acquisition of the area. For example, an image to be captured may be divided into small areas, and a desired or required small area may be designated and requested as a requested area.

Next, the image receiving system 2
(F) having one or more display units 201 and 203 for displaying at least a part of the photographed image;
(G) The captured image information generated based on the received image data is distributed or output to the display units 201 and 203 according to the “region request”.

  For example, each user of the terminal 21 and the control terminal 22 performs an input operation for designating a region (for example, a region of interest or a region to be observed) that is requested to be viewed in the image, and the image distribution that has received this input information. The device 20 generates an “area request” according to the input information. Here, the “mask” generated by the imaging robot 1 makes it possible to collectively reflect requests (requests) for the individual image areas in the image information. As a result, one mask process generates one image data reflecting the contents of each “region request”, and transmits this image data via a wireless network, so that the transmission rate of the network can be controlled. It is very easy to meet the restrictions.

  In other words, according to the present image transmission / reception system, information on the captured image is within the limit of the transmission rate in the transmission path for transmitting the image in a form corresponding to the “region request” for at least a part of the region of the captured image. Thus, it is possible to reliably transmit.

  In addition, since adjustment of the transmission rate according to the “area request” can be performed collectively using “mask”, for example, encoding processing after mask processing is used in existing wireless communication. Can be implemented using a circuit that has been developed.

  Furthermore, as long as the “mask” is properly generated, reliable and satisfactory image data transmission can be performed. Therefore, by devising an algorithm for generating the “mask”, it is suitable for various usage environments and applications. Image data transmission / reception can be realized.

  Incidentally, as will be described in detail later, an outline of one embodiment of the “mask” generation algorithm will be briefly described here. First, a field of view necessary for maneuvering the imaging robot 1 is set in the image screen, and an area request including part or all of the field of view is received with the highest priority. Next, “region requests” from the terminals 21 to which the priority (priority) is given are received in the order of priority, and the remaining “region requests” are received in the order of arrival. Next, when the sum of the small areas through which the image information passes reaches the upper limit, subsequent “area requests” are rejected.

  Here, when the accepted “region request” is withdrawn and there is a margin in the total area of the passable image region, a new “region request” is accepted within this margin. Further, the field of view necessary for maneuvering the robot 1 may be a region through which image information is passed regardless of the presence or absence of “region request”. When selecting an “area request” to which priority is not given, the “area request” sharing a small area to be superimposed may be preferentially selected instead of simply setting the order of arrival.

[Image transmission system: shooting robot]
Next, the functional configuration of the imaging robot 1 will be described using the functional block diagram shown in FIG.

  According to FIG. 1, the imaging robot 1 includes one or more cameras 101, a drive unit 102, a second communication interface 103, a sensor unit 104, and a processor memory. Here, the processor memory realizes an image transmission function by executing a program for causing the computer of the imaging robot 1 to function.

  Furthermore, the processor memory includes a mask generation unit 111 including a request region superimposition determination unit 111a, a mask processing unit 112, an encoding unit 113 including a bit rate measurement unit 113a, and an image synthesis unit 114 as functional components. , A speed / orientation determination unit 115 and a second communication control unit 116. 1 is understood as an embodiment of an image transmission / reception method according to the present invention.

  The one or more cameras 101 are visible light, near-infrared, or infrared-capable imaging units equipped with a solid-state image sensor such as a CCD image sensor or a CMOS image sensor.

  The drive unit 102 is a movable mechanism that can change one or both of a shooting position and a shooting direction in the camera 101. Here, the change may be realized by moving, rotating, and / or deforming the imaging robot 1 itself, or the change may be realized by changing the position, orientation, height, or the like of the camera 101.

  The second communication interface 103 is connected to a wireless network that is responsible for communication connection between the imaging robot 1 and the image distribution device 20, transmits image data that has been subjected to mask processing, and transmits information to “region request” information and the drive unit 102. It is an interface that receives control signals.

Similarly, according to FIG. 1, the mask generation unit 111 generates a “mask” that allows data of an image portion other than the requested region to pass at a lower rate based on the received “region request”. At this time, it is also preferable to generate an appropriate “mask” using one or more of the following embodiments (a) to (e). That is, the mask generation unit 111
(A) It is also preferable to generate a “mask” that passes the data of the image portion of the viewing area requested by the received “viewing area request” at a higher rate (higher passing rate) or with the highest priority. .
Here, the “field-of-view area request” is used when the user of the control terminal 22 requests acquisition of an image area corresponding to the field of view, which is a visual recognition area in the traveling direction, for example, which is necessary when the photographing robot 1 is operated. This is a generated request.

In addition, the mask generation unit 111
(B) Based on the assigned priority, decide whether to adopt the received “region request” in the generation of the “mask” or by the “region request” with higher priority It is also preferable to generate a “mask” that passes data of the image portion of the requested area at a higher rate.
Here, the priority of the “area request” can be a priority order assigned in advance to each of the terminal 21 and the control terminal 22. Alternatively, the priority value is determined based on the relative position and size of the requested area, such as a higher priority for the "area request" requesting an area closer to the center position of the captured image. It is also possible to do. In this way, even when “region requests” from multiple terminals compete, arbitration is performed using priority, and the region in the image to be masked to reduce the amount of data is appropriately determined. Can do.

Further, the mask generation unit 111
(C) Based on the received “region request”, it is also preferable to generate a “mask” that allows the data of the image portion of the request superimposed region, which is a region where the request is superimposed to a predetermined degree or more, to pass at a higher rate. .
Here, the request superimposition region is determined by the request region superimposition determination unit 111 a in the mask generation unit 111. For example, when ten “area requests” are received, an area where five or more of the areas requested by these “area requests” are overlapped can be set as a request overlapping area.

In addition, the mask generation unit 111
(D) It is also preferable to scale the mask value in the “mask” to be generated based on the bit rate measured by the bit rate measuring unit 113a described later so that the bit rate falls within a predetermined value or a predetermined range.
As a modification of this embodiment, the mask generation unit 111 sets the mask value so that the entire area of the passing region that is a region through which image information passes (the mask value is not zero) is equal to or less than a predetermined area threshold. Scaling is also preferred. Further, it is also preferable to scale the mask value so that the entire area of the region through which image information is passed with a high resolution (the region where the mask value is 1) is equal to or smaller than a predetermined area threshold.

Further, the mask generation unit 111
(E) An image area required for maneuvering is determined based on information on the moving speed and traveling direction of the imaging robot 1, and the image area is prioritized over the area requested by other “area request”. It is also preferable to reflect this in the mask value.

  Here, the image area required for the steering can be, for example, an area corresponding to the visual field range in the traveling direction of the robot 1. Among these, it is also preferable that the area of the region corresponding to the visual field range is adjusted so as to decrease as the moving speed of the robot 1 increases. As a result, only image information that is exactly necessary for steering is transmitted with high resolution and can be displayed efficiently. Thus, according to the present embodiment (e), for example, a field of view centered on the traveling direction of the imaging robot 1 is unconditionally set as a region to be transmitted, or a requested region including the field of view is requested. Can be preferentially adopted as a transmission target area.

  Further, in the embodiment (e), the information on the moving speed and the traveling direction may be generated by the speed / orientation determining unit 115 that has input the detection data from the sensor unit 104 and output to the mask generating unit 111. Here, in addition to the detection data from the sensor unit 104, the speed / orientation determination unit 115 acquires information on the rotation direction, speed, and direction of the driving wheels of the robot, for example, and determines information on the moving speed and the traveling direction. May be.

  The sensor unit 104 includes at least one of an acceleration sensor, a geomagnetic sensor, and a gyro sensor. The moving speed (acceleration) of the imaging robot 1, the azimuth and elevation angle of the robot, the amount of conversion of the direction, and the like Measurement values can be output as detection data. Further, although not shown, the sensor unit 104 includes a GPS (Global Positioning System) sensor, and outputs position information (latitude and longitude) of the imaging robot 1 and a change in position (a moving speed vector amount). Also good.

  FIG. 2 is a schematic diagram of a mask illustrating one embodiment of a method for determining a mask value based on a region request. Here, in the embodiment shown in the figure, priority is not given to the area request.

  As shown in FIGS. 2A and 2B, in the mask, as a typical example, small regions having uniform height and width in the vertical and horizontal directions are arranged in a mesh shape, and 0 to 1 for each small region. It is represented as having been given a mask value that takes values up to. Here, the mask value 1 indicates that the small region to which this value is added is a small region that does not suffer from a reduction in the data amount in the mask processing for the image information. The mask value 0 indicates that the small region to which this value is assigned is a small region in which the data amount is reduced to zero or a predetermined minimum amount in the mask process for the image information.

  Note that the division into small areas in the mask is not limited to the straight line as shown in FIGS. 2A and 2B, and for example, the area may be divided into small areas with a curve. Various shapes of the small area can be adopted. The mask value may be a binary value of 0 and 1, and may be an n value that takes n discrete values between 0 and 1, for example. Furthermore, it can be a real value between 0 and 1. In any case, the mask value corresponds to the importance (degree to be transmitted) of each small region in the image, and the image information of the small region having a larger mask value has a higher rate (with a higher passing rate). ) Set to pass.

  FIGS. 2A and 2B show an example in which mask values are determined based on three request areas for simplicity. Here, the request area is an area requested by the area request. In this embodiment, if the total area (the number of all small areas) of the small areas constituting the request area is within 15 which is the upper limit area threshold Ths, the mask value 1 is set to these request areas. . In FIG. 2A, the total area of the three request areas, that is, the number of small areas belonging to the three request areas is 15, which is equal to or less than the upper limit area threshold Ths. Accordingly, all the three request areas are accepted, and the corresponding image information is passed as it is with a mask value of 1.

On the other hand, in FIG. 2B, the total area (total number of small areas) of the three request areas is 18, which exceeds the upper limit area threshold Ths. For this reason,
(A) For two request areas having small areas that overlap each other, pass all of the small areas belonging to them,
(B) For the remaining one request area, only 2/3 of the small areas to which it belongs (only four of the six small areas in the figure) are allowed to pass. Alternatively, a mask value corresponding to a pass rate 2/3 may be assigned to each of all the small areas belonging to the request area.

  2A and 2B, the mask value for the small area outside the request area may be zero, but a value close to zero may be given. By assigning a small mask value to a small area outside the request area, the image viewer (viewer, viewer) finally sees an image (video) other than the requested or desired area image at a low resolution. For example, it can be used as a clue when changing the request area.

  As described above, request areas having overlapping areas are preferentially adopted as passing areas, so that requested or desired image areas are concentrated as much as possible or transmitted at a higher rate. Alternatively, the desired video or image can be more reliably provided to the desired terminal.

  FIG. 3 is a schematic diagram of a mask illustrating another embodiment of a method for determining a mask value based on a region request.

  According to FIG. 3, for two request areas, the mask value of the small area belonging to these areas is set to 1, and all image information relating to the small area is allowed to pass. Further, a small area surrounding these two request areas is set as a low-passage area, and a mask value of the low-passage area is set to a value between 0 and 1, for example, 0.3. That is, the image information relating to the low pass rate region is allowed to pass at a lower pass rate. Further, the mask value is set to zero in the remaining small areas.

  By setting the mask value in this way, not only the requested or desired image area but also the image information in the surrounding area is allowed to pass, reducing the amount of transmitted data, and useful for the requesting or requesting user. The obtained image information can be presented more effectively. For example, the state around the object to be viewed can be visually recognized with low resolution, or when the object to be viewed moves, its moving direction can be easily identified.

  Returning to FIG. 1, when there are a plurality of cameras 101, the image synthesizing unit 114 synthesizes a plurality of image signals output from them, and generates image information corresponding to a single image. At this time, if there is a portion that overlaps the image region corresponding to the real space between the image signals, it is also preferable to synthesize the image information by overlapping the portions and connecting the two. Further, when any image signal does not cover an image area corresponding to a real space, it is also possible to synthesize image information while expressing the image area in a single color.

  The mask processing unit 112 performs a mask process for reducing the amount of image data on the image information generated based on the image signal output from the camera 101 using the generated mask. Specifically, as the mask processing, for the image information input from the image composition unit 114, for each small region of the mask acquired from the mask generation unit 111, according to the mask value of the small region (a) in the spatial direction It is also preferable to perform one or more combinations of low-pass filtering, (b) temporal low-pass filtering, and (c) subtractive color processing. Further, color reduction or frame thinning may be performed instead of the above-described low-pass filtering process, and it is possible to paint with a fixed color as the color reduction process. Furthermore, when the mask value is 1, it is preferable that the image information of the small area is passed through without being processed.

  The encoding unit 113 performs encoding processing on the masked image information input from the mask processing unit 112 and generates image data to be transmitted to the image distribution device 20. As the encoding unit 113, circuits and algorithms that have been widely used in the past can be used. For example, widely used MPEG4, H.264, etc. can be employed as the encoding algorithm.

  The encoding unit 113 preferably includes a bit rate measurement unit 113a. The bit rate measurement unit 113 a measures the bit rate in the transmission of image data by the second communication control unit 116 and outputs the measured bit rate information to the mask generation unit 111. The mask generation unit 111 uses the input bit rate information as feedback information, scales the mask value in the generated mask based on the bit rate measurement value, and adjusts the bit rate in image data transmission.

  The second communication control unit 116 transmits the image data generated by encoding the masked image information to the image distribution device 20 via the wireless network. At this time, for example, image data based on a plurality of image signals output from a plurality of cameras 101, for example, image data reflecting a plurality of request areas requested from a plurality of terminals is displayed on a single screen, for example. It is possible to send the image data corresponding to one image to the wireless network as one image data. Here, since the amount of image data is appropriately adjusted by mask processing, the image data can be reliably transmitted within the limit of the transmission rate in the wireless network.

[Image transmission / reception method: mask generation and mask processing]
FIG. 4 is a flowchart showing an embodiment of mask generation and mask processing in the image transmission / reception method according to the present invention. In the present embodiment, the mask value is a non-negative integer value such as 0, 1, 2,..., Unlike the values of 0 to 1 in the above-described embodiment.

(S401) The received area request list is cleared (erased).
(S402) The mask values of all small areas in the mask are cleared (zeroed).
(S403 to S406) A loop for receiving notifications (via the image distribution device 20) from the terminals 21 and 22 for a predetermined time is entered.

(S404) The type of notification is determined. If it is determined that the notification is an area request, the area request is added to the list (step S405a). If it is determined that the notification is an area request withdrawal notification, the corresponding area request is deleted from the list (step S405b). Further, when it is determined that the notification is other notification that is neither an area request nor an area request withdrawal notification, nothing is performed on the area request list.

(S407) After a predetermined time elapses, whether or not the request area requested by the area request in the list overlaps the visual field area in the traveling direction, the priority (priority) of the corresponding area request, and the arrival order of the corresponding area request Sort the area requests in the list using as a key.
(S408) The area request is extracted from the head of the list, and the mask value 1 is added to the small area belonging to the request area related to the area request. For example, if 1 has already been assigned as the mask value, this mask value is set to 2. The extracted area request is removed from the top of the list. Therefore, when the process in this step is performed next, the area request that was originally listed next to this area request is taken out as the first request.

(S409) It is determined whether or not the total area (total number) of the small regions having a mask value of 1 or more is greater than or equal to the upper limit area threshold Ths. If it is determined that the upper limit area threshold Ths is not exceeded, the process returns to step S408, and the next area request is extracted from the list and processed. If all the area requests in the list have been extracted, the process proceeds to step S410, and the scaling process for the entire mask value is started.

(S410) On the other hand, when it is determined in step S409 that the upper limit area threshold value Ths is not less than, a scaling process for scaling the entire mask value is performed. A typical scaling process is a process of multiplying all mask values by a coefficient such as 0.9.
As described above, it is impossible to accurately control the transmission rate after the encoding process only by adjusting the total area of the small area having the specific mask value, but the scaling process using the actually measured bit rate is not possible. This control can be performed by doing so.

(S411) Mask processing, that is, image data amount reduction processing is performed using the generated mask.
Next, an encoding process is performed on the masked image information.

(S412) It is determined whether the bit rate of the image data after the encoding process is equal to or lower than the upper limit rate threshold value. Here, when it is determined that it is not equal to or lower than the upper limit rate threshold value, the process returns to step S410, the scaling process is performed again, and a measure for lowering the bit rate is performed.

(S413) On the other hand, if it is determined in step S412 that it is equal to or less than the upper limit rate threshold, then a request superimposition region in the generated image data is determined.
Here, the request superimposition area is an image area where requests are superimposed more than a predetermined degree. In this embodiment, the request superimposition area can be an image area composed of small areas with a mask value equal to or greater than a predetermined threshold, for example, 5 or more. .

(S414) Superimposition area information, which is information such as the position and area of the request superimposition area, is added to the generated image data.
As a result, the image distribution device 20 that has received the image data can acquire the overlap region information together and recognize the request overlap region in the acquired image data. Here, for example, the image distribution apparatus 20 can also provide the control terminal 22 with the request superimposition area distinguished, explicitly, or emphasized.

  Thus, one mask generation / mask process in the present embodiment is completed.

  Here, the processing related to the request superimposition region in the above steps S413 and S414 may be performed by the request region superimposition determination unit 111a (FIG. 1) of the mask generation unit 111. In other words, the request area superimposition determination unit 111a is regarded as a desired concentration area determination unit, and a large number of observers (viewers) of the other terminals 21 have detected which of the images from the operator of the robot 1 using the control terminal 22. It suggests whether the user wants to see the area, and, for example, performs the function of providing reference information for determining the direction in which the operator next moves the robot 1.

  It should be noted that such a desired concentrated region determination function can be handled by the request region superimposition determination unit 212a of the captured image distribution unit 212 in the image distribution device 20, as will be described in detail later. In this case, as described above, instead of extracting the outline of the entire area where the mask value is equal to or greater than the predetermined threshold value, when displaying an image, the control terminal 22 is provided with an image that distinguishes and clearly or emphasizes the request overlapping area. It becomes a shape. Naturally, the desired concentrated area determination function is provided to both the request area superimposition determination unit 111a (image transmission system) and the request area superimposition determination unit 212a (image reception system), so that both the above-described effects can be achieved. Is also possible.

[Image receiving system: image distribution device and terminal]
Next, functional configurations of the image distribution device 20, the terminal 21, and the control terminal 22 will be described using the functional block diagram shown in FIG.

  According to FIG. 1, the image distribution device 20 includes a first communication interface 205, a third communication interface 206, and a processor memory. Here, the processor memory realizes an image receiving function by executing a program for causing the computer of the image distribution apparatus 2 to function.

  Further, the processor memory includes an area request management unit 211, a captured image distribution unit 212 including a request area superimposition determination unit 212a, a first communication control unit 213, and a third communication control unit 214 as functional configuration units. Have. 1 is understood as an embodiment of an image transmission / reception method according to the present invention.

  Similarly, according to FIG. 1, the terminal 21 includes a display unit 201 and an input interface 202. In addition, the control terminal 22 includes a display unit 203, an input interface 204, and an operation control unit 221 as a function configuration unit.

  The first communication interface 205 is connected to a wireless network that is responsible for communication connection between the image distribution device 20 and the imaging robot 1, and transmits an “area request” information and a control signal to the drive unit 102 or is subjected to mask processing. It is an interface that receives data. The first communication control unit 213 controls transmission / reception of these pieces of information.

  The third communication interface 206 is an interface responsible for transmission / reception of information between the image distribution device 20 and the terminal 21 and the control terminal 22. For example, information for generating an area request from the terminal 21 and the control terminal 22 and a control signal for the drive unit 102 are received, and image data corresponding to the area request is distributed to each of the terminal 21 and the control terminal 22. To do. The third communication control unit 214 controls transmission / reception of these pieces of information.

  The area request management unit 211 generates one or more area requests for requesting acquisition of at least a part of an image to be captured. The area request is generated in response to a request from each of the terminal 21 and the control terminal 22. It is also preferable that the area request management unit 211 generates a visual field area request for requesting acquisition of a visual field area in a captured image related to the operation of the driving unit 102.

  Furthermore, it is also preferable that the area request management unit 211 assigns priority information indicating the priority employed in mask generation to the area request to be generated. Here, this priority information is transmitted to the imaging robot 1 together with the area request, and is used by the mask generation unit 111 of the imaging robot 1 to generate a mask reflecting the area request.

  The captured image distribution unit 212 distributes or outputs the captured image information generated based on the received image data to each of the terminal 21 and the terminal 22 according to the area request, and each of the corresponding display units 201 and 203. To display. Here, the photographed image information can be obtained by cutting image data including an area request from a distribution or output destination terminal from the received original image data. Alternatively, the received original image data itself can be used as photographed image information without cutting out the image portion according to the area request.

Specifically, as an embodiment of the distribution method,
(A) The photographed image distribution unit 212 duplicates the input image data and distributes the same data to the terminals 21 and 22. In this case, since each terminal recognizes the request area requested to be transmitted by itself, only the request area can be extracted from the distributed video data and displayed on the display units 201 and 203. Or
(B) The captured image distribution unit 212 performs a decoding process on the input image data, extracts a request area requested by each terminal from the decoded image data, and re-encodes each extracted image data. And distribute to each terminal.

  In addition, the captured image distribution unit 212 distributes or outputs at least the information of the image portion of the request superimposition area in the generated captured image information to the control terminal 22 and is displayed when the drive unit 102 is operated. It is also preferable to display on the screen 203.

  Furthermore, the captured image distribution unit 212 determines the image portion of the request superimposition region, which is a region where the request is superimposed more than a predetermined degree, based on the region request generated by the region request management unit 211, and the generated photographing It is also preferable that information in which at least the image portion of the image information is distinguished, explicitly or emphasized is delivered or output to the control terminal 22 and displayed on the display unit 203 that is visually recognized when the drive unit 102 is operated.

  Here, the determination of the image portion of the request superimposition region may be performed by the request region superimposition determination unit 212a in the captured image distribution unit 212. In this case, the request region superimposition determination unit 212a is regarded as a desired concentration region determination unit, in other words, a large number of observers (viewers) of the other terminals 21 have a large number of operators of the robot 1 using the control terminal 22. It suggests which area of the image the user wants to see, and provides, for example, reference information for determining the direction in which the operator next moves the robot 1. Specifically, it is possible to provide the control terminal 22 with image data in which the request superimposition areas are distinguished, explicitly or emphasized, and display the image data on the display unit 203. In the displayed image data, for example, the brightness of the image in the request area may change so as to increase or blink, or a pointer or a frame indicating the position or range of the request area may be displayed.

  The operation control unit 221 of the control terminal 22 receives an operation that can be performed while viewing the display unit 203 and controls the drive of the drive unit 102, and is directed to the imaging robot 1 based on the received operation. An operation control signal to be transmitted is generated.

[Other Embodiments of Image Transmission / Reception System]
FIG. 5 is a schematic diagram showing another embodiment of the image transmission / reception system according to the present invention.

  According to FIG. 5, the image transmission system 6 in the image transmission / reception system of this embodiment is a camera system including a remote camera 61 as a plurality of imaging units connected to a network and an image processing transmission device 64. The image receiving system 7 is a monitor system including an image distribution device 71 and a plurality of monitor displays as display units. Here, the monitor display may be a display provided in the terminal 73 and the control terminal 74 that control the remote camera 61. The terminal 73 and the control terminal 74 can be communicably connected to the image distribution apparatus 71 via, for example, the AP 72 or other network means.

  The plurality of remote cameras 61 can be communicably connected to the image processing transmission device 64 via an access point (AP) 62 via a wireless network. Here, at least one of the remote cameras 61 may be connected to the image processing transmission device 64 via a wired path, or via a shooting signal processing server 63 installed on the Internet, Communication connection with the image processing transmission apparatus 64 may be possible. In addition, at least one of the remote cameras 61 may be fixed and installed at a predetermined point, or may be installed and moved in a moving means such as an automobile or a transportation facility.

  The image processing transmission device 64 includes all of the functional units other than the camera 101 and the drive unit 102 in the imaging robot 1 shown in FIG. 1, while the image distribution device 71 is the image distribution device 20 shown in FIG. All the functional parts of In addition, both devices can be connected to each other via a wireless network for transmitting image data, similarly to the imaging robot 1 and the image distribution device 20.

  Also in the image transmission / reception system of such an embodiment, the information of the image captured by the remote camera 61 in the form corresponding to the area request by the terminals 73 and 74 is within the limit of the transmission rate in the wireless network for transmitting the image data. Thus, it is possible to reliably transmit. In addition, the embodiment which the image transmission / reception system of this invention can take is not limited to what was demonstrated above. The technology of the present invention can be applied to any system that transmits an image captured by one or more cameras via a wireless network and displays the image on one or more display units.

  As described above in detail, according to the present invention, by using a mask reflecting an area request for at least a part of an area of a captured image, a captured image is captured in a form corresponding to the area request. Information can be reliably transmitted within the limit of the transmission rate in the transmission path for transmitting images. For example, while adhering to the upper limit of the data transfer rate in a wireless LAN such as Wi-Fi (registered trademark), for example, a plurality of specific areas in an image captured by a remote control robot are simultaneously viewed on a plurality of different smartphones, It becomes possible to enjoy a stable desired image.

  Furthermore, according to the present invention, a camera having a wide angle of view is used and shooting is performed with a high-resolution standard such as 4K or 8K. By generating a video signal mainly including the video signal, it is possible to send the video signal even in a transmission path having a low transmission rate upper limit. Furthermore, depending on the embodiment of the present invention, even when video transmission requests from a plurality of smartphones on the receiving side compete, a moving image in the traveling direction that is necessary for the operation of the carrier equipped with the camera is preferentially transmitted. It becomes possible.

  In addition, the configuration and method of the present invention can be used to view a captured image such as a video conference system, an exhibition guidance system, a disaster area search / survey system, a high radiation area search / survey system, a specific area or wide area monitoring system, etc. It can be applied to various systems where needs exist.

  In the various embodiments of the present invention described above, various changes, modifications, and omissions in the technical idea and scope of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

1 Shooting robot (image transmission system)
101 Camera (shooting unit)
DESCRIPTION OF SYMBOLS 102 Drive part 103 2nd communication interface 104 Sensor part 111 Mask production | generation part 111a Request area superimposition determination part 112 Mask processing part 113 Coding part 113a Bit rate measurement part 114 Image composition part 115 Speed / orientation determination part 116 2nd communication control part 2 Image reception system 20, 71 Image distribution device 201, 203 Display unit 202, 204 Input interface 205 First communication interface 206 Third communication interface 21, 73 Terminal 211 Area request management unit 212 Captured image distribution unit 212a Request area superimposition determination unit 213 1st communication control part 214 3rd communication control part 22,74 Control terminal 221 Operation control part 6 Image transmission system 61 Remote camera 62,72 Access point (AP)
63 Server 64 Image processing transmitter

Claims (10)

An image transmission / reception system comprising an image transmission system capable of transmitting information on a photographed image and an image reception system capable of receiving information on the image,
The image receiving system includes:
An area request management means for generating one or more area requests that are requests for at least a part of an area of a captured image;
First communication control means for transmitting the area request to the image transmission system;
The image transmission system includes:
One or more imaging units;
Based on the received area request, a mask generating means for generating a mask that allows data of an image portion other than the requested area to pass at a lower rate;
Mask processing means for performing mask processing for reducing the amount of image data using the generated mask for image information generated based on the image signal output from the imaging unit;
Second communication control means for transmitting image data generated based on the masked image information to the image receiving system;
The image receiving system includes:
One or more display units for displaying at least part of the captured image;
An image transmission / reception system, further comprising: captured image distribution means for distributing or outputting captured image information generated based on the received image data to the display unit in response to the area request. The image transmission system further includes a drive unit that changes one or both of a shooting position and a shooting direction of the shooting unit,
The image receiving system receives an operation that can be performed while visually recognizing at least one of the one or more display units, and controls driving of the driving unit, and based on the operation An operation control unit that generates an operation control signal to be transmitted to the image transmission system;
The area request management unit generates a view area request for requesting acquisition of a view area in an image to be captured related to the operation of the drive unit,
2. The mask generation unit according to claim 1, wherein the mask generation unit generates a mask that allows the data of the image portion of the visual field requested by the received visual field request to be passed at a higher rate or with the highest priority. The image transmission / reception system described. The area request management means gives priority information indicating a priority adopted in generation of the mask to an area request to be generated,
The mask generation means determines whether to accept the received area request in the generation of the mask based on the assigned priority, or makes a request based on an area request with a higher priority. The image transmission / reception system according to claim 1, wherein a mask that allows data of an image portion of the region to be passed at a higher rate is generated. The image transmission system further includes a drive unit that changes one or both of a shooting position and a shooting direction of the shooting unit,
The image receiving system receives an operation that can be performed while visually recognizing at least one of the one or more display units, and controls driving of the driving unit, and based on the operation An operation control unit that generates an operation control signal to be transmitted to the image transmission system;
Based on the received area request, the mask generation means generates a mask that passes the data of the image portion of the request superimposed area, which is an area where the request is superimposed more than a predetermined degree, at a higher rate,
The photographed image distribution means distributes or outputs at least the information of the image portion of the request superimposition area in the generated photographed image information to a display unit that is visually recognized when operating the drive unit. The image transmission / reception system of any one of Claim 1 to 3. The image transmission system further includes a drive unit that changes one or both of a shooting position and a shooting direction of the shooting unit,
The image receiving system receives an operation that can be performed while visually recognizing at least one of the one or more display units, and controls driving of the driving unit, and based on the operation An operation control unit that generates an operation control signal to be transmitted to the image transmission system;
The captured image distribution unit determines an image portion of a request superimposition region, which is a region where requests are superimposed more than a predetermined degree, based on the region request generated by the region request management unit, and generates a captured image 5. The information according to claim 1, wherein information in which at least the image portion is distinguished, explicitly or emphasized among the information is distributed or output to a display unit that is visually recognized during operation of the drive unit. The image transmission / reception system according to item. The image transmission system further includes a bit rate measuring unit that measures a bit rate in transmission of the image data by the second communication control unit,
6. The mask generation unit according to claim 1, wherein the mask generation means scales a mask value in a mask to be generated based on the measured bit rate so that the bit rate falls within a predetermined value or a predetermined range. The image transmission / reception system of any one of Claims.   The mask processing means is one of a spatial low pass filtering process, a temporal low pass filtering process, and a color reduction process corresponding to the mask value of the generated mask for the image information. The image transmission / reception system according to any one of claims 1 to 6, wherein one or a plurality of combinations are performed.   The said image transmission system is a drivable robot, The said image receiving system contains the remote control terminal which can control the drive of this robot, The any one of Claim 1 to 7 characterized by the above-mentioned. Image transmission / reception system.   The image transmission system is a camera system including a plurality of cameras as imaging units connected to a network, and the image reception system is a monitor system including a plurality of monitor displays as display units. The image transmission / reception system according to any one of claims 1 to 7. An image transmission / reception method in an image transmission / reception system comprising an image transmission system capable of transmitting information of a photographed image and an image reception system capable of receiving the information of the image,
A first step in which the image receiving system generates one or more area requests that are requests for at least a partial area of an image to be captured, and transmits the area request to the image transmission system;
A second step in which the image transmission system generates a mask that allows data of an image portion other than the requested region to pass at a lower rate based on the received region request;
The image transmission system uses the generated mask to reduce the amount of image data for the image information generated based on the image signal output from one or more imaging units provided in the system. A third step of performing masking;
A fourth step in which the image transmission system transmits image data generated based on the masked image information to the image reception system;
A fifth step in which the image receiving system distributes or outputs the captured image information generated based on the received image data to a display unit provided in the system in response to the area request; An image transmission / reception method comprising:

Images