JP6789905B2 - Information processing equipment, information processing methods, programs and communication systems - Google Patents

Description

Embodiments of the present invention relate to information processing devices, information processing methods, programs and communication systems.

Conventionally, a technique called multi-hop communication in which a wireless terminal relays a wireless signal has been known. In a communication system to which multi-hop communication is applied, a wireless terminal to which a wireless signal does not directly reach can be made to communicate via another wireless terminal.

In addition, a system in which multi-hop communication is applied to mobile devices such as automatic guided vehicles, drones or robots is also known. In this system, a command can be transmitted to a distant mobile device to which a wireless signal does not reach directly from the operating device via the mobile device on the way. Further, in this system, an image captured by a distant mobile device can be transmitted to an operating device via the mobile device on the way. For example, in such a system, a robot or the like can be moved to a position where a wireless signal does not reach directly in a place where it is difficult for a person to work, such as a disaster site or a dangerous area, and the robot can work on behalf of the person. .. In addition, with such a system, it is possible to check the state of a dangerous place from a safe place.

However, in a system that performs multi-hop communication using a plurality of mobile devices, one of the mobile devices may move to a position where the radio cannot reach from the relay mobile device, and the communication may be interrupted. It was. For example, in such a system, as a result of giving a move command to the first mobile device, the first mobile device may move to a position where the radio cannot reach from the relay mobile device, and communication may be suddenly interrupted. was there.

Japanese Patent No. 3364237 Japanese Patent No. 3679375 Japanese Patent No. 4918807 Japanese Patent No. 4792823 JP-A-2002-341937 Japanese Patent No. 5912451

"Compound Eye Distance Image CMOS Image Sensor", [online], Toshiba / R & D Center / R & D Library, [Searched on September 7, 2017], Internet <URL: http://www.toshiba.co.jp/ rdc / detail / 13_t24.htm> "Development of imaging technology that can simultaneously acquire color images and distance images from a single image taken with a monocular camera", [online], Toshiba / R & D Center / R & D Library, [Search on September 7, 2017], Internet <URL: https://www.toshiba.co.jp/rdc/detail/1606_01.htm>

An object to be solved by the present invention is to accurately identify the movable area of the device.

The information processing device of the embodiment is an information processing device that calculates a movable area of the first device in a communication system including a movable first device and a second device capable of communicating with the first device. .. The second device has an image pickup device for imaging the first device . The information processing device includes a wireless reach area estimation unit, a line-of-sight area identification unit, and a movable area identification unit. The radio reach area estimation unit estimates a radio reach area representing a range of distances that a radio signal can reach from the second device. The line-of-sight area identification unit acquires a surveillance image captured by the second device, and identifies an obstacle that prevents the radio signal from directly reaching the first device based on the acquired surveillance image. An area within the imaging range of the second device and not hidden by the obstacle is specified as a line-through area. The movable area specifying unit specifies an area where the wireless reach area and the line-of-sight area overlap as the movable area.

The figure which shows the communication system which concerns on 1st Embodiment. The figure which shows an example of the imaging range of a mobile device and an operation device. The block block diagram of the communication system which concerns on 1st Embodiment. The block block diagram of the movable area calculation part and the front area calculation part. The sequence diagram which shows the processing flow of the communication system which concerns on 1st Embodiment. A flowchart showing the processing flow of the operating device. The figure which shows the 1st arrangement example of a mobile device and an operation device. The figure which shows the monitoring image image | captured by the operation apparatus in the 1st arrangement example. The figure which shows the moving image which imaged by the mobile device in the 1st arrangement example. The figure which shows the view area and the movable area in the 1st arrangement example. The figure which shows the front region in the 1st arrangement example. The figure which shows the moving image which combined the information represented by the character in the 1st arrangement example. The figure which shows the moving image which combined the graphic image in the 1st arrangement example. The figure which shows the 2nd arrangement example of a mobile device and an operation device. The figure which shows the monitoring image image | captured by the operation apparatus in the 2nd arrangement example. The figure which shows the moving image which was taken by the mobile device in the 2nd arrangement example. The figure which shows the front region in the 2nd arrangement example. The figure which shows the moving image which combined the information represented by a character in the 2nd arrangement example. The figure which shows the moving image which combined the graphic image in the 2nd arrangement example. The figure which shows the communication system which concerns on 2nd Embodiment. The block block diagram of the communication system which concerns on 2nd Embodiment. The sequence diagram which shows the processing flow of the communication system which concerns on 2nd Embodiment. The figure which shows the communication system which concerns on 3rd Embodiment. The block block diagram of the communication system which concerns on 3rd Embodiment. An example of a screen for the operator to select the target device. The sequence diagram which shows the processing flow of the communication system which concerns on 3rd Embodiment. The sequence diagram which shows the processing flow of the communication system which concerns on 4th Embodiment. The sequence diagram which shows the processing flow of the communication system which concerns on 5th Embodiment. The sequence diagram which shows the processing flow of the communication system which concerns on 6th Embodiment. The sequence diagram which shows the processing flow of the communication system which concerns on 7th Embodiment. The sequence diagram which shows the processing flow of the communication system which concerns on 8th Embodiment. The sequence diagram which shows the processing flow of the communication system which concerns on 9th Embodiment. The figure which shows the communication system which concerns on 10th Embodiment. The figure which shows an example of the hardware composition of a mobile device. The figure which shows an example of the hardware composition of the operation device.

Hereinafter, the communication system 10 according to the present embodiment will be described with reference to the drawings. In the following embodiments, the parts with the same reference numerals have substantially the same configuration and operation, and therefore, duplicate description will be omitted as appropriate except for differences.

(First Embodiment)
FIG. 1 is a diagram showing a communication system 10 according to the first embodiment. FIG. 2 is a diagram showing an example of an imaging range of the mobile device 20 and the operating device 30. The communication system 10 according to the first embodiment includes a mobile device 20 (first device) and an operating device 30 (second device).

The mobile device 20 is a movable device such as an automatic guided vehicle, a drone, or a robot. The mobile device 20 has a drive mechanism such as a wheel and a motor, and a control mechanism for controlling the drive mechanism.

The mobile device 20 and the operating device 30 can communicate with each other by a wireless signal. For example, the mobile device 20 and the operating device 30 can communicate by a wireless LAN (Local Area Network) standard method such as IEEE802.11. The mobile device 20 and the operating device 30 may be capable of wireless communication by optical communication or the like other than the wireless LAN. The wireless communication between the devices in the other embodiments is the same as in the present embodiment.

Further, in the mobile device 20 and the operation device 30, the partner device with which direct communication is performed is registered in advance. In the case of the present embodiment, the operating device 30 is registered in the mobile device 20 as a partner device for direct communication. Further, in the case of the present embodiment, the mobile device 20 is registered in the operation device 30 as a partner device for direct communication. In the mobile device 20 and the operating device 30, a device that communicates in place of the other device when it is not possible to directly communicate with the other device may be registered.

The operation device 30 is a device for the operator to operate the mobile device 20. The operating device 30 may be an information processing device having a wireless communication function. The operation device 30 receives an operation by the operator and transmits a command corresponding to the received operation to the mobile device 20 by wireless communication. The mobile device 20 receives a command from the operating device 30 and moves according to the received command.

As shown in FIG. 2, the mobile device 20 has an imaging device for imaging the front. The mobile device 20 images the front (moving direction) by the imaging device. The mobile device 20 transmits an image (moving image) of the front image to the operating device 30 via wireless communication. The operating device 30 receives the moving image from the mobile device 20 and displays the received moving image on the display device. As a result, the operator can operate the mobile device 20 while referring to the moving image captured by the mobile device 20.

Further, as shown in FIG. 2, the operating device 30 has an imaging device for imaging the mobile device 20. The operation device 30 images the mobile device 20 by the image pickup device. The operation device 30 calculates an area (movable area) in which the mobile device 20 can move to and from the operation device 30 without interruption based on an image (surveillance image) of the mobile device 20. To do. Further, the operating device 30 calculates an area (front area) within the imaging range of the mobile device 20 in the movable area.

Then, the operating device 30 synthesizes the information representing the front region with the moving image received from the mobile device 20 and displays it on the display device. As a result, the operating device 30 can display the movable area of the mobile device 20 together with the moving image captured by the mobile device 20. Therefore, the operator can operate the mobile device 20 while visually confirming the area that can be moved and the area that cannot be moved.

FIG. 3 is a block configuration diagram of the communication system 10 according to the first embodiment. The mobile device 20 includes a first communication unit 21, an analysis unit 22, a movement control unit 23, a first camera unit 24, and an encoding unit 25.

The first communication unit 21 wirelessly communicates with a pre-registered partner device. In the present embodiment, the first communication unit 21 wirelessly communicates with the operating device 30. When the data to be transmitted is given, the first communication unit 21 generates a packet and transmits it to the other device. Further, when the first communication unit 21 receives a packet destined for the mobile device 20 from the other device, the first communication unit 21 gives the received packet to the analysis unit 22.

The analysis unit 22 analyzes the packet received by the first communication unit 21. When the packet contains an instruction, the analysis unit 22 gives the instruction included in the packet to the movement control unit 23.

When the movement control unit 23 receives a command from the analysis unit 22, the movement control unit 23 controls the drive mechanism according to the received command. For example, when the movement control unit 23 receives a command to move forward by 1 m, the movement control unit 23 rotates the wheels or the like so that the mobile device 20 moves forward by 1 m.

The first camera unit 24 images the front of the mobile device 20, that is, the moving direction at a predetermined angle of view. The first camera unit 24 gives the captured image (moving image) to the encoding unit 25. The first camera unit 24 may capture a moving image or a still image as a moving image.

The encoding unit 25 compresses the moving image received from the first camera unit 24 by a predetermined method to generate compressed image data. The encoding unit 25 gives the generated compressed image data to the first communication unit 21. When the first communication unit 21 receives the compressed image data from the encoding unit 25, the first communication unit 21 generates a packet containing the compressed image data and transmits the packet to the operating device 30.

The operation device 30 includes an operation reception unit 31, a command generation unit 32, a second communication unit 33, a moving image decoding unit 34, a second camera unit 35, a movable area calculation unit 36, and a front area calculation unit. It has a 37, a presentation information generation unit 38, and a presentation unit 39.

The operation reception unit 31 receives an operation related to the movement of the mobile device 20 from the operator. For example, the operation reception unit 31 accepts that the button for moving the mobile device 20 forward is pressed once.

The command generation unit 32 generates a command for the mobile device 20 to perform an operation according to the received operation in response to the operation reception unit 31 receiving the operation from the operator. For example, when the operation reception unit 31 receives that the button for moving forward is pressed once, the command generation unit 32 generates a command to move forward by 1 m. The instruction generation unit 32 gives the generated instruction to the second communication unit 33.

The second communication unit 33 wirelessly communicates with the partner device registered in advance. In the present embodiment, the second communication unit 33 wirelessly communicates with the mobile device 20. When the data to be transmitted is given, the second communication unit 33 generates a packet and transmits it to the other device. For example, when an instruction is given from the instruction generation unit 32, the second communication unit 33 generates a packet containing the instruction and transmits it to the mobile device 20. When the second communication unit 33 receives a packet containing compressed image data in which the moving image is compressed from the mobile device 20, the second communication unit 33 gives the received packet to the moving image decoding unit 34.

The moving image decoding unit 34 decompresses the compressed image data included in the packet received from the mobile device 20 to generate a moving image. The moving image decoding unit 34 gives the generated moving image to the presentation unit 39.

The second camera unit 35 images the mobile device 20 at a predetermined angle of view. For example, the second camera unit 35 captures a range in which the mobile device 20 is scheduled to move. The second camera unit 35 gives an image (surveillance image) of the mobile device 20 to the movable area calculation unit 36 and the front area calculation unit 37.

The movable area calculation unit 36 calculates a movable area in which the mobile device 20 can move to and from the operating device 30 without interruption based on the monitoring image captured by the second camera unit 35. To do. The movable area may be information represented by the relative positional relationship between the mobile device 20 and the operating device 30, or map information represented by an absolute position (for example, latitude and longitude). Good. A more detailed configuration of the movable area calculation unit 36 will be described with reference to FIG.

The front area calculation unit 37 calculates an area (front area) within the imaging range of the mobile device 20 in the movable area. That is, the front area calculation unit 37 calculates the area included in the moving image among the movable areas as the front area. The front region may be information represented by a relative positional relationship from the operating device 30, or may be map information represented by an absolute position (for example, latitude and longitude). A more detailed configuration of the front region calculation unit 37 will be described with reference to FIG.

The presentation information generation unit 38 acquires the front area calculated by the front area calculation unit 37. The presentation information generation unit 38 generates information representing the front area to be combined and displayed with the moving image based on the front area. For example, the presentation information generation unit 38 may generate information in which the range of the front region is represented by characters. For example, the presentation information generation unit 38 may generate information such as "movable 5 m further forward". Further, for example, the presentation information generation unit 38 may generate graphic information representing a range corresponding to the front region in the moving image. For example, the presentation information generation unit 38 may generate a shaded image representing a range corresponding to the front region in the moving image.

The presentation unit 39 synthesizes information representing the front region generated by the presentation information generation unit 38 with the moving image acquired from the moving image decoding unit 34 and presents the information to the operator. For example, the presentation unit 39 causes the display device to display a composite image in which information representing the front region is combined with the moving image.

FIG. 4 is a block configuration diagram of the movable area calculation unit 36 and the front area calculation unit 37. The movable area calculation unit 36 includes a wireless reach area estimation unit 51, a line-of-sight area identification unit 52, and a movable area identification unit 53.

The radio reach area estimation unit 51 estimates a radio reach area representing a range of distances that a radio signal can reach from the operating device 30. For example, the radio reach area estimation unit 51 estimates the radio reach area based on the specifications of the radio system currently used and information on the radio wave condition at the present time (for example, RSSI (Received Signal Strength Indicator) value). For example, the radio reach area estimation unit 51 may estimate a circle having a radius corresponding to the magnitude of the RSSI value centered on the operating device 30 as the radio reach area.

The line-of-sight area specifying unit 52 identifies the line-of-sight area in which the operating device 30 can perform in-line-of-sight communication based on the surveillance image captured by the second camera unit 35. That is, the line-of-sight area specifying unit 52 identifies a area where the radio signal directly reaches without being disturbed by an obstacle as the line-of-sight area.

For example, the line-of-sight area identification unit 52 acquires a surveillance image, analyzes the acquired surveillance image, and identifies the mobile device 20 and obstacles other than the mobile device 20 included in the surveillance image. .. Since the outer shape of the mobile device 20 is known, the line-of-sight area identification unit 52 can detect the mobile device 20 included in the surveillance image by, for example, pattern matching. In addition, the line-of-sight area identification unit 52 identifies all the objects included in the surveillance image, and identifies the specified objects other than the mobile device 20 as obstacles.

The line-of-sight area identification unit 52 detects the position of each of the identified obstacles. For example, the line-of-sight area identification unit 52 detects the distance from the operating device 30 to the obstacle and the direction from the operating device 30 toward the obstacle. The line-of-sight area identification unit 52 may detect the direction from the operating device 30 toward the obstacle, for example, based on the horizontal position of the obstacle in the surveillance image. Further, for example, if the size of the obstacle is known, the line-of-sight area identification unit 52 detects the distance from the operating device 30 to the obstacle based on the size of the obstacle in the surveillance image. May be good. Further, the line-of-sight area identification unit 52 may detect the position of an obstacle by using, for example, a stereo camera technique.

Then, the line-of-sight area specifying unit 52 identifies a region within the imaging range of the operating device 30 that is not hidden by obstacles as the line-of-sight area. That is, the line-of-sight area specifying unit 52 specifies the area that can be imaged by the operating device 30 as the line-of-sight area without being hidden by obstacles. The area hidden by the mobile device 20 is included in the line-of-sight area.

The movable area identification unit 53 receives the radio arrival area estimated by the radio arrival area estimation unit 51 and the line-of-sight area specified by the line-of-sight area identification unit 52. Then, the movable area specifying unit 53 specifies an area where the wireless reach area and the line-of-sight area overlap as the movable area.

The front area calculation unit 37 includes a position detection unit 54, a posture detection unit 55, an imaging range calculation unit 56, and a front area identification unit 57.

The position detection unit 54 detects the position of the mobile device 20 based on the surveillance image captured by the second camera unit 35. For example, the position detection unit 54 detects the distance from the operating device 30 to the mobile device 20 and the direction from the operating device 30 to the mobile device 20 based on the monitoring image. The position detection unit 54 may detect the direction from the operating device 30 to the mobile device 20 based on the horizontal position in the surveillance image. Further, the position detection unit 54 may detect the distance from the operation device 30 to the mobile device 20 based on the size of the mobile device 20 in the surveillance image. Further, the position detection unit 54 may detect the position of the operating device 30 by using the technique of the stereo camera.

The mobile device 20 may have two light emitting diodes that blink periodically on the side surface. In this case, the position detection unit 54 can detect the direction from the operation device 30 to the mobile device 20 based on the positions of the two light emitting diodes that blink periodically in the surveillance image. If the distance between the two light emitting diodes is known, the position detection unit 54 detects the distance from the operating device 30 to the mobile device 20 based on the distance between the two light emitting diodes in the surveillance image. be able to.

The posture detection unit 55 detects the direction in which the mobile device 20 is facing. For example, the posture detection unit 55 detects the direction in which the mobile device 20 is facing based on the surveillance image captured by the second camera unit 35. The mobile device 20 may be provided with different identification marks on each of a plurality of side surfaces, for example. In this case, the posture detection unit 55 can detect the direction in which the mobile device 20 is facing based on the identification mark detected from the monitoring image. The posture detection unit 55 may identify the shape, pattern, object provided on the side surface, or the like on the side surface, and detect the direction in which the mobile device 20 is facing.

The imaging range calculation unit 56 calculates an area within the imaging range of the mobile device 20 by the first camera unit 24 based on the position of the mobile device 20 and the direction in which the mobile device 20 is facing. For example, the imaging range calculation unit 56 stores information regarding the specifications of the first camera unit 24 of the mobile device 20 in advance, and the position of the mobile device 20, the direction in which the mobile device 20 is facing, and the first camera unit 24 Based on the angle of view, the area within the imaging range of the first camera unit 24 of the mobile device 20 is calculated. As a result, the imaging range calculation unit 56 can calculate the region included in the moving image.

The front area identification unit 57 receives the movable area calculated by the movable area calculation unit 36 and the area within the image pickup range by the first camera unit 24 of the mobile device 20 calculated by the image pickup range calculation unit 56. .. Then, the front area specifying unit 57 specifies a region where the movable region and the region within the imaging range of the mobile device 20 overlap as the front region. The front area specifying unit 57 gives the specified front area to the presentation information generation unit 38.

FIG. 5 is a sequence diagram showing a processing flow of the communication system 10 according to the first embodiment. First, the mobile device 20 images the front and generates a moving image (S111). Subsequently, the mobile device 20 encodes the moving image and transmits it to the operating device 30 by wireless communication (S112).

Subsequently, the operating device 30 captures the mobile device 20 and generates a monitoring image (S113). Subsequently, the operating device 30 calculates the movable area based on the monitoring image (S114). Subsequently, the operating device 30 calculates an area (front area) within the imaging range of the mobile device 20 in the movable area (S115). Subsequently, the operating device 30 generates information (provided information) representing the front region (S116). Subsequently, the operating device 30 synthesizes information (provided information) representing the front region with the moving image and displays it on the display device (S117).

Further, the operator operates the operating device 30 while referring to the moving image in which the information representing the front region is combined (S121). When the operation device 30 receives an operation from the operator, the operation device 30 transmits a command corresponding to the operation to the mobile device 20 by wireless communication (S122). Upon receiving the command, the mobile device 20 performs movement control according to the command (S123). In the communication system 10, the movement control of the mobile device 20 is performed by repeating the above processing.

FIG. 6 is a flowchart showing a processing flow of the operating device 30. The operating device 30 executes the process shown in FIG. 6 in more detail in S114 and S115 of FIG.

The operating device 30 executes the processes of S131 to S135 of FIG. 6 in S114. In S131, the operating device 30 estimates the radio reach area. Subsequently, in S132, the operating device 30 analyzes the monitoring image to identify an obstacle. For example, the operating device 30 identifies an object other than the mobile device 20 as an obstacle among the objects included in the surveillance image.

Subsequently, in S133, the operating device 30 detects the position of each of the identified obstacles. For example, the operating device 30 detects the distance from the operating device 30 to the obstacle and the direction from the operating device 30 toward the obstacle.

Subsequently, in S134, the operating device 30 specifies an area within the imaging range of the operating device 30 where communication within the line of sight is possible as a line-of-sight area. That is, the operating device 30 specifies an area included in the surveillance image that is not hidden by obstacles as a line-of-sight area. Then, in S135, the operating device 30 specifies an area where the radio reach area estimated in S131 and the line-of-sight area specified in S134 overlap as a movable area.

Further, the operating device 30 executes the processes of S136 to S140 of FIG. 6 in S115. In S136, the operating device 30 analyzes the monitoring image to identify the mobile device 20. Subsequently, in S137, the operating device 30 detects the position of the specified mobile device 20. For example, the operating device 30 detects the distance from the operating device 30 to the mobile device 20 and the direction from the operating device 30 to the mobile device 20.

Subsequently, in S138, the operating device 30 detects the direction in which the mobile device 20 is facing. For example, the operating device 30 analyzes the monitoring image and detects the direction in which the mobile device 20 is facing.

Subsequently, in S139, the operating device 30 calculates a region within the imaging range of the mobile device 20 based on the position of the mobile device 20 detected in S137 and the direction in which the mobile device 20 detected in S138 is facing. That is, the operating device 30 calculates the area included in the moving image. Then, in S140, the operating device 30 specifies a region in which the movable region calculated in S135 and the region within the imaging range calculated by the mobile device 20 calculated in S139 overlap as the front region.

FIG. 7 is a diagram showing a first arrangement example of the mobile device 20 and the operation device 30. FIG. 8 is a diagram showing a monitoring image captured by the operating device 30 in the first arrangement example. FIG. 9 is a diagram showing a moving image captured by the mobile device 20 in the first arrangement example.

For example, it is assumed that the mobile device 20 and the operating device 30 are arranged as shown in FIG. That is, it is assumed that the mobile device 20, the obstacle A, and the obstacle B exist within the imaging range of the operating device 30. In the first arrangement example, the mobile device 20 moves away from the operating device 30. The obstacle A has a wall shape and is located in front of the mobile device 20 (beyond the moving direction). The obstacle B has a rectangular shape and is located outside the imaging range of the mobile device 20.

In this case, the operating device 30 generates a monitoring image including the mobile device 20, the obstacle A, and the obstacle B, as shown in FIG. Further, as shown in FIG. 9, the mobile device 20 generates a moving image including the obstacle A.

FIG. 10 is a diagram showing a radio reach area, a line-of-sight area, and a movable area in the first arrangement example. The wireless reach area is, for example, a circular area centered on the operating device 30. For example, in the first arrangement example, the wireless reach area is a circular area having a radius of 50 m centered on the operating device 30.

The line-of-sight area is within the imaging range (within the angle of view) of the operating device 30, and is an area that is not hidden by obstacles (area that can be imaged without being hidden by obstacles). In the first arrangement example, the line-of-sight area is within the imaging range (within the angle of view) of the operating device 30, and is an area excluding the area hidden by the obstacle A and the obstacle B.

The movable area is an area where the radio reach area and the line-of-sight area overlap. When the mobile device 20 moves within the range of the movable area, the mobile device 20 can wirelessly communicate with the operating device 30.

FIG. 11 is a diagram showing a region within the imaging range by the mobile device 20 and a front region in the first arrangement example. In the first arrangement example, the region within the imaging range of the mobile device 20 is within the angle of view of the mobile device 20 and is a region up to the front of the obstacle A.

The front region is an region where the region within the imaging range of the mobile device 20 and the movable region overlap. The mobile device 20 can move forward within the range of this front area.

FIG. 12 is a diagram showing a moving image in which information represented by characters is combined in the first arrangement example. The operating device 30 acquires a moving image captured by the mobile device 20. Then, the operating device 30 synthesizes information representing the front region with the acquired moving image and presents it to the operator.

The information representing the forward region may be, for example, a character indicating a distance that can be moved in the forward direction. For example, when the distance L ₁ from the operating device 30 to the mobile device 20 is 20 m and the radius L ₂ of the wireless reach area of the operating device 30 is 50 m, the front region is operated from the radius L ₂ of the wireless reach area. The distance from the device 30 to the mobile device 20 is the distance obtained by subtracting the distance L ₁ (L _2- L ₁ = 30 m). Therefore, in this case, as shown in FIG. 12, the operating device 30 displays "30 m movable" as information representing the front region on the moving image.

FIG. 13 is a diagram showing a moving image in which information represented by a graphic image is combined in the first arrangement example. Further, the information representing the front region may be, for example, information representing a region that can be moved in the forward direction with a graphic image. For example, as shown in FIG. 13, the operating device 30 may combine a graphic image representing the boundary line of the front region with the moving image.

FIG. 14 is a diagram showing a second arrangement example of the mobile device 20 and the operation device 30. FIG. 15 is a diagram showing a monitoring image captured by the operating device 30 in the second arrangement example. FIG. 16 is a diagram showing a moving image captured by the mobile device 20 in the second arrangement example.

For example, assume that the mobile device 20 and the operating device 30 are arranged as shown in FIG. That is, it is assumed that the mobile device 20, the obstacle A, and the obstacle B exist within the imaging range of the operating device 30. In the second arrangement example, the mobile device 20 moves to the left side when viewed from the operating device 30. Further, there is a wall-shaped obstacle C in front of the mobile device 20 (beyond the moving direction). Further, the obstacle B exists within the imaging range of the mobile device 20.

In this case, as shown in FIG. 15, the operating device 30 generates a monitoring image including the mobile device 20, the obstacle A, and the obstacle B. In addition, as shown in FIG. 16, the mobile device 20 generates a moving image including the obstacle B and the obstacle C.

In the second arrangement example, the operation device 30 is arranged in the same orientation and position as in the first arrangement example. Further, in the second arrangement example, the direction in which the mobile device 20 faces is different from that of the first arrangement example, but the position of the mobile device 20 is the same as that of the first arrangement example. Therefore, in the second arrangement example, the radio reach area, the line-of-sight area, and the movable area are the same as those in the first arrangement example.

FIG. 17 is a diagram showing a region within the imaging range by the mobile device 20 and a front region in the second arrangement example. In the second arrangement example, the region within the imaging range of the mobile device 20 is within the angle of view of the mobile device 20 and is a region before the obstacle B and the obstacle C.

In the second arrangement example, the mobile device 20 is moving toward the region where the radio signal is blocked by the obstacle B. Therefore, in the second arrangement example, the front region is narrower than that in the first arrangement example.

FIG. 18 is a diagram showing a moving image in which information represented by characters is combined in the second arrangement example. The operating device 30 may calculate the distance from the current position of the mobile device 20 to the position where it reaches the movable area, for example, from the map information representing the front area. Then, for example, as shown in FIG. 18, the operating device 30 displays "5 m movable" as information representing the front region on the moving image.

FIG. 19 is a diagram showing a moving image in which information represented by a graphic image is combined in the second arrangement example. Further, for example, as shown in FIG. 19, the operating device 30 may synthesize a graphic image representing the boundary line of the front region with the moving image.

As described above, the communication system 10 according to the first embodiment represents the line-of-sight area where the operating device 30 can perform in-line communication and the range of the distance that the wireless signal can reach from the operating device 30. An area that overlaps with the radio reach area is specified as a movable area. Thereby, the communication system 10 according to the first embodiment can accurately identify the area where the mobile device 20 can move without interrupting the wireless signal.

Further, the communication system 10 calculates a region within the imaging range of the mobile device 20 based on the position of the mobile device 20 and the direction in which the mobile device 20 is facing. The communication system 10 specifies a region where the region within the imaging range of the mobile device 20 and the movable region overlap as the front region. Then, the communication system 10 synthesizes and displays the information representing the front region with the moving image captured by the mobile device 20. As a result, the communication system 10 can present to the operator a region in which the mobile device 20 can move without interruption of the wireless signal, together with the moving image captured by the mobile device 20. Therefore, according to the communication system 10, it is possible to prevent the mobile device 20 from being accidentally moved to a position where the wireless signal does not reach.

The second camera unit 35 included in the operating device 30 is an example of a sensing unit that senses the position of a surrounding object. The operating device 30 may include another sensing unit instead of the second camera unit 35. In this case, the line-of-sight area specifying unit 52 identifies the line-of-sight area based on the position of the surrounding object detected by the sensing unit.

The sensing unit may be, for example, a sensor capable of measuring the distance to an object with high directivity. For example, it may be a LIDAR (Laser Imaging Detection And Ranging) device or the like. The lidar device detects the distance to the target by irradiating the target with a pulsed laser and measuring the scattered light from the target.

When recognizing the mobile device 20 and an obstacle from the surveillance image, the operating device 30 detects the recognition and position of the mobile device 20 and the obstacle by using a conventionally known image analysis technique. For example, the operating device 30 may use stereo camera technology to measure the positions of the mobile device 20 and obstacles. Further, for example, the operating device 30 may use a small CMOS (complementary metal oxide semiconductor) image sensor module in which the distance to the subject can be known from the captured image (see, for example, Non-Patent Document 1), or a color image. An imaging technique capable of simultaneously acquiring a distance image may be used (see, for example, Non-Patent Document 2). Further, the operating device 30 may recognize the mobile device 20 and the obstacle by using the processor for image recognition, or may recognize the mobile device 20 and the obstacle by using the cloud service.

(Second Embodiment)
FIG. 20 is a diagram showing a communication system 10 according to the second embodiment. The communication system 10 according to the second embodiment includes a first mobile device 20-1 (first device), a second mobile device 20-2 (second device), and an operating device 30.

In the first mobile device 20-1, the second mobile device 20-2 is registered as a partner device for direct communication. The first mobile device 20-1 wirelessly communicates with the second mobile device 20-2.

In the second mobile device 20-2, the first mobile device 20-1 and the operating device 30 are registered as partner devices for direct communication. The second mobile device 20-2 wirelessly communicates with the first mobile device 20-1 and the operating device 30.

When the first mobile device 20-1 and the second mobile device 20-2 receive the packet to which they are destined, they interpret the contents of the packet by themselves and control according to the instruction contained in the packet. Run.

When the second mobile device 20-2 receives a packet that is not the destination of the second mobile device 20-2, the second mobile device 20-2 transfers the packet to the first mobile device 20-1 or the operating device 30. That is, the second mobile device 20-2 relays the communication between the first mobile device 20-1 and the operating device 30.

Specifically, when the second mobile device 20-2 receives a packet addressed to the first mobile device 20-1 from the operating device 30, the second mobile device 20-2 transfers the packet to the first mobile device 20-1. When the second mobile device 20-2 receives a packet addressed to the operating device 30 from the first mobile device 20-1, the second mobile device 20-2 transfers the packet to the operating device 30. As a result, the communication system 10 can realize multi-hop communication between the first mobile device 20-1 and the operating device 30.

In addition, the first mobile device 20-1 images the front. Further, the second mobile device 20-2 images the first mobile device 20-1.

The first mobile device 20-1 transmits an image (moving image) captured in front to the operating device 30 by multi-hop communication. The operating device 30 displays the received moving image on a display device such as a monitor. As a result, the operator can operate the first mobile device 20-1 while referring to the moving image captured by the first mobile device 20-1.

Further, the second mobile device 20-2 transmits an image (surveillance image) of the first mobile device 20-1 to the operation device 30. The operating device 30 calculates an area (movable area) in which the first mobile device 20-1 can move to and from the second mobile device 20-2 without interruption based on the monitoring image. .. Further, the operating device 30 calculates an area (front area) within the imaging range of the first mobile device 20-1 in the movable area. Then, the operating device 30 synthesizes the information representing the front region with the moving image and displays it on the display device. As a result, the operator can operate the first mobile device 20-1 while visually confirming the area that can be moved and the area that cannot be moved.

FIG. 21 is a block configuration diagram of the communication system 10 according to the second embodiment. The first mobile device 20-1 and the second mobile device 20-2 have the same block configuration as the mobile device 20 according to the first embodiment.

However, the first mobile device 20-1 and the second mobile device 20-2 have different counterpart devices for direct communication. The first communication unit 21 included in the first mobile device 20-1 wirelessly communicates with the second mobile device 20-2. Further, the first communication unit 21 included in the second mobile device 20-2 wirelessly communicates with the first mobile device 20-1 and the operation device 30. Further, when the first communication unit 21 of the second mobile device 20-2 receives a packet addressed to the first mobile device 20-1 from the operating device 30, the first communication unit 21 sends the packet to the first mobile device 20-1. Forward. Further, when the first communication unit 21 of the second mobile device 20-2 receives a packet addressed to the operation device 30 from the first mobile device 20-1, the first communication unit 21 transfers the packet to the operation device 30.

The operation device 30 according to the second embodiment has a surveillance image decoding unit 61 instead of the second camera unit 35. When the second communication unit 33 receives the packet containing the compressed image data obtained by compressing the moving image from the first mobile device 20-1, the second communication unit 33 gives the received packet to the moving image decoding unit 34. Further, when the second communication unit 33 receives the packet containing the compressed image data in which the surveillance image is compressed from the second mobile device 20-2, the second communication unit 33 gives the received packet to the surveillance image decoding unit 61.

The surveillance image decoding unit 61 decompresses the compressed image data included in the packet received from the second mobile device 20-2 to generate a surveillance image. The monitoring image decoding unit 61 gives the generated monitoring image to the movable area calculation unit 36 and the front area calculation unit 37.

The movable area calculation unit 36 calculates the movable area based on the monitored image extended by the monitored image decoding unit 61. The front area calculation unit 37 calculates a region (front region) within the imaging range of the first mobile device 20-1 among the movable regions based on the surveillance image stretched by the surveillance image decoding unit 61.

FIG. 22 is a sequence diagram showing a processing flow of the communication system 10 according to the second embodiment.

First, the first mobile device 20-1 captures the front and generates a moving image (S151). Subsequently, the first mobile device 20-1 encodes the moving image and transmits it to the operating device 30 via the second mobile device 20-2 (S152).

Subsequently, the second mobile device 20-2 captures the first mobile device 20-1 and generates a monitoring image (S153). Subsequently, the second mobile device 20-2 encodes the monitoring image and transmits it to the operating device 30 (S154).

Subsequently, the operating device 30 calculates the movable area based on the monitoring image (S155). Subsequently, the operating device 30 calculates an area (front area) within the imaging range of the first mobile device 20-1 among the movable areas (S156). Subsequently, the operating device 30 generates information (provided information) representing the front region (S157). Subsequently, the operating device 30 synthesizes information (provided information) representing the front region with the moving image and displays it on the display device (S158).

Further, the operator operates the operating device 30 while referring to the moving image in which the information representing the front region is combined (S161). When the operation device 30 receives an operation from the operator, the operation device 30 transmits a command according to the operation content to the first mobile device 20-1 via the second mobile device 20-2 (S162). Then, when the first mobile device 20-1 receives the command, the first mobile device 20-1 performs movement control according to the command (S163). In the communication system 10, the movement control of the first mobile device 20-1 is performed by repeating the above processing.

As described above, the communication system 10 according to the second embodiment can move and operate the first mobile device 20-1 that cannot directly communicate with the operating device 30. Further, the communication system 10 according to the second embodiment can accurately identify the movable region of the first mobile device 20-1.

Further, in the communication system 10 according to the second embodiment, the area where the first mobile device 20-1 can move without interruption of the wireless signal is imaged by the first mobile device 20-1 to the operator. It can be presented together with the moving image. Therefore, according to the communication system 10, even when moving the first mobile device 20-1 that cannot directly communicate with the operating device 30, the first mobile device 20-1 is moved to a position where the wireless signal does not reach. It is possible to prevent accidental movement.

(Third Embodiment)
FIG. 23 is a diagram showing a communication system 10 according to a third embodiment. The communication system 10 according to the third embodiment includes N mobile devices (N is an integer of 2 or more) and an operating device 30.

The N mobile devices 20 include a first mobile device 20-1, a second mobile device 20-2, a third mobile device 20-3, ..., And an Nth mobile device 20-N.

As for the first mobile device 20-1 out of the N mobile devices 20, the second mobile device 20-2 is registered as a partner device for direct communication. The first mobile device 20-1 wirelessly communicates with the second mobile device 20-2.

When N is 3 or more, the Xth mobile device 20-X (X is an integer of 2 or more and N-1 or less) among the N mobile devices 20 is the third (X) as a partner device for direct communication. X-1) Mobile device 20- (X-1) and th (X + 1) mobile device 20- (X + 1) are registered. The Xth mobile device 20-X wirelessly communicates with the (X-1) mobile device 20- (X-1) and the (X + 1) mobile device 20- (X + 1).

For example, the second mobile device 20-2 wirelessly communicates with the first mobile device 20-1 and the third mobile device 20-3. Further, the third mobile device 20-3 wirelessly communicates with the second mobile device 20-2 and the fourth mobile device 20-4.

In the Nth mobile device 20-N, the (N-1) mobile device 20- (N-1) and the operating device 30 are registered as partner devices for direct communication. The Nth mobile device 20-N wirelessly communicates with the (N-1) mobile device 20- (N-1) and the operating device 30.

When each mobile device 20 receives a packet destined for itself, each mobile device 20 interprets the contents of the packet by itself and executes control according to an instruction included in the packet.

Further, when each mobile device 20 receives a packet that is not the destination of itself, each mobile device 20 transfers the packet to another mobile device 20. That is, the Xth mobile device 20-X relays the communication between the (X-1) th mobile device 20- (X-1) and the (X + 1) th mobile device 20- (X + 1). Further, the Nth mobile device 20-N relays the communication between the (N-1) mobile device 20- (N-1) and the operating device 30.

As a result, the communication system 10 can realize multi-hop communication in a state where N mobile devices 20 are connected in series. More specifically, the communication system 10 is in the order of the first mobile device 20-1, the second mobile device 20-2, the third mobile device 20-3, ..., the Nth mobile device 20-N, and the operating device 30. Multi-hop communication is realized in the state of being connected in series with.

In addition, the first mobile device 20-1 images the front. Further, the Xth mobile device 20-X images the (X-1) mobile device 20- (X-1). For example, the second mobile device 20-2 images the first mobile device 20-1. Further, the third mobile device 20-3 images the second mobile device 20-2.

Further, the Nth mobile device 20-N images the (N-1) mobile device 20- (N-1). Further, the operating device 30 images the Nth mobile device 20-N.

The operation device 30 selects any one of the N mobile devices 20 as the target device to be operated according to the operation of the operator. When the target device is selected, the operation device 30 determines the mobile device 20 or the operation device 30 that images the target device as the monitoring device. For example, the operating device 30 determines the second mobile device 20-2 as the monitoring device when the first mobile device 20-1 is selected as the target device. Further, the operating device 30 determines the third mobile device 20-3 as the monitoring device when the second mobile device 20-2 is selected as the target device. The operating device 30 determines the operating device 30 as a monitoring device when the Nth mobile device 20-N is selected as the target device.

The mobile device 20 selected as the target device transmits an image (moving image) of the front image to the operating device 30 by multi-hop communication. The operating device 30 displays the received moving image on a display device such as a monitor. As a result, the operator can operate the target device while referring to the moving image captured by the target device.

Further, the mobile device 20 determined as the monitoring device transmits an image (monitoring image) of the target device to the operating device 30 by multi-hop communication. Based on the monitoring image, the operating device 30 calculates an area (movable area) in which the target device can move to and from the monitoring device without interruption of wireless communication. Further, the operating device 30 calculates an area (front area) included in the imaging range of the target device in the movable area. Then, the operating device 30 synthesizes the information representing the front region with the moving image and displays it on the display device. As a result, the operator can operate the target device while visually confirming the area that can be moved and the area that cannot be moved.

FIG. 24 is a block configuration diagram of the communication system 10 according to the third embodiment. FIG. 25 is an example of a screen for allowing the operator to select a target device.

Each of the N mobile devices 20 has the same block configuration as the mobile device 20 according to the first embodiment. However, each of the N mobile devices 20 has a different partner device for direct communication.

The operation device 30 according to the third embodiment further includes a monitoring image decoding unit 61 and a selection reception unit 71.

The selection reception unit 71 selects one of the N mobile devices 20 as the target device according to the operation by the operator. For example, the selection reception unit 71 displays a selection screen as shown in FIG. 25 and accepts the selection of the target device from the operator. When the selection reception unit 71 receives the target device, the selection reception unit 71 determines the mobile device 20 or the operation device 30 that images the target device as the monitoring device.

For example, the selection reception unit 71 determines the second mobile device 20-2 as the monitoring device when the first mobile device 20-1 is selected as the target device. Further, when the second mobile device 20-2 is selected as the target device, the selection reception unit 71 determines the third mobile device 20-3 as the monitoring device. When the Nth mobile device 20-N is selected as the target device, the selection receiving unit 71 determines the operating device 30 as the monitoring device. When the selection reception unit 71 selects the target device and determines the monitoring device, the selection result and the determination result are given to the second communication unit 33, the second camera unit 35, the monitoring image decoding unit 61, and the like.

The second communication unit 33 notifies the mobile device 20 selected as the target device that it has been selected as the target device. Then, the mobile device 20 selected as the target device compresses the moving image captured in front and transmits it to the operating device 30.

In addition, the second communication unit 33 notifies the mobile device 20 determined as the monitoring device that it has been selected as the monitoring device. Then, the mobile device 20 selected as the monitoring device compresses the monitoring image captured by the target device and transmits it to the operating device 30.

When the second communication unit 33 receives a packet containing the compressed image data obtained by compressing the moving image from the target device, the second communication unit 33 gives the compressed image data included in the received packet to the moving image decoding unit 34. The moving image decoding unit 34 decompresses the compressed image data included in the packet received from the target device to generate a moving image. The moving image decoding unit 34 gives the generated moving image to the presentation unit 39.

When the second communication unit 33 receives a packet containing the compressed image data obtained by compressing the surveillance image from the monitoring device, the second communication unit 33 gives the compressed image data included in the received packet to the monitoring image decoding unit 61. When any mobile device 20 is determined as the monitoring device, the monitoring image decoding unit 61 decompresses the compressed image data included in the packet received from the monitoring device to generate the monitoring image. The monitoring image decoding unit 61 gives the generated monitoring image to the movable area calculation unit 36 and the front area calculation unit 37.

When the operating device 30 is determined as the monitoring device, the second camera unit 35 images the target device (in this case, the Nth mobile device 20-N). Then, the second camera unit 35 gives an image (surveillance image) of the target device to the movable area calculation unit 36 and the front area calculation unit 37.

FIG. 26 is a sequence diagram showing a processing flow of the communication system 10 according to the third embodiment. First, the operator selects one of the N mobile devices 20 as the target device (S171). When the target device is selected, the operation device 30 notifies the mobile device 20 selected as the target device that the target device has been selected (S172).

Further, when the target device is selected, the operation device 30 determines a monitoring device for imaging the target device. The operating device 30 notifies the mobile device 20 selected as the monitoring device that it has been determined as the monitoring device (S173). The operating device 30 does not notify the mobile device 20 when the operating device 30 is determined as a monitoring device.

Subsequently, the target device images the front and generates a moving image (S174). Subsequently, the target device encodes the moving image and transmits it to the operating device 30 (S175).

Subsequently, the monitoring device captures the target device and generates a monitoring image (S176). Subsequently, the monitoring device encodes the monitoring image and transmits it to the operating device 30 (S177).

Subsequently, the operating device 30 calculates the movable area based on the monitoring image (S178). Subsequently, the operating device 30 calculates an area (front area) within the imaging range of the target device in the movable area (S179). Subsequently, the operating device 30 generates information (provided information) representing the front region (S180). Subsequently, the operating device 30 synthesizes information (provided information) representing the front region with the moving image and displays it on the display device (S181).

Further, the operator operates the operating device 30 while referring to the moving image in which the information representing the front region is combined (S191). When the operation device 30 receives an operation from the operator, the operation device 30 transmits a command according to the operation content to the target device (S192). Then, when the target device receives the command, the target device performs movement control according to the command (S193).

In the communication system 10, the movement control of the target device is performed by repeating the processes of S174 to S193. Further, when operating the other mobile device 20, the operator executes the process from S171 and selects the other mobile device 20 as the target device.

As described above, the communication system 10 according to the third embodiment can move and operate any mobile device 20 among the N mobile devices 20. Further, the communication system 10 according to the third embodiment can accurately identify the movable region of the target device.

Further, the communication system 10 according to the third embodiment can present to the operator a region in which the target device can move without interruption of the wireless signal together with the moving image captured by the target device. .. Therefore, according to the communication system 10 according to the third embodiment, even when the N mobile devices 20 are moved by multi-hop communication, the target devices are not moved to a position where the wireless signal does not reach. Can be done.

(Fourth Embodiment)
FIG. 27 is a sequence diagram showing a processing flow of the communication system 10 according to the fourth embodiment. The communication system 10 according to the fourth embodiment has the same configuration as the communication system 10 according to the first embodiment. That is, the communication system 10 according to the fourth embodiment includes a mobile device 20 and an operating device 30.

In the fourth embodiment, the mobile device 20 receives the movable area from the operating device 30. Then, the mobile device 20 makes an autonomous movement within the movable area, or restricts the movement so as not to move to the outside of the movable area.

Specifically, in the fourth embodiment, the communication system 10 executes the process according to the flow shown in FIG. 27. First, the operating device 30 captures the mobile device 20 and generates a monitoring image (S211). Subsequently, the operating device 30 calculates the movable area based on the monitoring image (S212). Subsequently, the operating device 30 transmits the movable area to the mobile device 20 (S213). Then, the mobile device 20 autonomously moves within the received movable area (S214).

The mobile device 20 may receive a command from the operating device 30 and move according to the received command. In this case, the operator operates the operating device 30 (S221). When the operation device 30 receives an operation from the operator, the operation device 30 transmits a command corresponding to the operation to the mobile device 20 (S222). When the mobile device 20 receives the command, the mobile device 20 controls the movement according to the command while restricting the movement within the movable area (S223).

As described above, the communication system 10 according to the fourth embodiment can move the mobile device 20 within the range of the position where the wireless signal from the operating device 30 reaches.

(Fifth Embodiment)
FIG. 28 is a sequence diagram showing a processing flow of the communication system 10 according to the fifth embodiment. The communication system 10 according to the fifth embodiment has the same configuration as the communication system 10 according to the first embodiment. That is, the communication system 10 according to the fifth embodiment includes a mobile device 20 and an operating device 30.

In the fifth embodiment, the mobile device 20 has a movable area calculation unit 36 that calculates a movable area. The mobile device 20 makes, for example, autonomously moves within the movable area, or restricts the movement so as not to move outside the movable area.

Specifically, in the fifth embodiment, the communication system 10 executes the process according to the flow shown in FIG. 28. First, the operating device 30 captures the mobile device 20 and generates a monitoring image (S231). Subsequently, the operating device 30 encodes the monitoring image and transmits it to the mobile device 20 (S232).

Subsequently, the mobile device 20 calculates the movable area based on the received surveillance image (S233). Then, the mobile device 20 autonomously moves within the calculated movable area (S234).

The mobile device 20 may receive a command from the operating device 30 and move according to the received command. In this case, the communication system 10 executes the same processing as S221 to S223 described with reference to FIG. 27.

As described above, the communication system 10 according to the fifth embodiment can move the mobile device 20 within the range of the position where the wireless signal from the operating device 30 reaches. Further, the communication system 10 according to the fifth embodiment allows the mobile device 20 to calculate the movable area.

(Sixth Embodiment)
FIG. 29 is a sequence diagram showing a processing flow of the communication system 10 according to the sixth embodiment. The communication system 10 according to the sixth embodiment has the same configuration as the case of N = 2 of the communication system 10 according to the third embodiment. That is, the communication system 10 according to the sixth embodiment includes a first mobile device 20-1, a second mobile device 20-2, and an operating device 30.

In the sixth embodiment, each of the first mobile device 20-1 and the second mobile device 20-2 receives the movable area from the operating device 30. Then, each of the first mobile device 20-1 and the second mobile device 20-2 performs autonomous movement within the movable area, or restricts movement so as not to move outside the movable area.

Specifically, in the sixth embodiment, the communication system 10 executes the process according to the flow shown in FIG. First, the second mobile device 20-2 captures the first mobile device 20-1 and generates a monitoring image (S241). Subsequently, the second mobile device 20-2 encodes the monitoring image and transmits it to the operating device 30 (S242). Subsequently, the operating device 30 calculates the movable area of the first mobile device 20-1 based on the monitoring image (S243). Subsequently, the operating device 30 transmits the movable area to the first mobile device 20-1 (S244). Then, the first mobile device 20-1 autonomously moves within the received movable area (S245).

Further, in parallel with the processing of S241 to S245, the communication system 10 also executes the processing of S251 to S254. First, the operating device 30 captures the second mobile device 20-2 and generates a monitoring image (S251). Subsequently, the operating device 30 calculates the movable area of the second mobile device 20-2 based on the monitoring image (S252). Subsequently, the operating device 30 transmits the movable area to the second mobile device 20-2 (S253). Then, the second mobile device 20-2 autonomously moves within the received movable area (S254).

The first mobile device 20-1 and the second mobile device 20-2 may receive a command from the operating device 30 and move according to the received command. In this case, the communication system 10 executes the same processing as S221 to S223 described with reference to FIG. 27.

As described above, the communication system 10 according to the sixth embodiment can move each of the first mobile device 20-1 and the second mobile device 20-2 within the range of the position where the radio signal can reach.

(7th Embodiment)
FIG. 30 is a sequence diagram showing a processing flow of the communication system 10 according to the seventh embodiment. The communication system 10 according to the seventh embodiment has the same configuration as the case of N = 2 of the communication system 10 according to the third embodiment. That is, the communication system 10 according to the seventh embodiment includes a first mobile device 20-1, a second mobile device 20-2, and an operating device 30.

In the seventh embodiment, each of the second mobile device 20-2 and the operating device 30 has a movable area calculation unit 36 for calculating a movable area. The first mobile device 20-1 receives the movable area from the second mobile device 20-2. Further, the second mobile device 20-2 receives the movable area from the operating device 30. Then, each of the first mobile device 20-1 and the second mobile device 20-2 performs autonomous movement within the movable area, or restricts movement so as not to move outside the movable area.

Specifically, in the seventh embodiment, the communication system 10 executes the process according to the flow shown in FIG. First, the second mobile device 20-2 captures the first mobile device 20-1 and generates a monitoring image (S261). Subsequently, the second mobile device 20-2 calculates the movable area of the first mobile device 20-1 based on the monitoring image (S262). Subsequently, the second mobile device 20-2 transmits the movable area to the first mobile device 20-1 (S263). Then, the first mobile device 20-1 autonomously moves within the received movable area (S264).

Further, in parallel with the processing of S261 to S264, the communication system 10 also executes the processing of S271 to S274. First, the operating device 30 captures the second mobile device 20-2 and generates a monitoring image (S271). Subsequently, the operating device 30 calculates the movable area of the second mobile device 20-2 based on the monitoring image (S272). Subsequently, the operating device 30 transmits the movable area to the second mobile device 20-2 (S273). Then, the second mobile device 20-2 autonomously moves within the received movable area (S274).

The first mobile device 20-1 and the second mobile device 20-2 may receive a command from the operating device 30 and move according to the received command. In this case, the communication system 10 executes the same processing as S221 to S223 described with reference to FIG. 27.

As described above, the communication system 10 according to the seventh embodiment can move each of the first mobile device 20-1 and the second mobile device 20-2 within the range of the position where the radio signal can reach. Further, in the communication system 10 according to the seventh embodiment, the movable area of the first mobile device 20-1 is calculated by the second mobile device 20-2, and the movable area of the second mobile device 20-2 is used as the operating device. It can be calculated by 30.

(8th Embodiment)
FIG. 31 is a sequence diagram showing a processing flow of the communication system 10 according to the eighth embodiment. The communication system 10 according to the eighth embodiment has the same configuration as the case of N = 2 of the communication system 10 according to the third embodiment. That is, the communication system 10 according to the eighth embodiment includes a first mobile device 20-1, a second mobile device 20-2, and an operating device 30.

In the eighth embodiment, the second mobile device 20-2 has a movable area calculation unit 36 for calculating a movable area. The first mobile device 20-1 receives the movable area from the second mobile device 20-2. Further, the second mobile device 20-2 uses the movable area calculated by itself. Then, each of the first mobile device 20-1 and the second mobile device 20-2 performs autonomous movement within the movable area, or restricts movement so as not to move outside the movable area.

Specifically, in the eighth embodiment, the communication system 10 executes the process according to the flow shown in FIG. First, the second mobile device 20-2 captures the first mobile device 20-1 and generates a monitoring image (S281). Subsequently, the second mobile device 20-2 calculates the movable area of the first mobile device 20-1 based on the monitoring image (S282). Subsequently, the second mobile device 20-2 transmits the movable area to the first mobile device 20-1 (S283). Then, the first mobile device 20-1 autonomously moves within the received movable area (S284).

Further, in parallel with the processing of S281 to S284, the communication system 10 also executes the processing of S291 to S294. First, the operating device 30 captures the second mobile device 20-2 and generates a monitoring image (S291). Subsequently, the operating device 30 encodes the monitoring image and transmits it to the second mobile device 20-2 (S292). Subsequently, the second mobile device 20-2 calculates the movable area of the second mobile device 20-2 based on the monitoring image (S293). Then, the second mobile device 20-2 autonomously moves within the calculated movable area (S294).

The first mobile device 20-1 and the second mobile device 20-2 may receive a command from the operating device 30 and move according to the received command. In this case, the communication system 10 executes the same processing as S221 to S223 described with reference to FIG. 27.

As described above, the communication system 10 according to the eighth embodiment can move each of the first mobile device 20-1 and the second mobile device 20-2 within the range of the position where the radio signal can reach. Further, in the communication system 10 according to the eighth embodiment, the movable area of the first mobile device 20-1 and the second mobile device 20-2 can be calculated by the second mobile device 20-2.

(9th Embodiment)
FIG. 32 is a sequence diagram showing a processing flow of the communication system 10 according to the ninth embodiment. The communication system 10 according to the ninth embodiment has the same configuration as the case of N = 2 of the communication system 10 according to the third embodiment. That is, the communication system 10 according to the ninth embodiment includes a first mobile device 20-1, a second mobile device 20-2, and an operating device 30.

In the ninth embodiment, the first mobile device 20-1 and the second mobile device 20-2 have a movable area calculation unit 36 for calculating a movable area. Further, each of the first mobile device 20-1 and the second mobile device 20-2 uses the movable area calculated by itself. Then, each of the first mobile device 20-1 and the second mobile device 20-2 performs autonomous movement within the movable area, or restricts movement so as not to move outside the movable area.

Specifically, in the ninth embodiment, the communication system 10 executes the process according to the flow shown in FIG. First, the second mobile device 20-2 captures the first mobile device 20-1 and generates a monitoring image (S301). Subsequently, the second mobile device 20-2 encodes the surveillance image and transmits it to the first mobile device 20-1 (S302). Subsequently, the first mobile device 20-1 calculates the movable area of the first mobile device 20-1 based on the monitoring image (S303). Then, the first mobile device 20-1 autonomously moves within the calculated movable area (S304).

Further, in parallel with the processing of S301 to S304, the communication system 10 also executes the processing of S311 to S314. First, the operating device 30 captures the second mobile device 20-2 and generates a monitoring image (S311). Subsequently, the operating device 30 encodes the monitoring image and transmits it to the second mobile device 20-2 (S312). Subsequently, the second mobile device 20-2 calculates the movable area of the second mobile device 20-2 based on the monitoring image (S313). Then, the second mobile device 20-2 autonomously moves within the calculated movable area (S314).

The first mobile device 20-1 and the second mobile device 20-2 may receive a command from the operating device 30 and move according to the received command. In this case, the communication system 10 executes the same processing as S221 to S223 described with reference to FIG. 27.

As described above, the communication system 10 according to the ninth embodiment can move each of the first mobile device 20-1 and the second mobile device 20-2 within the range of the position where the radio signal can reach. Further, in the communication system 10 according to the ninth embodiment, the movable area of the first mobile device 20-1 is calculated by the first mobile device 20-1, and the movable area of the second mobile device 20-2 is calculated by the second mobile device 20-2. It can be calculated by the mobile device 20-2.

(10th Embodiment)
FIG. 33 is a diagram showing a communication system 10 according to the tenth embodiment. The communication system 10 according to the tenth embodiment includes a first mobile device 20-1, a second mobile device 20-2, a third mobile device 20-3, and an operating device 30.

In the first mobile device 20-1, the second mobile device 20-2 and the third mobile device 20-3 are registered as partner devices for direct communication. Further, in each of the second mobile device 20-2 and the third mobile device 20-3, the operation device 30 is registered as a partner device for direct communication. Therefore, the packet transmitted from the operating device 30 to the first mobile device 20-1 is transferred from both the second mobile device 20-2 and the third mobile device 20-3 to the first mobile device 20-1. ..

In this case, the operating device 30 identifies the second movable area based on the radio reach area and the line-of-sight area of the second mobile device 20-2. Further, the operating device 30 identifies the third movable area based on the radio reach area and the line-of-sight area of the third mobile device 20-3. Then, the operating device 30 specifies an area including both the second movable area and the third movable area as the first movable area in which the first mobile device 20-1 can move.

As described above, the first mobile device 20-1 has a plurality of partner devices that directly communicate with each other, and can perform multi-hop communication with the operating device 30 via a plurality of routes. In such a case, the communication system 10 sets a movable area for the first mobile device 20-1 so that communication with at least one device among the plurality of partner devices is not interrupted. As a result, the communication system 10 can set a wide movable area of the first mobile device 20-1.

(Hardware configuration)
FIG. 34 is a diagram showing an example of the hardware configuration of the mobile device 20. The mobile device 20 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, a storage device 404, a mobile drive mechanism 405, a communication device 406, and an imaging device. It includes 407. Each part is connected by a bus.

The CPU 401 is a processor that executes information processing, expands the program stored in the storage device 404 into the RAM 403, executes the program, controls each unit to perform input / output, and processes data. The CPU 401 may be composed of one or more processors. Further, the mobile device 20 may include not only the CPU 401 but also another processor as long as it can execute the program. The ROM 402 stores a start program for reading the boot program from the storage device 404 into the RAM 403. The RAM 403 stores data as a work area of the CPU 401.

The storage device 404 is, for example, a hard disk drive or a flash memory. The storage device 404 stores a program executed by the mobile device 20 and an operating system. The mobile drive mechanism 405 is a power and mechanism for moving the mobile device 20. The communication device 406 is a device that wirelessly communicates with other devices. The image pickup device 407 is a device for taking an image of the front of the mobile device 20.

The program executed by the mobile device 20 is recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD in an installable or executable format file. Provided. Further, the program executed by the mobile device 20 may be stored on a computer connected to a network such as the Internet and provided by downloading the program via the network. Further, the program executed by the mobile device 20 may be configured to be provided or distributed via a network such as the Internet. Further, the program executed by the mobile device 20 may be configured to be provided by incorporating it into the ROM 402 or the like in advance.

The program executed by the mobile device 20 includes a first communication module, an analysis module, a movement control module, an encoding module, and a first camera module. In the mobile device 20, each module is loaded on the main storage device (RAM403) by the processor (CPU401) reading a program from the storage medium (storage device 404 or the like) and executing the program, and the processor (CPU401) or other members , 1st communication unit 21, analysis unit 22, movement control unit 23, 1st camera unit 24, and encoding unit 25. Note that some or all of these may be realized by hardware other than the processor.

FIG. 35 is a diagram showing an example of the hardware configuration of the operating device 30. As an example, the operating device 30 is realized by a hardware configuration similar to that of a general information processing device. The operation device 30 includes a CPU 501, a ROM 502, a RAM 503, a storage device 504, an operation unit 505, a display unit 506, a communication device 507, and an image pickup device 508. Each part is connected by a bus.

The CPU 501 is a processor that executes information processing, expands the program stored in the storage device 504 into the RAM 503 and executes it, and controls each unit to perform input / output and process data. The CPU 501 may be composed of one or more processors. Further, the operating device 30 may include not only the CPU 501 but also another processor as long as it can execute the program. The ROM 502 stores a start program for reading the boot program from the storage device 504 into the RAM 503. The RAM 503 stores data as a work area of the CPU 501.

The storage device 504 is, for example, a hard disk drive or a flash memory. The storage device 504 stores a program executed by the operating device 30 and an operating system. The operation unit 505 is an input device such as a mouse or a keyboard, receives information input from the user as an instruction signal, and outputs the instruction signal to the CPU 501. The display unit 506 is a display device such as an LCD (Liquid Crystal Display). The display unit 506 displays various information based on the display signal from the CPU 501. The communication device 507 is a device that wirelessly communicates with other devices. The image pickup device 508 is a device for taking an image of the front of the operating device 30.

The program executed by the operating device 30 is recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD in an installable or executable format file. Provided. Further, the program executed by the operating device 30 may be stored on a computer connected to a network such as the Internet and provided by downloading the program via the network. Further, the program executed by the operating device 30 may be configured to be provided or distributed via a network such as the Internet. Further, the program executed by the operating device 30 may be configured to be provided by incorporating it into the ROM 502 or the like in advance.

The programs executed by the operation device 30 include an operation reception module, an instruction generation module, a second communication module, a moving image decoding module, a second camera module, a movable area calculation module, and a front area calculation module. , It has a presentation information generation module and a presentation module. In the operating device 30, each module is loaded on the main storage device (RAM 503) by the processor (CPU 501) reading a program from the storage medium (storage device 504, etc.) and executing the program, and the processor (CPU 501) or other members , Operation reception unit 31, command generation unit 32, second communication unit 33, moving image decoding unit 34, second camera unit 35, movable area calculation unit 36, front area calculation unit 37, presentation information generation unit 38 and presentation unit. Functions as 39. Note that some or all of these may be realized by hardware other than the processor.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

20 Mobile device 20-1 1st mobile device 20-2 2nd mobile device 21 1st communication unit 22 Analysis unit 23 Mobile control unit 24 1st camera unit 25 Encoding unit 30 Operation device 31 Operation reception unit 32 Command generation unit 33 2 Communication unit 34 Moving image decoding unit 35 Second camera unit 36 Movable area calculation unit 37 Front area calculation unit 38 Presentation information generation unit 39 Presentation unit 51 Radio reach area estimation unit 52 Line-of-sight area identification unit 53 Movable area identification unit Unit 54 Position detection unit 55 Posture detection unit 56 Imaging range calculation unit 57 Front area identification unit

Claims (13)

An information processing device that calculates a movable area of the first device in a communication system including a movable first device and a second device capable of communicating with the first device.
The second device has an imaging device for imaging the first device .
The information processing device
A radio reach area estimation unit that estimates a radio reach area representing a range of distances that a radio signal can reach from the second device, and a radio reach area estimation unit.
A surveillance image captured by the second device is acquired, an obstacle that prevents the radio signal from directly reaching the first device is identified based on the acquired surveillance image, and within the imaging range of the second device. An area that is not hidden by the obstacle and is specified as a line- of- sight area.
A movable area specifying unit that specifies an area where the wireless reach area and the line-of-sight area overlap as the movable area, and
Information processing device equipped with. The first device has an imaging device for imaging the front, and has an imaging device.
The information processing device
A position detection unit that detects the position of the first device with respect to the second device from the monitoring image, and
A posture detection unit that detects which direction the first device is facing, and
An imaging range calculation unit that calculates an area within the imaging range of the first device based on the position of the first device and the direction in which the first device is facing.
A front area specifying unit that specifies an area where the movable area and the area within the imaging range of the first device overlap as a front area, and
The information processing apparatus according to claim 1 , further comprising. The information processing device
A presentation information generation unit that generates information representing the front region, and
A presentation unit that synthesizes information representing the front region with a moving image captured by the first device and presents it to the operator.
The information processing apparatus according to claim 2 , further comprising. The information processing device according to any one of claims 1 to 3 , wherein the second device is an operating device operated by an operator. The information processing device according to any one of claims 1 to 3 , wherein the second device is a movable device. The information processing device according to any one of claims 1 to 5 , which is provided in the second device. The information processing device according to any one of claims 1 to 5 , which is provided in a device other than the first device and the second device. The information processing device according to any one of claims 1 to 7 , wherein the first device acquires the movable area and moves within the movable area. An information processing method executed in an information processing device that calculates a movable area of the first device in a communication system including a movable first device and a second device capable of communicating with the first device. ,
The second device has an imaging device for imaging the first device .
The information processing device
A radio reach area representing the range of distance that the radio signal can reach from the second device is estimated.
A surveillance image captured by the second device is acquired, an obstacle that prevents the wireless signal from directly reaching the first device is identified based on the acquired surveillance image, and within the imaging range of the second device. an area not hidden by the obstacle there identified as forecast area,
An information processing method for specifying an area where the wireless reach area and the line-of-sight area overlap as the movable area. A program executed in an information processing device that calculates a movable area of the first device in a communication system including a movable first device and a second device capable of communicating with the first device.
The second device has an imaging device for imaging the first device .
The information processing device
A radio reach area estimation unit that estimates a radio reach area representing a range of distances that a radio signal can reach from the second device, and a radio reach area estimation unit.
A surveillance image captured by the second device is acquired, an obstacle that prevents the wireless signal from directly reaching the first device is identified based on the acquired surveillance image, and within the imaging range of the second device. An area that is not hidden by the obstacle and is specified as a line- of- sight area.
A movable area specifying unit that specifies an area where the wireless reach area and the line-of-sight area overlap as the movable area, and
A program that works. A communication system including a movable first device, a second device capable of communicating with the first device, and an information processing device for calculating a movable area of the first device.
The second device has an imaging device for imaging the first device .
The information processing device
A radio reach area estimation unit that estimates a radio reach area representing a range of distances that a radio signal can reach from the second device, and a radio reach area estimation unit.
A surveillance image captured by the second device is acquired, an obstacle that prevents the wireless signal from directly reaching the first device is identified based on the acquired surveillance image, and within the imaging range of the second device. An area that is not hidden by the obstacle and is specified as a line- of- sight area.
A movable area specifying unit that specifies an area where the wireless reach area and the line-of-sight area overlap as the movable area, and
Communication system including. A communication system including a first mobile device, a second mobile device, and an operating device.
The second mobile device relays communication between the first mobile device and the operating device.
The first mobile device transmits a moving image captured in front of the operating device to the operating device.
The second mobile device transmits a monitoring image of the first mobile device to the operating device.
The operating device is
A radio reach area estimation unit that estimates a radio reach area representing a range of distances that a radio signal can reach from the second mobile device, and a radio reach area estimation unit.
Based on the monitoring image, the line-of-sight area specifying unit that specifies the line-of-sight area that the second mobile device can perform in-line communication, and
A movable area specifying unit that specifies an area where the wireless reach area and the line-of-sight area overlap as a movable area, and
A position detection unit that detects the position of the first mobile device with respect to the second mobile device using the monitoring image, and
A posture detection unit that detects which direction the first mobile device is facing, and
An imaging range calculation unit that calculates an area within the imaging range of the first mobile device based on the position of the first mobile device and the direction in which the first mobile device is facing.
A front area specifying unit that specifies an area where the movable area and the area within the imaging range of the first mobile device overlap as a front area, and
A presentation information generation unit that generates information representing the front region, and
A presentation unit that synthesizes information representing the front region with the moving image and presents it to the operator.
Communication system with. A communication system including N (2 or more integers) mobile devices and operating devices.
The Nth mobile device among the N mobile devices relays communication between the (N-1) mobile device and the operating device.
When N is 3 or more, the Xth (X is an integer of 2 or more and N-1 or less) mobile device among the N mobile devices is the (X-1) mobile device and the (X + 1) th mobile device. Relay communication with
The first mobile device out of the N mobile devices images the front.
The X-th mobile device images the (X-1) mobile device and obtains an image.
The Nth mobile device images the (N-1) mobile device and obtains an image.
The operating device images the Nth mobile device and
The operating device is
Depending on the operation from the operator, any one of the above N mobile devices is selected as the target device, and the device is selected.
The mobile device that images the target device or the operating device is determined as the monitoring device.
The target device transmits a moving image captured in front of the target device to the operating device.
The monitoring device transmits a monitoring image of the target device to the operating device.
The operating device is
A radio reach area estimation unit that estimates a radio reach area representing a range of distances that a radio signal can reach from the monitoring device, and a radio reach area estimation unit.
Based on the monitoring image, the line-of-sight area identification unit that specifies the line-of-sight area that the monitoring device can perform in-line communication, and
A movable area specifying unit that specifies an area where the wireless reach area and the line-of-sight area overlap as a movable area, and
A position detection unit that detects the position of the target device with respect to the monitoring device using the monitoring image,
A posture detection unit that detects in which direction the target device is facing, and
An imaging range calculation unit that calculates an area within the imaging range of the target device based on the position of the target device and the direction in which the target device is facing.
A front area specifying unit that specifies an area where the movable area and the area within the imaging range of the target device overlap as a front area.
A presentation information generation unit that generates information representing the front region, and
A presentation unit that synthesizes information representing the front region with the moving image and presents it to the operator.
Communication system with.

Images