WO2022158255A1 - 情報処理装置、および移動装置、並びに情報処理システム - Google Patents

情報処理装置、および移動装置、並びに情報処理システム Download PDF

Info

Publication number
WO2022158255A1
WO2022158255A1 PCT/JP2021/048264 JP2021048264W WO2022158255A1 WO 2022158255 A1 WO2022158255 A1 WO 2022158255A1 JP 2021048264 W JP2021048264 W JP 2021048264W WO 2022158255 A1 WO2022158255 A1 WO 2022158255A1
Authority
WO
WIPO (PCT)
Prior art keywords
area
stage
map
data
potential map
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/048264
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
直之 佐藤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to US18/260,619 priority Critical patent/US20240053169A1/en
Priority to JP2022577067A priority patent/JP7823590B2/ja
Priority to CN202180090687.0A priority patent/CN116783565A/zh
Publication of WO2022158255A1 publication Critical patent/WO2022158255A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/20—Instruments for performing navigational calculations
    • G01C21/206—Instruments for performing navigational calculations specially adapted for indoor navigation
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
    • G01C21/3804—Creation or updating of map data
    • G01C21/3833—Creation or updating of map data characterised by the source of data
    • G01C21/3837—Data obtained from a single source
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/005—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 with correlation of navigation data from several sources, e.g. map or contour matching
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions

Definitions

  • the present disclosure relates to information processing devices, mobile devices, and information processing systems. More specifically, in a configuration where a performance such as a performer's song performed on the stage is filmed with a camera attached to a moving device (automatic traveling robot) that moves on the stage, it collides with an object such as a performer or a speaker on the stage.
  • the present invention relates to an information processing device, a moving device, and an information processing system that generate a map that sets a route that does not interfere with the line of sight of spectators, and controls movement according to the map.
  • the camera When performing such image capturing, the camera is mounted on a moving device (cart) such as an automatic traveling robot, moves to various positions on the stage, and captures images from various angles.
  • a moving device such as an automatic traveling robot
  • Patent Document 1 Japanese Unexamined Patent Application Publication No. 2020-087061
  • Patent Document 2 Patent No. 5160322
  • Patent Document 1 discloses an unmanned mobile body that monitors a person so as not to interfere with the person to be monitored. Specifically, it is a configuration in which a region in which a moving object is difficult to be sensed by the sensory organs of a person to be monitored is determined, and monitoring is performed from the determined region.
  • Patent Document 2 discloses a robot device that follows a certain object, and even if an obstacle enters between the object to be followed and the robot and the robot is about to lose sight of the object to be followed, the robot continues the following process.
  • An enabling configuration is disclosed.
  • Patent Document 1 selects an area to be monitored that is difficult for a human to perceive, and monitors from the selected position. , each merely disclosing a configuration for achieving a particular purpose.
  • Patent Documents 1 and 2 do not disclose a control configuration for determining the optimum route for moving the camera in consideration of such various situations.
  • the present disclosure has been made, for example, in view of the above problems, and it is possible to optimize the camera according to various situations such as the movement position of the performer performing on the stage, the position of the equipment on the stage, and the line of sight of the audience.
  • the present invention provides an information processing device, a mobile device, and an information processing system that determine a suitable position and travel route and control the movement of a mobile device (camera).
  • a first aspect of the present disclosure includes: having a data processing unit that generates a potential map that defines a travel allowable area of an image capturing robot that moves on a stage and captures an image;
  • the data processing unit Acquiring at least one of action schedule data of the performer on the stage and arrangement schedule data of the object on the stage, and determining a probability of collision with the performer or the object on the stage based on the acquired data.
  • An information processing device that generates a potential map that defines a permissible travel area below a value.
  • a second aspect of the present disclosure is A map generated based on at least one of action schedule data of a performer on the stage and arrangement schedule data of an object on the stage, wherein the probability of collision with the performer or object on the stage is a specified threshold value.
  • a storage unit that stores travel route information generated based on a potential map that defines the following permissible travel regions, or at least one of a communication unit that acquires the travel route information from an external device, The mobile device executes travel processing according to either the travel route information acquired from the storage unit or the travel route information acquired via the communication unit.
  • a third aspect of the present disclosure is An information processing system having an image capturing robot and a server
  • the image capturing robot is an image capturing robot that moves on a stage to capture an image
  • the server is a data processing unit that generates a potential map that defines an allowable travel area of the imaging robot;
  • the data processing unit Acquiring at least one of action schedule data of the performer on the stage and arrangement schedule data of the object on the stage, and determining a probability of collision with the performer or the object on the stage based on the acquired data.
  • the imaging robot is An information processing system that travels along a travel route determined based on the potential map generated by the server.
  • a system is a logical collective configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same housing.
  • a map for determining a safe running route that does not collide with performers and objects on the stage is generated, and the imaging robot is caused to travel along the route determined based on the map. becomes possible.
  • a potential map is generated that defines an allowable travel area for an image capturing robot that moves on a stage to capture images.
  • the data processing unit acquires the action schedule data of the performer on the stage, the arrangement schedule data of the object, and the lighting control schedule data on the stage, and based on the acquired data, an area that does not collide with the performer or the object and is not conspicuous due to lighting. is defined as the permissible travel area.
  • the robot travel route is determined and the robot travels.
  • the robot travel route is determined and the robot travels.
  • FIG. 2 is a diagram illustrating an overview of a live stage and an overview of processing of the present disclosure
  • FIG. FIG. 2 is a diagram illustrating an overview of a live stage and an overview of processing of the present disclosure
  • FIG. FIG. 2 is a diagram illustrating an overview of a live stage and an overview of processing of the present disclosure
  • FIG. It is a figure explaining the structure and process of the information processing apparatus of this indication. It is a figure explaining the structure and process of the information processing system of this indication.
  • FIG. 4 is a diagram illustrating an example of a performer-based potential map generated by an individual potential map generation unit
  • FIG. 4 is a diagram illustrating an example of a lighting base potential map generated by an individual potential map generation unit
  • FIG. 4 is a diagram illustrating an example of an object-based potential map generated by an individual potential map generation unit;
  • FIG. 4 is a diagram illustrating an example of a pre-generated potential map generated by a potential map synthesizing unit; It is a figure explaining the structure and process of the information processing apparatus of this indication.
  • FIG. 4 is a diagram illustrating an example of an imaging robot travel route determined based on a pre-generated potential map;
  • FIG. 10 is a diagram showing a flowchart describing a performer-based potential map generation processing sequence executed by the information processing apparatus of the present disclosure;
  • FIG. 10 is a diagram showing a flowchart describing a lighting-based potential map generation processing sequence executed by the information processing apparatus of the present disclosure;
  • FIG. 10 is a diagram showing a flowchart describing a lighting-based potential map generation processing sequence executed by the information processing apparatus of the present disclosure
  • FIG. 4 is a diagram showing a flowchart describing an object-based potential map generation processing sequence executed by the information processing apparatus of the present disclosure
  • FIG. 5 is a diagram showing a flowchart describing a pre-generated potential map generation processing sequence executed by the information processing apparatus of the present disclosure
  • FIG. 4 is a diagram describing an example of a simulation image generated by the information processing apparatus of the present disclosure
  • FIG. 11 is a diagram illustrating processing in a second embodiment of the present disclosure
  • FIG. FIG. 11 is a diagram illustrating processing in a second embodiment of the present disclosure
  • FIG. 10 is a diagram illustrating a configuration example and processing of an information processing apparatus according to a second embodiment of the present disclosure
  • FIG. 10 is a diagram illustrating an example of a priority spectator & TV camera base potential map generated by an information processing apparatus according to a second embodiment of the present disclosure
  • FIG. 4 is a diagram showing a flowchart describing a processing sequence for generating a priority spectator & TV camera base potential map executed by the information processing apparatus of the present disclosure
  • FIG. 12 is a diagram showing a flowchart describing a pre-generated potential map generation processing sequence executed by the information processing apparatus according to the second embodiment of the present disclosure
  • FIG. 11 is a diagram illustrating a configuration example and processing of an information processing apparatus according to a third embodiment of the present disclosure
  • FIG. 11 is a diagram illustrating the configuration and processing of an information processing system according to Example 3 of the present disclosure
  • FIG. 11 is a diagram illustrating an example of a potential map generated by an information processing apparatus according to Example 3 of the present disclosure
  • FIG. 11 is a diagram illustrating an example of a potential map generated by an information processing apparatus according to Example 3 of the present disclosure
  • FIG. 12 is a diagram illustrating a flowchart describing a real-time data reflection potential map generation processing sequence executed by an information processing apparatus according to a third embodiment of the present disclosure
  • FIG. 12 is a diagram illustrating a flowchart describing a real-time data reflection potential map generation processing sequence executed by an information processing apparatus according to a third embodiment of the present disclosure
  • It is a figure explaining the hardware structural example of the information processing apparatus of this indication.
  • FIG. 1 is a diagram showing an example of a performance such as a live music performance on a stage.
  • the example shown in FIG. 1 is an example in which a performer 20 who is a duo of idol singers is performing live music on the stage 10 .
  • a large number of spectators 30 are in front of the stage 10 and are watching the performance of the performers 20 .
  • the image capturing robot 50 shown in FIG. 1 is a mobile device on which a camera is mounted, that is, a running robot, and moves around the stage to capture the performance of the performer 20 from various angles.
  • the image capturing robot 50 is an automatic traveling robot (moving device) such as a trolley equipped with a camera, and captures images from various angles while traveling on the stage, for example, following a predetermined running route.
  • moving device such as a trolley equipped with a camera
  • the image capturing robot 50 needs to select a safe running route and move so as not to collide with the performers moving around on the stage and equipment such as microphones and speakers installed on the stage. In addition, it is also required to move so as not to interfere with the line of sight of the audience in front of the stage and surroundings.
  • speakers 12, monitors 13, and various decorative objects 14 are placed on a stage 10 where live performances are actually performed, and lighting 11 is also applied to performers 20 and the like.
  • the performer 20 moves around the stage as the live performance progresses, and the position, brightness, and color of the lighting 11 change according to this movement. Also, the decoration objects 14 are replaced one after another according to the progress of the live performance.
  • the stage state shown in FIG. 2 is a scene at time t1 during a live performance.
  • the stage state at a certain time t2 after that for example, as shown in FIG. 3, the position of the performer 20 is changed, and the position, brightness, and color of the illumination 11 are also changed.
  • the decoration object 14 is also replaced.
  • the position of the performer 20 on the stage, the position, brightness and color of the lighting 11, the position of the decoration object, etc. are sequentially changed according to the time period during which the live performance is performed.
  • the image capturing robot 50 needs to run so as not to collide with the performer 20, the decoration object 14 on the stage, the speaker 12, and other objects arranged thereon.
  • the present disclosure determines the optimal camera position and travel route according to various situations such as the movement position of the performer performing on the stage, the position of the equipment on the stage, the lighting, the line of sight of the audience, etc. camera). Details of the configuration and processing of the present disclosure will be described below.
  • FIG. 4 is a diagram illustrating a configuration example of the information processing apparatus 100 according to the first embodiment of the present disclosure.
  • the information processing apparatus 100 may be configured inside the moving apparatus running on the stage shown in FIGS.
  • a device that is independent of the image capturing robot 50 may be used.
  • FIG. 5 shows a configuration example of an information processing system in which the information processing apparatus having the configuration shown in FIG. 4 is independent of the image capturing robot 50 .
  • an information processing system 180 is constructed in which an information processing device (server) 100, a live venue information acquisition device 60, an image capturing robot 50 in the live venue, etc. are connected via a communication network. do.
  • the information processing apparatus (server) 100 has the configuration shown in FIG.
  • the live venue information acquisition device 60 is composed of a camera that captures images of the live venue, a microphone that acquires audio information of the live venue, an illuminance meter that detects lighting conditions, a color analysis device, etc., and acquires information such as the acquired image. , to the information processing apparatus (server) 100 via the communication network.
  • the information processing device (server) 100 performs processing for generating a map (potential map) for determining the running route of the image capturing robot 50 in the live venue, determining the running route using the map, and determining the determined running route. Accordingly, processing for generating travel control information for the image capturing robot 50 and the like are performed.
  • the information processing device (server) 100 further transmits the generated travel control information to the image capturing robot 50 via the communication network.
  • the image capturing robot 50 runs on the stage according to the running control information received from the information processing device (server) 100 . For example, it is possible to perform processing using such an information processing system.
  • the information processing device 100 has a storage unit 110 , an individual potential map generation unit 120 , a potential map synthesis unit 130 and a travel route generation unit 160 .
  • the information processing apparatus 100 may be configured inside the moving apparatus that runs on the stage shown in FIGS. It may be a device independent of the image capturing robot 50, such as a device capable of communicating with the image capturing robot 50, like the information processing device (server) 100 shown.
  • the storage unit 110 stores the following three data.
  • A. performer action schedule data 111 B. stage lighting control schedule data 112
  • these three scheduled data are data that are prepared in advance and stored in the storage unit 110 before the start of the live performance. That is, it is schedule data prepared in advance according to a program such as a live progress table prepared before the start of the live performance.
  • the information processing apparatus 100 uses these schedule data to generate a map (potential map) for setting a safe travel route that does not cause the image capturing robot 50 to collide with objects such as performers and speakers.
  • a map potential map
  • the image capturing robot 50 is caused to travel according to a safe travel route selected using the generated potential map.
  • the performer action schedule data 111 is time-series position data of performers moving around on stage during a live performance. That is, it is time-series position data of the performer from the start to the end of the live (performance).
  • the stage lighting control schedule data 112 is time-series data of lighting control information including lighting setting information such as the lighting position, brightness, and color of the lighting from the start to the end of the live (performance).
  • the on-stage object placement schedule data 113 is time-series data of on-stage object placement position information including the placement positions of on-stage objects from the start to the end of a live performance (performance).
  • the on-stage objects include speakers, monitors, decorative objects, and the like placed on the stage.
  • All of these three types of data are time-series scheduled data from the start to the end of a live performance (performance). In other words, for example, if it is a one-hour live, A. performer action schedule data 111, B. stage lighting control schedule data 112, C. On-stage object placement schedule data 113, These three types of time-series schedule data for one hour are recorded in the storage unit 110 .
  • the data stored in the storage unit 110 is used in the individual potential map generation unit 120.
  • FIG. The individual potential map generating section 120 generates the following three types of individual potential maps by using the three types of time-series data A, B, and C individually.
  • map a performer base potential map
  • map b lighting base potential map
  • map c object-based potential map
  • the potential map is a map that defines a travel prohibited area, a travel caution area, and a travel permitted area for the image capturing robot 50 described with reference to FIGS. All of the above three types of maps a to c are time-series maps.
  • Map b lighting base potential map shows each area (no driving area, caution driving area, permitted driving area) according to the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of the live (performance).
  • Map c Object-based potential map is a time series map that dynamically changes each area (no driving area, caution driving area, permitted driving area) according to the object placement position from the start to the end of the live (performance) is.
  • the individual potential map generation unit 120 executes each process of steps S11 to S13 shown in the figure. i.e.
  • steps S11 to S13 can be executed as parallel processing. These processes will be sequentially described below.
  • Performer Base Potential Map is a time series map that dynamically changes each area (no driving area, caution driving area, permitted driving area) according to the position of the performer from the start to the end of the live (performance).
  • FIG. 6 shows an example of a performer base potential map at four timings (t1 to t4).
  • the map (t1) in the upper left of FIG. 6 is an example of the "performer base potential map" at time t1.
  • the potential map is a map generated before the actual start of the live performance, and the position of the performer is the expected position of the performer estimated according to the live program such as the live progress chart.
  • the action schedule data indicating the position of the performer from the start to the end of the live (performance) is stored in A. It is generated and recorded in advance as performer action schedule data 111 .
  • Performer Base Potential Map is a map in which different colors are assigned for each area according to the area (driving prohibited area, cautionary driving area, permitted driving area) determined according to the distance from the performer's position as shown below. is.
  • a close-range position of the performer position is set to red as a travel-prohibited area.
  • the mid-distance position of the performer position is set to yellow as a driving caution area.
  • the long-distance position of the performer's position is set to blue as the travel-permissible region.
  • the map (t2) in the upper right of FIG. 6 is an example of the "performer's base potential map" at time t2 after a certain period of time has elapsed from time t1.
  • the two actors are moving to different positions than at time t1.
  • the setting of these three areas ie, the travel prohibited area (red), the travel caution area (yellow), and the travel permitted area (blue), also changes.
  • the map (t3) in the lower left of FIG. 6 is an example of the "performer's base potential map" at time t3 after a certain period of time has elapsed from time t2.
  • the two actors are moving to positions different from those at times t1 and t2.
  • the setting of these three areas ie, the travel prohibited area (red), the travel caution area (yellow), and the travel permitted area (blue), also changes.
  • the map (t4) in the lower right of FIG. 6 is an example of the "performer's base potential map" at time t4 after a certain period of time has elapsed from time t3.
  • the two performers are moving to positions different from those at times t1-t3.
  • the setting of these three areas ie, the travel prohibited area (red), the travel caution area (yellow), and the travel permitted area (blue), also changes.
  • the "Performer Base Potential Map” dynamically changes each area (driving prohibited area, cautionary driving area, permitted driving area) according to the position of the performer from the start time to the end time of the live (performance). This is a time series map.
  • the action schedule data indicating the position of the performer from the start to the end of the live (performance) is stored in A. It is generated and recorded in advance as performer action schedule data 111 .
  • the individual potential map generation unit 120 stores the A.
  • Generate a ⁇ performer-based potential map''.
  • the "Lighting Base Potential Map” is used for each area (no-driving area, caution-not-driving area, This is a time-series map in which the allowable area) is dynamically changed, and is a time-series map in which the following three areas (b1 to b3) are set.
  • No-travel area (red) Area conspicuous due to lighting conditions (bright area or lighting area with a different color than the image capturing robot) b3.
  • Permissible travel area (blue) Inconspicuous area due to lighting conditions (dark area or lighting area with the same color as the image capturing robot) b2.
  • Driving caution area (yellow) intermediate area between b1 and b3
  • FIG. 7 shows an example of illumination base potential maps at four timings (t1 to t4).
  • the map (t1) in the upper left of FIG. 7 is an example of the "illumination base potential map" at time t1.
  • This map (t1) is an example of a map in a state where the illumination is dark overall at the start of a live performance, for example.
  • the map (t2) in the upper right of FIG. 7 is an example of the "illumination base potential map" at time t2 after a certain period of time has elapsed from time t1.
  • the illumination state (illumination position, brightness, color, etc.) is set to a state different from that at time t1.
  • the setting of these three areas ie, no-travel area (red), caution-driving area (yellow), and permissible-travel area (blue), also changes.
  • Lighting Base Potential Map is based on areas (no-driving areas, caution-driving areas, permissible driving areas) determined according to lighting conditions (lighting position, brightness, color, etc.) as shown below.
  • a map with different colors That is, areas that are conspicuous due to lighting conditions (bright areas or illumination areas with a color different from that of the image capturing robot) are set to red as travel-prohibited areas. Areas that are inconspicuous due to the lighting conditions (dark areas and illumination areas of the same color as the image capturing robot) are set to blue as travel-permissible areas. An intermediate area between the travel-prohibited area and the travel-allowed area is set in yellow as a travel caution area.
  • the map (t3) in the lower left of FIG. 7 is an example of the "illumination base potential map" at time t3 after a certain period of time has elapsed from time t2.
  • the lighting conditions (illumination position, brightness, color, etc.) are different from those at times t1 and t2.
  • the setting of these three areas ie, no-travel area (red), caution-driving area (yellow), and permissible-travel area (blue), also changes.
  • the map (t4) in the lower right of FIG. 7 is an example of the "illumination base potential map" at time t4 after a certain period of time has elapsed from time t3.
  • the lighting conditions (illumination position, brightness, color, etc.) are set to a state different from that at times t1 to t3.
  • the setting of these three areas ie, no-travel area (red), caution-driving area (yellow), and permissible-travel area (blue), also changes.
  • the "Lighting Base Potential Map” can be used for each area (no driving area, caution driving area, etc.) according to changes in the lighting conditions (lighting position, brightness, color, etc.) from the start time to the end time of the live (performance). It is a time-series map in which the area, driving allowable area) is dynamically changed.
  • transition schedule data of lighting conditions (lighting position, brightness, color, etc.) from the start to the end of a live performance (performance) are stored in B. It is generated and recorded in advance as stage lighting control schedule data 112 .
  • the individual potential map generation unit 120 stores the B.
  • Generate map b “illumination base potential map”.
  • Objects are objects placed on the stage, such as speakers, monitors, and decorative objects.
  • FIG. 8 shows examples of object-based potential maps at four timings (t1 to t4).
  • the map (t1) in the upper left of FIG. 8 is an example of the "object-based potential map" at time t1. Speakers, monitors, decorative objects, and the like are placed on the stage, and some of these are moved or replaced as the live progresses.
  • the "object-based potential map” has different colors for each area according to the area (no-driving area, cautionary driving area, permitted driving area) determined according to the distance from the object placement position as shown below. is a map.
  • the close distance position of the object arrangement position is set to red as a travel prohibited area.
  • a middle-distance position of the object arrangement position is set to yellow as a driving caution area.
  • the long-distance position of the object placement position is set to blue as the travel permissible region.
  • the map (t2) on the upper right of FIG. 8 is an example of the "object-based potential map" at time t2 after a certain period of time has elapsed from time t1.
  • the object on the stage has been moved or replaced to a different position than at time t1.
  • the setting of these three areas ie, the travel prohibited area (red), the travel caution area (yellow), and the travel permitted area (blue), also changes.
  • the map (t3) in the lower left of FIG. 8 is an example of the "object-based potential map" at time t3 after a certain period of time has elapsed from time t2.
  • the object on the stage has been moved or replaced to a position different from that at times t1 and t2.
  • the setting of these three areas ie, the travel prohibited area (red), the travel caution area (yellow), and the travel permitted area (blue), also changes.
  • the map (t4) in the lower right of FIG. 8 is an example of the "object-based potential map" at time t4 after a certain period of time has elapsed from time t3.
  • the object on the stage has moved or been replaced to a position different from that at times t1-t3.
  • the setting of these three areas ie, the travel prohibited area (red), the travel caution area (yellow), and the travel permitted area (blue), also changes.
  • the "object-based potential map” moves each area (no driving area, caution driving area, permitted driving area) according to the position of objects on the stage from the start time to the end time of the live (performance). It is a time-series map that changes dynamically.
  • on-stage object placement schedule data indicating the placement positions of objects from the start to the end of a live (performance) is stored in the C.I. It is generated and recorded in advance as on-stage object placement schedule data 113 .
  • the individual potential map generation unit 120 stores the C.I.
  • the on-stage object placement schedule data 113 is acquired, the acquired data is referenced, and each area (no-driving area, caution-driving area, permissible-driving area) is dynamically changed from the start to the end of the live (performance).
  • Generate a series map c “object-based potential map”.
  • map a performer base potential map
  • map b lighting base potential map
  • map c object-based potential map
  • the potential map synthesizing unit 130 generates a pre-generated potential map 150 by synthesizing the three individual potential maps generated by the individual potential map generating unit 120 .
  • the pre-generated potential map 150 is also a time-series map in which each region (driving prohibited region, cautionary driving region, permitted driving region) is dynamically changed from the start to the end of the live (performance).
  • FIG. 1 A specific example of the pre-generated potential map 150 generated by the potential map synthesizing unit 130 is shown in FIG.
  • FIG. 9 shows an example of pre-generated potential maps 150 at four timings (t1 to t4) similar to the individual potential maps described with reference to FIGS. 6 to 8.
  • FIG. 9 shows an example of pre-generated potential maps 150 at four timings (t1 to t4) similar to the individual potential maps described with reference to FIGS. 6 to 8.
  • the pre-generated potential maps for the four timings (t1 to t4) shown in FIG. 9 are respectively generated by synthesizing the individual potential maps for the same four timings (t1 to t4) shown in FIGS. 6 to 8 for each timing. It is a map that
  • the map (t1) in the upper left of FIG. 9 is the “pre-generated potential map” at time t1.
  • This "pre-generated potential map” at time t1 is a performer base potential map at time t1 shown in FIG. 6 (t1); an illumination base potential map at time t1 shown in FIG. 7 (t1); Object-based potential map at time t1 shown in FIG. 8 (t1), This map is obtained by synthesizing these three individual potential maps at the same timing (t1).
  • a specific synthesizing process sequence will be described later, but for example, the synthesizing process is performed by the following process.
  • Such quantification is performed, the numerical values of each region of each individual potential map are added, and a pre-generated potential map as a composite map is generated based on the addition result.
  • an area with an additional value of 10 or more is a driving prohibited area
  • an area with an additional value of 5 or more and less than 10 is a driving caution area
  • an area with an additional value of less than 5 is a driving permitted area.
  • the "pre-generated potential map" shown in FIG. 9 (t1) is a performer base potential map at time t1 shown in FIG. 6 (t1); an illumination base potential map at time t1 shown in FIG. 7 (t1); Object-based potential map at time t1 shown in FIG. 8 (t1), A map generated by digitizing and adding each area (driving prohibition area, caution driving area, driving allowable area) of the individual potential map at the same timing (t1), and performing area division based on the above added value. be.
  • Pre-generated potential map considers all of the performer's position, lighting conditions (lighting position, brightness, color), and object placement position, and determines the driving prohibited area, driving caution area, and driving permitted area. This is a map in which the following different colors are assigned for each area.
  • the no-travel area is set in red.
  • the driving caution area is set to yellow.
  • the permissible travel area is set in blue.
  • the map (t2) in the upper right of FIG. 9 is an example of a composite map at time t2 after a certain period of time has elapsed from time t1, that is, a "previously generated potential map.”
  • the performer's position, lighting conditions (lighting position, brightness, color), and object placement positions are set differently from those at time t1.
  • the "pre-generated potential map" at time t2 is a performer base potential map at time t2 shown in FIG. 6 (t2); an illumination base potential map at time t2 shown in FIG. 7 (t2); Object-based potential map at time t2 shown in FIG. 8 (t2), This map is generated by quantifying and adding the respective regions (driving prohibited region, cautionary driving region, and permitted driving region) of these three individual potential maps at the same timing (t2), and performing region division based on the added value.
  • the map (t3) in the lower left of FIG. 9 is an example of a synthesized map at time t3 after a certain period of time has elapsed from time t2, that is, a "previously generated potential map.”
  • the performer's position, lighting conditions (lighting position, brightness, color), and object placement positions are set differently from those at times t1 and t2.
  • the "pre-generated potential map" at time t3 is a performer base potential map at time t3 shown in FIG. 6 (t3); an illumination base potential map at time t3 shown in FIG. 7 (t3); Object-based potential map at time t3 shown in FIG. 8 (t3), This map is generated by quantifying and adding the respective regions (driving prohibited region, cautionary driving region, and permitted driving region) of these three individual potential maps at the same timing (t3), and performing region division based on the added value.
  • the map (t4) in the lower right of FIG. 9 is an example of a composite map at time t4 after a certain period of time has elapsed from time t3, that is, a "previously generated potential map.”
  • the performer's position, lighting conditions (lighting position, brightness, color), and object placement positions are set differently from those at times t1 to t3.
  • the "pre-generated potential map" at time t4 is a performer base potential map at time t4 shown in FIG. 6 (t4); an illumination base potential map at time t4 shown in FIG. 7 (t4); Object-based potential map at time t4 shown in FIG. 8 (t4), This map is generated by quantifying and adding the respective regions (driving prohibited region, cautionary driving region, and permitted driving region) of these three individual potential maps at the same timing (t4), and performing region division based on the added value.
  • the potential map synthesizing unit 130 digitizes and adds each region (driving prohibited region, cautionary driving region, permissible driving region) of a plurality of individual potential maps at the same timing, performs region segmentation based on the added value, and generates in advance. Generate a potential map 150 .
  • the pre-generated potential map 150 generated by the potential map synthesizing section 130 is provided to the driving route generating section 160, and the driving route generating section 160 generates a map from the start to the end of the live (performance) based on the pre-generated potential map 150.
  • a travel route of the image capturing robot 50 is determined.
  • the travel route generating unit 160 receives the pre-generated potential map 150 generated by the potential map synthesizing unit 130, and the image capturing robot 50 performs live (performance) from start to finish.
  • Running route information 165 is generated in which a route for selecting and running an area that is not conspicuous due to illumination is selected without colliding with objects on the stage or on the stage.
  • the travel route generator 160 for example, creates a travel route that travels only in the permissible travel area in the pre-generated potential map 150.
  • the generated travel route is provided to the travel control unit 170 that controls travel of the image capturing robot 50 , and the travel control unit 170 causes the image capturing robot 50 to travel according to the generated travel route information 165 .
  • the traveling control unit 170 may be configured as an information processing device within the image capturing robot 50 or may be configured as an information processing device capable of communicating with a robot outside the image capturing robot 50 .
  • the image capturing robot 50 can be placed on the performer or on the stage from the start to the end of the live (performance). It is possible to select and run an area that is not conspicuous due to illumination without colliding with objects.
  • FIG. 11 shows an example of the travel route of the image capturing robot 50 from time (t3) to (t4).
  • the travel route of the image capturing robot 50 from time (t3) to (t4) shown in FIG. is.
  • the performer and objects on the stage will be affected from the start to the end of the live (performance). It is possible to select and run an area that is not conspicuous by lighting without colliding.
  • FIG. 12 Sequence for generating map a "performer base potential map”
  • Fig. 13 to Fig. 14 Sequence for generating map b "illumination base potential map”
  • Fig. 15 Map c "object base potential map” ” generation sequence
  • FIG. 16 is a processing sequence for generating a "previously generated potential map", which is a synthetic map, executed by the potential map synthesizing unit 130 of the information processing apparatus 100 shown in FIG.
  • the processing according to the flow described below can be executed, for example, according to a program stored in the storage unit of the information processing device, under the control of a control unit having a program execution function such as a CPU. is executed in Details of the processing of the flow shown in FIG. 14 will be sequentially described below.
  • step S11 executed by the individual potential map generation unit 120 described above with reference to FIG. Equivalent to detailed sequence.
  • map a base potential of the performer, which is a time-series map in which each area (no-driving area, caution-not-driving area, permitted-driving area) is dynamically changed according to the position of the performer from the start to the end of the live performance. 4 is a detailed sequence of map generation processing;
  • This is, for example, performer action schedule data generated based on a preset live program, that is, "A. performer action schedule data 111" stored in the storage unit 110 of the information processing apparatus 100 shown in FIG. Executed as a process to get from
  • Step S102 the data processing unit of the information processing device 100 selects one performer P to be analyzed.
  • step S103 the data processing unit of the information processing apparatus 100 acquires time-series behavior data of the analysis target performer P from the start to the end of the live performance.
  • This process also includes, for example, performer action schedule data generated based on a preset live program, that is, “A. ” is executed as a process to obtain from
  • Step S104 the data processing unit of the information processing apparatus 100 generates a potential map based on the time-series action data of the performer P to be analyzed from the start to the end of the live performance.
  • the close distance position of the position of the performer P is set to red as a travel prohibition area.
  • the medium-distance position of the position of the performer P is set to yellow as a driving caution area.
  • the long-distance position of the position of the performer P is set in blue as the travel-permissible region.
  • step S105 the data processing unit of the information processing device 100 determines the presence or absence of an analysis-unprocessed performer. That is, it is determined whether or not generation of all performer base potential maps for the number of performers na acquired in step S101 has been completed.
  • step S105 If there is an unprocessed performer, the determination in step S105 is Yes. In this case, the processing of step S102 and subsequent steps is executed for the unprocessed performers. On the other hand, if there is no unprocessed performer, that is, if it is determined that generation of all performer base potential maps for the number of performers na obtained in step S101 has been completed, the determination in step S105 is No. In this case, the process proceeds to step S106.
  • Step S106 When the generation of all actor-based potential maps is completed, the data processing unit of the information processing apparatus 100 executes the processing from step S106 onwards.
  • step S107 the data processing unit of the information processing apparatus 100 adds the numerical values corresponding to the regions of the potential maps of all the performers 1 to na for each region to calculate the added value corresponding to the regions.
  • Step S108 the data processing unit of the information processing apparatus 100 resets the area divisions based on the added values corresponding to the areas of the potential maps of all the performers 1 to na.
  • step S109 the data processing unit of the information processing apparatus 100 outputs the potential map whose area segmentation has been reset in step S108 to the potential map synthesizing unit 130 as a "performer base potential map.”
  • map a is a time-series map in which each area (driving prohibited area, cautionary driving area, permitted driving area) is dynamically changed according to the performer's position from the start to the end of the live (performance). A base potential map is generated.
  • step S12 executed by the individual potential map generator 120 described above with reference to FIG. It corresponds to the detailed sequence of generation processing.
  • a map b which is a series map, is a detailed sequence of processing for generating an illumination base potential map.
  • This processing is performed, for example, by the lighting control schedule data generated based on a preset live program, that is, the “B. 112”.
  • Step S122 the data processing unit of the information processing device 100 selects one illumination position segmented region Q to be analyzed.
  • step S123 the data processing unit of the information processing apparatus 100 acquires time-series lighting brightness information from the start to the end of the live performance of one lighting position segmented region Q selected as an analysis target.
  • step S124 the data processing unit of the information processing device 100 generates a potential map based on the time-series lighting brightness data from the start to the end of the live performance of one lighting position segmented region Q selected as an analysis target. .
  • the following area setting and color setting are performed.
  • the brightness of the illumination position segmented region Q is equal to or higher than the specified threshold value Thd1, it is set to red as a travel-prohibited region.
  • the brightness of the lighting position segmented region Q is within the range of the prescribed threshold values Thd1 and Thd2, it is set to yellow as the driving caution region. If the brightness of the illumination position segmented region Q is equal to or less than the specified threshold value Thd2, it is set to blue as the travel-permissible region.
  • an illumination brightness base potential map corresponding to one illumination position segmented area Q is generated.
  • step S125 the data processing unit of the information processing apparatus 100 determines whether or not there is an unanalyzed illumination position segmented area. In other words, it is determined whether or not the generation of the lighting brightness base potential maps for all of the number of lighting position segmented regions nb obtained in step S121 has been completed.
  • step S125 If there is an unprocessed illumination position segmented area, the determination in step S125 is Yes. In this case, the processing from step S122 onward is executed for the unprocessed illumination position segmented regions. On the other hand, if there is no unprocessed illumination position segmented region, that is, if it is determined that the illumination brightness base potential map generation for all the illumination position segmented regions nb acquired in step S121 has been completed, the determination in step S125 is No. In this case, the process proceeds to step S126.
  • Step S126 When the generation of all lighting brightness base potential maps is completed, the data processing unit of the information processing device 100 executes the processing from step S126 onward.
  • step S126 one illumination position segmented area Q to be analyzed is selected.
  • step S127 the data processing unit of the information processing apparatus 100 acquires time-series lighting color information from the start to the end of the live performance of one lighting position segmented region Q selected as an analysis target.
  • Step S128 the data processing unit of the information processing device 100 generates a potential map based on the time-series illumination light color data from the start to the end of the live performance of one illumination position segmented region Q selected as an analysis target. .
  • the following area setting and color setting are performed.
  • red is set as the travel prohibition area.
  • the illumination light color of the illumination position segmented area Q is a color different from that of the image capturing robot, if it is not a similar color, it is set to yellow as a travel caution area.
  • blue is set as the travel permitted area.
  • an illumination color base potential map corresponding to one illumination position segmented area Q is generated.
  • step S129 the data processing unit of the information processing device 100 determines whether or not there is an unanalyzed illumination position segmented area. That is, it is determined whether or not all the lighting color base potential maps for the lighting position segmented region number nb obtained in step S121 have been generated.
  • step S129 If there is an unprocessed illumination position segmented area, the determination in step S129 is Yes. In this case, the processing from step S126 onward is executed for the unprocessed illumination position segmented regions. On the other hand, if there is no unprocessed illumination position segmented region, that is, if it is determined that the illumination color base potential map generation for all the illumination position segmented regions nb acquired in step S121 is completed, the determination in step S129 is No. becomes. In this case, the process proceeds to step S131.
  • Step S131 When the generation of all lighting color base potential maps is completed, the data processing unit of the information processing device 100 executes the processing from step S131 onward.
  • Step S132 the data processing unit of the information processing apparatus 100 adds the numerical values corresponding to the set areas of the illumination brightness base potential map and the illumination color base potential map for each area, Calculate the value.
  • the map is time-series data, ie, run for all stage positions on the map at all times.
  • Step S133 the data processing unit of the information processing apparatus 100 resets the area divisions based on the added values corresponding to the areas of the illumination brightness base potential map and the illumination color base potential map.
  • Step S134 the data processing unit of the information processing apparatus 100 outputs the potential map whose area division has been reset in step S133 to the potential map synthesizing unit 130 as an "illumination base potential map.”
  • map b illumination base potential map generation processing executed by the individual potential map generation unit 120 .
  • each area no driving area, caution driving area, permitted driving area
  • a map b illumination base potential map, which is a time-series map, is generated.
  • processing according to the flowchart shown in FIG. 15 is the processing of step S13 executed by the individual potential map generation unit 120 described above with reference to FIG. Equivalent to detailed sequence.
  • Objects are objects placed on the stage, such as speakers, monitors, and decorative objects.
  • This is, for example, stage object placement schedule data generated based on a preset live program, that is, "C.
  • Stage object placement schedule data stored in the storage unit 110 of the information processing apparatus 100 shown in FIG. It is executed as a process of acquiring from the schedule data 113.
  • Step S152 the data processing unit of the information processing device 100 selects one object Ob to be analyzed.
  • Step S153 the data processing unit of the information processing apparatus 100 acquires time-series arrangement data from the start to the end of the live performance of the object Ob to be analyzed.
  • This process is also performed, for example, on-stage object placement schedule data generated based on a preset live program, that is, "C. on-stage object arrangement data" stored in the storage unit 110 of the information processing apparatus 100 shown in FIG. It is executed as a process of acquiring from the arrangement schedule data 113.
  • Step S154 the data processing unit of the information processing device 100 generates a potential map based on the time-series arrangement data from the start to the end of the live performance of the object Ob to be analyzed.
  • the short distance position of the arrangement position of the object Ob is set to red as a travel prohibited area.
  • the middle distance position of the arrangement position of the object Ob is set to yellow as a driving caution area.
  • the long-distance position of the arrangement position of the object Ob is set to blue as the travel-permissible region.
  • step S155 the data processing unit of the information processing apparatus 100 determines whether or not there is an analysis-unprocessed object. That is, it is determined whether generation of all object-based potential maps for the number of objects nc acquired in step S151 has been completed.
  • step S155 If there is an unprocessed object, the determination in step S155 is Yes. In this case, the processing of step S152 and subsequent steps is executed for the unprocessed object. On the other hand, if there are no unprocessed objects, that is, if it is determined that generation of all object-based potential maps for the number of objects nc acquired in step S151 has been completed, the determination in step S155 is No. In this case, the process proceeds to step S156.
  • Step S156 When the generation of all object-based potential maps is completed, the data processing unit of the information processing device 100 executes the processes from step S156 onward.
  • step S157 the data processing unit of the information processing apparatus 100 adds the numerical values corresponding to the regions of the potential maps of all the objects 1 to nc for each region to calculate the added value corresponding to the regions.
  • Step S158 the data processing unit of the information processing apparatus 100 resets the area divisions based on the added values corresponding to the areas of the potential maps of all the objects 1 to nc.
  • Step S159 the data processing unit of the information processing apparatus 100 outputs the potential map whose area division has been reset in step S158 to the potential map synthesizing unit 130 as an "object-based potential map.”
  • map c object-based potential map generation processing executed by the individual potential map generation unit 120 .
  • the map c which is a time-series map in which each area (driving prohibition area, caution driving area, driving permitted area) is dynamically changed according to the object arrangement position from the start to the end of the live (performance). An object-based potential map is generated.
  • the flowchart shown in FIG. 16 is a sequence of generating a pre-generated potential map executed by the potential map synthesizing unit 130 .
  • processing of each step of the flow shown in FIG. 16 will be described in order.
  • step S174 the data processing unit of the information processing apparatus 100 generates the potential map with the area segmentation reset in step S173 as a synthesized map, that is, a "previously generated potential map.”
  • the pre-generated potential map 150 generated by the potential map synthesizing section 130 is provided to the driving route generating section 160, and the driving route generating section 160 generates a map from the start to the end of the live (performance) based on the pre-generated potential map 150.
  • a travel route of the image capturing robot 50 is determined.
  • the travel route generating unit 160 receives the pre-generated potential map 150 generated by the potential map synthesizing unit 130, and the image capturing robot 50 performs live (performance).
  • Running route information 165 is generated in which a route for selecting and running an area that is not conspicuous by illumination without colliding with performers or objects on the stage from the start to the end of is set. For example, a travel route that travels only in the permissible travel area in the pregenerated potential map 150 is generated.
  • the generated travel route is provided to the travel control unit 170 that controls travel of the image capturing robot 50 , and the travel control unit 170 causes the image capturing robot 50 to travel according to the generated travel route information 165 .
  • the image capturing robot 50 can be placed on the performer and on the stage from the start to the end of the live (performance). It is possible to select and run an area that is not conspicuous due to illumination without colliding with objects.
  • the travel route of the image capturing robot 50 generated using the pre-generated potential map 150 can be displayed as a simulation image on the display unit of the information processing device 100, for example.
  • the data processing unit of the information processing device 100 displays the planned travel route of the image capturing robot 50 from the start to the end of the live performance based on the travel route information 165 generated using the generated pre-generated potential map 150. Generate simulation data for A specific display example of this simulation data is shown in FIG.
  • FIG. 17 shows an example of display data indicating the running positions of the image capturing robot 50 at time t3 and time t4, as a display example of simulation data.
  • a user such as a robot control operator can confirm the travel route generated using the generated pre-generated potential map 150 .
  • map a performer base potential map
  • map b lighting base potential map
  • map c object-based potential map
  • the generated pre-generated potential map is used to determine the traveling route of the image capturing robot 50. It was a configuration that determined the For example, it is configured to perform processing for selecting a travel-permissible region in the pre-generated potential map and determining a travel route for the image capturing robot 50 .
  • a fourth individual potential map is created in consideration of the line of sight of the spectators and the line of sight of the television camera on the side of the spectator seats. This is an embodiment for generating a pre-generated potential map that also considers the 4 individual potential maps.
  • the image capturing robot 50 shown in FIG. 18 is a mobile device on which a camera is mounted, that is, a traveling robot, similar to the image capturing robot 50 shown in FIG. Shoot from an angle.
  • the image capturing robot 50 may enter between the performer 20 and the audience 30 or TV camera 31 on the audience seat side.
  • FIG. 19 shows an example of a live venue where the processing of the second embodiment is performed. As shown in FIG. 19, on a stage 10 where a live performance is actually performed, a speaker 12, a monitor 13, and various decoration objects 14 are placed.
  • the performer 20 moves around the stage as the live performance progresses, and the position, brightness, and color of the lighting 11 change according to this movement.
  • the decoration object 14 is also changed in various ways.
  • FIG. 19 there are many spectators 30 on the side of the audience seats, and a TV camera 31 is also arranged to photograph the live performance of the performer 20 .
  • a potential map is generated in consideration of the viewpoint positions of some of the spectators 30, that is, the priority spectators 35 shown in the figure and the viewpoint positions of the TV camera 31.
  • a potential map is generated in consideration of the viewpoint positions of all spectators, a potential map is generated in which the entire stage is set as a travel-prohibited area, and there is almost no travel-permissible area for the image capturing robot 50. Processing considering the viewpoint position of the priority spectator 35 is performed.
  • the priority spectator 35 shown in the drawing is, for example, a seat reserved as a seat for related persons, and a reserved seat already reserved as a reserved seat for an important audience.
  • a potential map is generated in consideration of the viewpoint position of the priority spectator 35 and the viewpoint position of the TV camera 31 . Details of the second embodiment will be described with reference to FIG. 20 and subsequent drawings.
  • FIG. 20 is a diagram illustrating a configuration example of the information processing apparatus 100b according to the second embodiment of the present disclosure.
  • the information processing device 100b may be configured inside the mobile device running on the stage shown in FIGS. or an independent device from the image capturing robot 50.
  • FIG. 20 is a diagram illustrating a configuration example of the information processing apparatus 100b according to the second embodiment of the present disclosure.
  • the information processing device 100b may be configured inside the mobile device running on the stage shown in FIGS. or an independent device from the image capturing robot 50.
  • the information processing device 100 b has a storage section 110 , an individual potential map generation section 120 , a potential map synthesis section 130 and a travel route generation section 160 .
  • These basic configurations are the same as those of the information processing apparatus 100 previously described with reference to FIG.
  • the information processing device 100b shown in FIG. 20 may also be configured inside the moving device traveling on the stage shown in FIGS. 5, it may be a device independent of the image capturing robot 50, such as a device (server) capable of communicating with the image capturing robot 50.
  • a device server
  • the storage unit 110 stores the following four data.
  • A. performer action schedule data 111 B. stage lighting control schedule data 112
  • C. On-stage object placement schedule data 113 D. Priority spectators, TV camera viewpoint position data 114,
  • these four scheduled data are data that are prepared in advance and stored in the storage unit 110 before the start of the live performance. That is, it is schedule data prepared in advance according to a program such as a live progress table prepared before the start of the live performance.
  • Data A to C are the same data as the data described with reference to FIG. 4 in the first embodiment. i.e. A.
  • the performer action schedule data 111 is time-series position data of performers moving around on stage during a live performance. That is, it is time-series position data of the performer from the start to the end of the live (performance).
  • the stage lighting control schedule data 112 is time-series data of lighting control information including lighting setting information such as the lighting position, brightness, and color of the lighting from the start to the end of the live (performance).
  • the on-stage object placement schedule data 113 is time-series data of on-stage object placement position information including the placement positions of on-stage objects from the start to the end of a live performance (performance).
  • the on-stage objects include speakers, monitors, decorative objects, and the like placed on the stage.
  • Example 2 in addition to these data A to C, D. Priority spectators, TV camera viewpoint position data 114, This data D is stored in the storage unit 110 .
  • the priority spectator/TV camera viewpoint position data 114 is the viewpoint position data of the priority spectator on the spectator seat side from the start to the end of the live (performance) and the viewpoint position data of the TV camera.
  • the priority spectator and TV camera viewpoint position data 114 are chronological data that dynamically change with the passage of time. However, when the viewpoint position of the priority audience and the viewpoint position of the TV camera are at the same position from the start to the end of the live (performance), D.
  • the priority spectator and TV camera viewpoint position data 114 can be one fixed data.
  • the data A to D stored in the storage unit 110 are used in the individual potential map generation unit 120.
  • FIG. The individual potential map generation unit 120 uses the four types of data A, B, C, and D individually to generate the following four types of individual potential maps.
  • map a performer base potential map
  • map b lighting base potential map
  • map c object-based potential map
  • map d priority spectator, TV camera viewpoint position base potential map,
  • Maps a to c are the individual potential maps previously described in the first embodiment.
  • Map b lighting base potential map shows each area (no driving area, caution driving area, permitted driving area) according to the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of the live (performance). This is a dynamically changed time series map.
  • Map c Object-based potential map is a time series map that dynamically changes each area (no driving area, caution driving area, permitted driving area) according to the object placement position from the start to the end of the live (performance) is.
  • Map d priority spectator, TV camera viewpoint position base
  • the potential map is a map in which each area (driving prohibited area, caution caution area, permitted driving area) is set according to the priority spectator on the spectator seat side and TV camera viewpoint position. be.
  • this map changes the setting area (running prohibited area) with the passage of time.
  • driving caution area, driving permissible area) becomes time-series data that dynamically changes.
  • the viewpoint position of the priority spectators and the viewpoint position of the TV camera are at the same position from the start to the end of the live performance (performance), the set areas (no driving area, caution driving area, permitted driving area) will move. It becomes one map that does not change dynamically.
  • the individual potential map generator 120 generates the following four maps in steps S11 to S14 shown in FIG. i.e.
  • step S14 the individual potential map generation unit 120 generates a moving image from a short-distance position to a long-distance position on a straight line connecting the priority audience viewpoint position and the TV camera viewpoint position to the stage center position.
  • Generate a map d priority spectator, TV camera viewpoint position base potential map divided into prohibited area (red), driving caution area (yellow), and driving permitted area (blue).
  • the individual potential map generator 120 generates the following four individual potential maps.
  • map a performer base potential map
  • map b lighting base potential map
  • map c object-based potential map
  • map d priority spectator, TV camera viewpoint position base potential map
  • the potential map synthesizing unit 130 performs synthesizing processing of the four individual potential maps generated by the individual potential map generating unit 120 to generate pre-generated potential maps b, 150b.
  • the pre-generated potential maps b, 150b are also time-series maps in which each region (driving prohibited region, cautionary driving region, permitted driving region) is dynamically changed from the start to the end of the live (performance).
  • the “pre-generated potential map b” generated in the second embodiment takes into consideration all of the performer position, lighting conditions (lighting position, brightness, color), object placement position, and the priority audience viewpoint and TV camera viewpoint.
  • This is a map in which a driving prohibited area, a driving caution area, and a driving permitted area are determined by using the map, and the following different colors are assigned for each determined area.
  • the no-travel area is set in red.
  • the driving caution area is set to yellow.
  • the permissible travel area is set in blue.
  • the pre-generated potential map b, 150b generated by the potential map synthesizing unit 130 is provided to the driving route generating unit 160, and the driving route generating unit 160 starts the live (performance) based on the pre-generated potential map b, 150b.
  • a running route of the image capturing robot 50 from 1 to 1 is determined.
  • the running route generating unit 160 receives the pre-generated potential maps b and 150b generated by the potential map synthesizing unit 130, and the image capturing robot 50 collides with performers and objects on the stage from the start to the end of the live (performance). Also, the driving route information 165 is generated in which a route for driving is set by selecting an area that is not conspicuous due to lighting and an area that does not interfere with the line of sight of the priority spectators 35 or the photographing of the TV camera 31. .
  • the travel route generation unit 160 for example, generates a travel route that travels only in the permissible travel area in the pre-generated potential map b, 150b.
  • the generated travel route is provided to the travel control unit 170 that controls travel of the image capturing robot 50 , and the travel control unit 170 causes the image capturing robot 50 to travel according to the generated travel route information 165 .
  • the image capturing robot 50 can be used as a live (performance) map. From the start to the end of , select and run an area that does not collide with performers or objects on the stage, does not stand out due to lighting, and does not interfere with the line of sight of the priority audience 35 or the shooting of the TV camera 31. becomes possible.
  • Step S202 the data processing unit of the information processing apparatus 100 selects one priority spectator viewpoint or TV camera viewpoint S to be analyzed.
  • step S203 the data processing unit of the information processing apparatus 100 acquires time-series position data from the start of the live performance to the end of the priority audience viewpoint or the TV camera viewpoint S to be analyzed.
  • This processing is also performed, for example, based on the priority spectator viewpoint and TV camera viewpoint position data generated based on a preset live program, that is, the "D. Priority audience, TV camera viewpoint position data 114”.
  • the time-series position data from the live start to the end of the priority audience viewpoint or the TV camera viewpoint S may be data that dynamically changes with the passage of time, or one fixed data that does not change. Sometimes.
  • Step S204 the data processing unit of the information processing apparatus 100 generates a potential map based on the position data from the live start to the end of the priority audience viewpoint or TV camera viewpoint S selected as the analysis target.
  • each area (travel prohibition area, travel caution area, travel permission area) is set according to the map as described above with reference to FIG. Generate a map.
  • a priority spectator/TV camera viewpoint base potential map corresponding to one priority spectator viewpoint or TV camera viewpoint S to be analyzed is generated.
  • step S205 the data processing unit of the information processing apparatus 100 determines whether or not there is a priority spectator viewpoint or a TV camera viewpoint that has not been analyzed yet. That is, it is determined whether or not generation of the priority spectator/TV camera viewpoint base potential map for all the priority spectators and the TV camera viewpoint number nd acquired in step S201 is completed.
  • step S205 If there is an unprocessed priority audience viewpoint or TV camera viewpoint, the determination in step S205 is Yes. In this case, the processing from step S202 onward is executed for the unprocessed priority spectator viewpoint or TV camera viewpoint. On the other hand, if there are no unprocessed priority spectator viewpoints or TV camera viewpoints, that is, generation of the priority spectator and TV camera viewpoint base potential map for all of the priority spectator and TV camera viewpoints nd acquired in step S201 is completed. If so, the determination in step S205 is No. In this case, the process proceeds to step S206.
  • Step S206 When the generation of the base potential maps for all priority spectators and TV camera viewpoints is completed, the data processing unit of the information processing apparatus 100 executes the processing from step S206 onwards.
  • step S207 the data processing unit of the information processing apparatus 100 adds the numerical values corresponding to the regions of the potential maps of all the priority spectator viewpoints or the TV camera viewpoints 1 to nd for each region, Calculate the added value.
  • the total number of priority spectator and TV camera viewpoints is 3, and individual priority spectator and TV camera viewpoint base potential maps (m1 to m3) corresponding to the three priority spectator viewpoints or TV camera viewpoints are generated.
  • Such addition processing is performed for all maps. Note that when the map is time-series data, all stage positions of the map at all times are executed.
  • step S208 the data processing unit of the information processing apparatus 100 resets the area divisions based on the added values corresponding to the areas of the potential maps of all the priority spectator viewpoints or the TV camera viewpoints 1 to nd.
  • step S209 the data processing unit of the information processing apparatus 100 outputs the potential map whose area segmentation has been reset in step S208 to the potential map synthesizing unit 130 as a "priority spectator, TV camera viewpoint base potential map".
  • the flowchart shown in FIG. 23 is the sequence of generating the pre-generated potential map b executed by the potential map synthesizing unit 130 .
  • processing of each step of the flow shown in FIG. 23 will be described in order.
  • Such addition processing is performed for all maps.
  • the map is time-series data, ie, run for all stage positions on the map at all times.
  • step S224 the data processing unit of the information processing apparatus 100 generates the potential map with the area segmentation reset in step S223 as a synthesized map, ie, "previously generated potential map b".
  • the pre-generated potential maps b and 150b generated by the potential map synthesizing section 130 are provided to the driving route generating section 160, and the driving route generating section 160 performs live (performance) from the start to the end based on the pre-generated potential map 150. Determine the travel route of the image capturing robot 50 up to .
  • the travel route generating unit 160 receives the pre-generated potential maps b and 150b generated by the potential map synthesizing unit 130, and the image capturing robot 50 performs live ( (Performance) from start to finish without colliding with performers or objects on stage, select areas that will not be conspicuous due to lighting, and will not obstruct the view of the priority audience or TV cameras.
  • the travel route information 165 that sets the route to be taken is generated. For example, a travel route that travels only in the permissible travel area in the pregenerated potential map 150 is generated.
  • the generated travel route is provided to the travel control unit 170 that controls travel of the image capturing robot 50 , and the travel control unit 170 causes the image capturing robot 50 to travel according to the generated travel route information 165 .
  • the image capturing robot 50 can be placed on the performer or on the stage from the start to the end of the live (performance). It is possible to select and run an area that does not collide with an object, is not conspicuous due to illumination, and does not obstruct the view of priority spectators or TV cameras.
  • Example 3 Regarding an example of generating a real-time data reflection potential map using information during live performance
  • Example 3 an example of generating a real-time data reflection potential map using information during live performance will be described.
  • data that can be acquired before a live performance on the stage is started that is, data stored in the storage unit 110 of the information processing apparatus 100 shown in FIGS.
  • a potential map was generated using the following data.
  • Each of the above data A to D used in the first and second embodiments is generated on the premise that the performer acts according to the live program, lighting control is performed, and objects are arranged. data. However, when a live performance actually starts, the performer's movement, lighting control, and the like may be performed differently from the original live program.
  • Embodiment 3 described below is intended to prevent such a situation, and is an embodiment in which a potential map is generated by acquiring real-time data during an actual live performance, i.e., when performers are performing. be.
  • FIG. 24 shows a configuration example of the information processing apparatus 200 of the third embodiment.
  • the information processing device 200 includes a real-time stage information acquisition unit 201, a real-time spectator seat information acquisition unit 202, a real-time net information acquisition unit 203, a real-time stage information analysis unit 204, a storage unit 205, and a pre-generated potential map correction unit. It has a unit 206 , a real-time attention area analysis unit 207 , and a real-time data reflection potential map generation unit 208 .
  • the information processing apparatus 200 shown in FIG. 24 may be configured, for example, inside the moving apparatus running on the stage shown in FIGS.
  • a device independent of the image capturing robot 50 such as a device that can communicate with the image capturing robot 50, may be used.
  • FIG. 25 shows a configuration example of an information processing system in which the information processing apparatus having the configuration shown in FIG. 24 is independent of the image capturing robot 50 .
  • an information processing device (server) 200 a live venue information acquisition device 60, an image capturing robot 50 in the live venue, and an SNS server 290 are connected via a communication network.
  • Build system 280 a communication network.
  • the information processing apparatus (server) 100 has the configuration shown in FIG.
  • the live venue information acquisition device 60 is composed of a camera that captures images of the live venue, a microphone that acquires audio information of the live venue, an illuminance meter that detects lighting conditions, a color analysis device, etc., and acquires information such as the acquired image. , to the information processing apparatus (server) 200 via the communication network.
  • the information processing device (server) 200 analyzes information received from the live venue information acquisition device 60 and information acquired from the SNS server 290, and determines the travel route of the image capturing robot 50 in the live venue. A map (potential map) generation process, a travel route determination process using the map, and a travel control information generation process for the image capturing robot 50 according to the determined travel route are performed.
  • the SNS server 290 is a server that collects comment information such as tweets of users browsing the live on the Internet.
  • the information processing device (server) 200 analyzes the information acquired from the SNS server 290 and analyzes the attention area of the user who is watching the live performance via the net.
  • the information processing device (server) 200 further transmits the generated travel control information to the image capturing robot 50 via the communication network.
  • the image capturing robot 50 runs on the stage according to the running control information received from the information processing device (server) 100 . For example, it is possible to perform processing using such an information processing system.
  • the real-time stage information acquisition unit 201 is configured by, for example, a camera, an illuminance meter, and the like, and acquires in real time information about an actual live performance, that is, a stage where a performer is performing.
  • the real-time stage information acquisition unit 201 acquires the following information as real-time data, as shown in the drawing. performer position 211, lighting conditions 212; on-stage object position 213; image capturing robot position 214;
  • the illumination state includes information on the position of the illuminated stage, the brightness of the illumination, and the color of the illumination.
  • the real-time stage information acquisition unit 201 continuously acquires the performer position 211, the lighting state 212, the on-stage object position 213, the image capturing robot position 214, and each of these information as real-time data during the live performance period. is output to the real-time stage information analysis unit 204 .
  • the real-time stage information analysis unit 204 acquires the performer position 211, the lighting state 212, the on-stage object position 213, and the image capturing robot position 214 input from the real-time stage information acquisition unit 201, and information stored in the storage unit 205 in advance. Compare schedule data 231 based on the live program currently in use and the pre-generated potential map 150 generated according to Example 1 or Example 2 above.
  • the schedule data 231 based on the live program stored in the storage unit 205 is the same data as the following information described above with reference to FIGS. 4 and 20.
  • the real-time stage information analysis unit 204 acquires the performer position 211, the lighting state 212, the on-stage object position 213, and the image capturing robot position 214 input from the real-time stage information acquisition unit 201, and information stored in the storage unit 205 in advance.
  • the schedule data 231 based on the live program that is currently running is compared to confirm whether or not there is a difference.
  • the image capturing robot 50 runs along the running route set using the pre-generated potential map 150 generated according to the first or second embodiment, it does not collide with the performer or the object. , and can be driven without being conspicuous by lighting.
  • the image capturing robot 50 when the image capturing robot 50 is run along the running route set using the pre-generated potential map 150 generated according to the first or second embodiment, it will come into contact with the performer or the object. There is a possibility that the driving may be conspicuous due to lighting.
  • the real-time stage information analysis unit 204 acquires the performer position 211, the lighting state 212, the on-stage object position 213, and the image capturing robot position 214 input from the real-time stage information acquisition unit 201, and information stored in the storage unit 205 in advance. If a difference is detected in the schedule data 231 based on the live program that is currently running, furthermore, whether there is any problem in driving the driving route according to the pre-generated potential map 150 generated according to the first or second embodiment. determine whether
  • the pre-potential map correction unit 206 When a correction request for the pre-generated potential map is input from the real-time stage information analysis unit 204, the pre-potential map correction unit 206 reduces the possibility of contact with performers and objects and the possibility of conspicuous running due to lighting. , modify the pre-generated potential map 150 .
  • a pre-potential map correction unit 206 corrects the pre-generated potential map 150 to generate a real-time stage information reflecting potential map.
  • the real-time stage information reflection potential map reflects the real-time performer position 211, the lighting state 212, the on-stage object position 213, and the image capturing robot position 214 acquired by the real-time stage information acquisition unit 201, and each area (running prohibited area, running It is a map in which a caution area and a driving allowable area) are set.
  • FIG. 26 shows the following two potential maps.
  • Pre-generated potential map (2) Real-time stage information reflection potential map
  • the pre-generated potential map is the pre-generated potential map 150 generated according to the first or second embodiment, and is a potential map stored in the storage unit 205 of the information processing apparatus 200 shown in FIG. be.
  • the real-time stage information reflecting potential map is a map generated by the pre-potential map correction unit 206 correcting the pre-generated potential map 150 . That is, each area (no-run area, caution-run area, permitted-run area) reflects the real-time performer position 211, lighting state 212, on-stage object position 213, and image capturing robot position 214 acquired by the real-time stage information acquisition unit 201. ) is reconfigured.
  • a pre-potential map correction unit 206 corrects the pre-generated potential map using real-time stage information to generate a real-time stage information-reflecting potential map.
  • the real-time stage information reflecting potential map generated by the pre-potential map correction unit 206 is a map generated by reflecting the performer's position, lighting state, and object position in real time, and is a map in which the following area setting is performed, for example.
  • Allowable travel area (blue) area far from real-time performer position and object position, and determined to be the least conspicuous based on real-time lighting conditions
  • No-running area (red) area close to real-time performer position and object position, and determined to be conspicuous based on real-time lighting conditions
  • Caution driving area (yellow) Intermediate area between the allowable driving area (blue) and the prohibited driving area (red),
  • the real-time stage information reflecting potential map generated by the pre-potential map correcting unit 206 is output to the real-time data reflecting potential map generating unit 208 .
  • the real-time audience seat information acquisition unit 202 is specifically composed of, for example, a camera, and acquires in real time the information of the audience seats in front of the stage where the performers are performing.
  • the real-time spectator seat information acquisition unit 202 acquires the following information as real-time data, as shown in the figure. Audience line of sight direction 215, Audience seat side TV camera shooting direction 216,
  • the audience viewpoint position 215 is the line-of-sight direction of the audience in front of the stage where the live performance is being performed.
  • the audience seat side TV camera shooting direction 216 is the shooting direction of the TV camera in front of the stage where the live performance is being performed.
  • the real-time net information acquisition unit 203 has a communication unit connected to a communication network such as the Internet, and acquires so-called SNS information 217 such as tweets of many live viewers on the net.
  • the obtained SNS information 217 is input to the real-time attention area analysis unit 207 .
  • the real-time attention area analysis unit 207 analyzes the audience line-of-sight direction 215 input from the real-time audience seat information acquisition unit 202, the audience seat side TV camera shooting direction 216, and the SNS information 217 input from the real-time net information acquisition unit 203. , which part of the stage the audience is looking at, which part of the stage is captured by the TV camera, and which part of the stage the live viewers on the Internet pay attention to.
  • Real-time attention area analysis section 207 estimates the current attention area on the stage based on these analysis results, and outputs the estimated area to real-time data reflection potential map generation section 208 as real-time attention area information 221 shown in the figure. .
  • the real-time data reflecting potential map generation unit 208 receives the following data. (1) “Real-time stage information reflecting potential map” generated by the prior potential map correction unit 206 (2) Real-time attention area information 221 generated by the real-time attention area analysis unit 207 The real-time data reflecting potential map generation unit 208 receives each of these data and generates a real-time data reflecting potential map 230 based on these input data.
  • the real-time data reflecting potential map generation unit 208 sequentially executes the following processes 1 and 2.
  • Process 1 Based on the real-time region-of-interest information 221 generated by the real-time region-of-interest analysis unit 207, a real-time region-of-interest-reflecting potential map that reflects the real-time region of interest is generated.
  • a "real-time stage information reflection potential map" in which each area (travel prohibited area, caution caution area, allowable travel area) is set by reflecting the performer position 211, the lighting state 212, the on-stage object position 213, and the image capturing robot position 214. , the real-time attention area reflecting potential maps generated in process 1 are combined to generate a real-time data reflecting potential map 230 .
  • FIG. 27 shows the following three potential maps.
  • the real-time attention area reflected potential map is a map generated by the above (process 1) executed by the real-time data reflected potential map generation unit 208, and the real-time attention area information 221 generated by the real-time attention area analysis unit 207 is a potential map generated by reflecting the real-time region of interest based on.
  • the real-time region-of-interest-reflecting potential map is a map generated by reflecting real-time audience, TV camera, and net information, and is a map in which the following region settings are made.
  • Permissible driving area (blue) Non-attention area with the lowest attention based on real-time spectators, TV cameras, and net information
  • No-driving areas (red) attention areas with the highest degree of attention based on real-time spectators, TV cameras, and internet information
  • Caution driving area (yellow) Intermediate area between the allowable driving area (blue) and the prohibited driving area (red)
  • the (4) real-time data reflection potential map 230 shown in FIG. 27 is shown in FIG. (2) Real-time stage information reflecting potential map (3) Real-time attention area reflecting potential map This map is generated by synthesizing two potential maps.
  • the real-time data reflection potential map 230 generated by the real-time data reflection potential map generation unit 208 is a map generated by reflecting real-time information on the stage, real-time audience seat information, and real-time net information.
  • the real-time data reflection potential map 230 generated by the real-time data reflection potential map generation unit 208 is a map generated by reflecting all real-time performer positions, lighting conditions, object positions, audience, TV cameras, and net information. area is set.
  • Permissible travel area (blue) The most inconspicuous non-attention area based on real-time performer position, lighting condition, object position, audience, TV camera, net information
  • No-running area (red) most conspicuous area of interest based on real-time performer position, lighting conditions, object position, audience, TV camera, and internet information
  • Caution driving area (yellow) Intermediate area between the allowable driving area (blue) and the prohibited driving area (red)
  • the real-time data reflecting potential map 230 generated by the real-time data reflecting potential map generating unit 208 is provided to the running route generating unit 240, and the running route generating unit 240 determines the running of the imaging robot 50 based on the real-time data reflecting potential map 230. determine the route.
  • the travel route generation unit 240 inputs the real-time data reflection potential map 230, and the image capturing robot 50 does not collide with performers or objects on the stage, does not stand out due to lighting, and furthermore, travels to the audience and TV.
  • Driving route information is generated in which a route for selecting and driving an area other than the attention area of the viewer on the camera and the Internet is set.
  • the travel route generation unit 240 can set the travel route as described above, for example, by generating a travel route that travels only in the permissible travel area in the real-time data reflecting potential map 230 .
  • the generated travel route is provided to the travel control unit that performs travel control of the image capturing robot 50, and the travel control unit causes the image capturing robot 50 to travel according to the generated travel route information.
  • the image capturing robot 50 is caused to travel according to the travel route information generated using the real-time data reflection potential map 230 generated in the third embodiment, thereby performing a live performance of the image capturing robot 50.
  • Step S301 First, in step S301, the information processing apparatus 200 acquires real-time stage information (performer position, lighting state, on-stage object position, image capturing robot position).
  • This processing is processing executed by the real-time stage information acquisition unit 201 of the information processing apparatus 200 shown in FIG.
  • the real-time stage information acquisition unit 201 is configured by, for example, a camera, an illuminance meter, etc., and acquires the performer's position, the performer's position, the performer's position, the information of the stage where the performer's performance is performed, as real-time stage information. Information on lighting conditions, object positions on the stage, and image capturing robot positions is acquired.
  • Step S302 the information processing apparatus 200 modifies the previously generated potential map based on the real-time stage information acquired in step S301 to generate a potential map reflecting real-time stage information.
  • step S302 is processing executed by the prior potential map correction unit 206 of the information processing apparatus 200 shown in FIG.
  • step S301 the real-time stage information analysis unit 204 of the information processing apparatus 200 shown in FIG. position, illumination state, object position on the stage, image capturing robot position) and schedule data 231 based on the live program stored in advance in the storage unit 205, and the running route according to the pre-generated potential map 150 Analyze the problems of running.
  • step S302 is a subsequent process, and when a request for correction of the pre-generated potential map is input from the real-time stage information analysis unit 204, the pre-potential map correction unit 206 receives a request for correction of the pre-generated potential map from the real-time stage information analysis unit 204.
  • the pre-generated potential map 150 is modified to generate a real-time stage information reflecting potential map so as to reduce the possibility of conspicuous running due to .
  • the real-time stage information reflecting potential map generated by the pre-potential map correction unit 206 is a map generated by reflecting the performer's position, lighting state, and object position in real time.
  • Allowable travel area (blue) area far from real-time performer position and object position, and determined to be the least conspicuous based on real-time lighting conditions;
  • No-running area (red) area close to real-time performer position and object position, and determined to be conspicuous based on real-time lighting conditions;
  • Caution driving area (yellow) Intermediate area between the allowable driving area (blue) and the prohibited driving area (red),
  • step S303 the information processing apparatus 200 acquires real-time spectator seat information and real-time net information.
  • the real-time spectator seat information acquisition unit 202 acquires each of the following information as real-time data, as described above with reference to FIG. Audience line of sight direction 215, Audience seat side TV camera shooting direction 216, Also, the real-time net information acquisition unit 203 acquires so-called SNS information 217 such as tweets of impressions of many live viewers on the Internet, and inputs the acquired SNS information 217 to the real-time attention area analysis unit 207 .
  • SNS information 217 such as tweets of impressions of many live viewers on the Internet
  • Step S304 the information processing apparatus 200 analyzes the real-time attention area based on the real-time spectator seat information and the real-time net information acquired in step S303.
  • This process is executed by the real-time attention area analysis unit 207 of the information processing apparatus 200 shown in FIG.
  • the real-time attention area analysis unit 207 analyzes the audience line-of-sight direction 215 input from the real-time audience seat information acquisition unit 202, the audience seat side TV camera shooting direction 216, and the SNS information 217 input from the real-time net information acquisition unit 203. , which part of the stage the audience is looking at, which part of the stage the TV camera is shooting, and which part of the stage the live viewers on the Internet are paying attention to.
  • Real-time attention area analysis section 207 estimates the current attention area on the stage based on these analysis results, and outputs the estimated area to real-time data reflection potential map generation section 208 as real-time attention area information 221 shown in FIG. do.
  • Step S305 the information processing apparatus 200 generates a "real-time attention area reflecting potential map" reflecting the real-time attention area analyzed based on the real-time spectator seat information and the real-time net information.
  • This process is executed by the real-time data reflecting potential map generation unit 208 of the information processing apparatus 200 shown in FIG.
  • the real-time data reflection potential map generation unit 208 generates a "real-time attention area reflection potential map" that reflects real-time attention areas analyzed based on real-time spectator seat information and real-time net information.
  • the real-time region-of-interest-reflecting potential map is a map generated by reflecting real-time audience, TV camera, and net information, and is a map in which the following region settings are made.
  • Permissible driving area (blue) Non-attention area with the lowest attention based on real-time spectators, TV cameras, and net information
  • No-driving areas (red) attention areas with the highest degree of attention based on real-time spectators, TV cameras, and internet information
  • Caution driving area (yellow) Intermediate area between the allowable driving area (blue) and the prohibited driving area (red)
  • step S306 the information processing apparatus 200 quantifies the "real-time stage information reflecting potential map", the “real-time attention area reflecting potential map”, and the set areas of each map.
  • step S307 the information processing apparatus 200 adds the "real-time stage information reflecting potential map", the "real-time attention area reflecting potential map”, and the numerical values corresponding to the regions of each map to each region. Calculate the added value of
  • This process is also a process executed by the real-time data reflection potential map generation unit 208 of the information processing apparatus 200 shown in FIG.
  • step S308 the data processing unit of the information processing apparatus 200 resets the area division based on the added value corresponding to the area calculated in step S307.
  • This process is also a process executed by the real-time data reflection potential map generation unit 208 of the information processing apparatus 200 shown in FIG.
  • step S309 the data processing unit of the information processing apparatus 200 outputs the potential map with the area segmentation reset in step S308 as a "real-time data reflected potential map.”
  • This process is also a process executed by the real-time data reflection potential map generation unit 208 of the information processing apparatus 200 shown in FIG.
  • steps S301 and S302 of the flowchart shown in FIG. 28 the following processes are performed. Acquisition processing of real-time stage information (performer position, lighting state, on-stage object position, image capturing robot position) by the real-time stage information acquisition unit 201 of the information processing device 200 shown in FIG. Real-time stage information reflecting potential map generation processing is performed by correcting the potential map.
  • real-time stage information performer position, lighting state, on-stage object position, image capturing robot position
  • step S309 the real-time data reflection potential map 230 is generated. Based on this, the travel route of the image capturing robot 50 is determined and travel control is performed.
  • the flowchart shown in FIG. 29 is a flow that can be executed by replacing the processing of steps S301 to S302 of the flow shown in FIG. Hereinafter, processing of each step of the flow shown in FIG. 29 will be described in order.
  • Step S321 First, in step S321, the information processing apparatus 200 shown in FIG. 24 determines whether it is time to acquire real-time stage information.
  • This process is executed by the real-time stage information acquisition unit 201 of the information processing apparatus 200 shown in FIG. 24 or by the control unit that controls the real-time stage information acquisition unit 201 .
  • the acquisition timing of the real-time stage information is defined in advance such that the real-time stage information is acquired at regular intervals, such as every 10 seconds.
  • step S321 it is determined whether or not it is time to acquire real-time stage information according to this rule. If it is determined that it is time to acquire real-time stage information, the process proceeds to step S322.
  • Step S322 When it is determined in step S321 that it is time to acquire real-time stage information, the information processing apparatus 200 acquires real-time stage information (performer position, lighting state, on-stage object position, image capturing robot position) in step S322. .
  • the real-time stage information acquisition unit 201 is configured by, for example, a camera, an illuminance meter, etc., and acquires the performer's position, the performer's position, the performer's position, the information of the stage where the performer's performance is performed, as real-time stage information. Information on lighting conditions, object positions on the stage, and image capturing robot positions is acquired.
  • Step S323 the information processing apparatus 200 determines whether or not it is time for the image capturing robot to finish running.
  • Information about the travel end time of the image capturing robot is stored in advance in the storage unit 205 of the information processing apparatus 200 . For example, it is recorded in association with the scheduled travel route of the image capturing robot.
  • the information processing apparatus 200 refers to this recorded information and determines whether or not the current time is the end time of the image capturing robot running.
  • step S324 If the current time is the running end time of the image capturing robot, the process ends. On the other hand, if the current time is not the running end time of the image capturing robot, the process proceeds to step S324.
  • Step S324 If it is determined in step S323 that it is not the travel end time of the image capturing robot, the information processing apparatus 200 determines in step S324 whether it is the travel route change permissible time of the image capturing robot.
  • Information as to whether or not it is the travel route change permissible time for the image capturing robot is stored in advance in the storage unit 205 of the information processing apparatus 200 . For example, it is recorded in association with the scheduled travel route of the image capturing robot.
  • the information processing apparatus 200 refers to this recorded information and determines whether or not the current time is the travel route change permissible time for the image capturing robot.
  • step S325 If the current time is not the travel route change permissible time for the image capturing robot, the process returns to step S321 without proceeding to step S325, and repeats the processes of steps S321 to S324. Only when it is determined that the current time is within the allowable travel route change time for the image capturing robot, the process proceeds to step S325.
  • Step S325) If it is determined in step S324 that the current time is within the allowable travel route change time for the image capturing robot, the process of step S325 is executed.
  • step S325 the information processing apparatus 200 combines the real-time stage information acquired in step S321 (the position of the performer, the lighting state, the position of the object on the stage, the position of the image capturing robot) with the previously generated potential map. comparison processing.
  • step S325 is a process executed by the real-time stage information analysis unit 204 of the information processing device 200 shown in FIG.
  • Step S326 the information processing apparatus 200 converts the travel route of the image capturing robot based on the pre-generated potential map into the real-time stage information (performer position, lighting state, on-stage object position, image capturing robot position) acquired in step S321. It is determined whether or not the setting is to pass through the dangerous area estimated from the position).
  • step S326 is also a process executed by the real-time stage information analysis unit 204 of the information processing apparatus 200 shown in FIG.
  • a real-time stage information analysis unit 204 analyzes real-time stage information (performer position, lighting state, object position on stage, image capturing robot position) input from the real-time stage information acquisition unit 201 and a live program stored in the storage unit 205 in advance. and the risk of the travel route according to the pre-generated potential map 150.
  • step S321 If it is determined that there is no danger, that is, if the image capturing robot 50 runs along the running route set using the pre-generated potential map 150, there is a possibility that the robot will come into contact with a performer or an object, and the running will be conspicuous due to lighting. If it is determined that there is no possibility of being caught, the process returns to step S321 without proceeding to step S327, and the processes from step S321 onward are repeated.
  • step S327 when it is determined that there is a danger, that is, when the image capturing robot 50 is caused to travel along the travel route set using the pre-generated potential map 150, there is a possibility that the robot 50 may come into contact with the performer or an object, or the travel may be conspicuous due to lighting. is likely to be performed, the process proceeds to step S327.
  • Step S327) If it is determined in step S326 that the travel route set using the pre-generated potential map 150 has a risk of colliding with the performer, the information processing device executes the processing of step S327.
  • step S327 the information processing apparatus 200 modifies the pre-generated potential map 150 stored in the storage unit 205 to generate a real-time stage information reflecting potential map.
  • step S327 is a process executed by the pre-generated potential map correction unit 206 of the information processing apparatus 200 shown in FIG.
  • a pre-potential map correction unit 206 corrects the pre-generated potential map 150 so as to reduce the possibility of contact with performers and objects and the possibility of conspicuous running due to lighting, and generates a potential map reflecting real-time stage information. do.
  • the real-time stage information reflecting potential map generated by the pre-potential map correction unit 206 is a map generated by reflecting the performer's position, lighting state, and object position in real time.
  • Allowable travel area (blue) area far from real-time performer position and object position, and determined to be the least conspicuous based on real-time lighting conditions;
  • No-running area (red) area close to real-time performer position and object position, and determined to be conspicuous based on real-time lighting conditions;
  • Caution driving area (yellow) Intermediate area between the allowable driving area (blue) and the prohibited driving area (red),
  • step S303 onward of the flow shown in FIG. 28 is executed. That is, the processes of steps S303 to S309 are executed, and finally in step S309, the real-time data reflection potential map 230 is generated. A travel route is determined and travel control is performed.
  • FIG. 30 includes the information processing apparatus 100 described with reference to FIG. 4, the information processing apparatus 100b described with reference to FIG. 20, and the information processing apparatus 200 described with reference to FIG. It is an example of the hardware configuration of.
  • the hardware configuration shown in FIG. 30 will be described.
  • a CPU (Central Processing Unit) 301 functions as a data processing section that executes various processes according to programs stored in a ROM (Read Only Memory) 302 or a storage section 308 . For example, the process according to the sequence described in the above embodiment is executed.
  • a RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301 . These CPU 301 , ROM 302 and RAM 303 are interconnected by a bus 304 .
  • the CPU 301 is connected to an input/output interface 305 via a bus 304.
  • the input/output interface 305 includes an input unit 306 including various sensors, a camera, a switch, a keyboard, a mouse, a microphone, and the like, and an output unit 307 including a display, a speaker, and the like. is connected.
  • a storage unit 308 connected to the input/output interface 305 consists of, for example, a hard disk, and stores programs executed by the CPU 301 and various data.
  • a communication unit 309 functions as a transmission/reception unit for data communication via a network such as the Internet or a local area network, and communicates with an external device.
  • a drive 310 connected to the input/output interface 305 drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card to record or read data.
  • a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card to record or read data.
  • the technique disclosed in this specification can take the following configurations. (1) having a data processing unit that generates a potential map that defines an allowable travel area of an image capturing robot that moves on a stage and captures images; The data processing unit Acquiring at least one of the action schedule data of the performer on the stage and the arrangement schedule data of the object on the stage, and determining the probability of collision with the performer or the object on the stage based on the acquired data. An information processing device that generates a potential map that defines a permissible travel area below a value.
  • the data processing unit Acquiring time-series data of action schedule data of the performer on the stage or arrangement schedule data of the object on the stage, and based on the obtained data, the probability of collision with the performer or the object on the stage is equal to or less than a prescribed threshold.
  • the information processing device which generates time-series data of a potential map that defines a permissible travel area of .
  • the data processing unit Acquisition of scheduled data for brightness control of illumination on the stage, and based on the acquired data, an area other than an area where the brightness of the illumination illuminated on the stage is equal to or higher than a predetermined threshold value is defined as an allowable travel area.
  • the information processing apparatus according to any one of (1) to (3), which generates a defined potential map.
  • the data processing unit Color control schedule data for lighting on the stage is acquired, and based on the acquired data, a potential map is generated in which an area other than an area where the illumination on the stage has a predetermined color value is defined as an allowable travel area.
  • the information processing apparatus according to any one of (1) to (4).
  • the data processing unit a permissible travel area in which the probability of collision with performers or objects on the stage is below a specified threshold;
  • the information processing apparatus according to any one of (1) to (5), which generates a potential map defining a travel-prohibited area, which is an area closer to the performer or object on the stage than the allowable travel area.
  • the data processing unit The information processing apparatus according to (6), which generates a potential map defining a travel caution area at an intermediate position between the allowable travel area and the prohibited travel area.
  • the data processing unit (a) a performer-based potential map that defines a permissible travel region in which the possibility of collision with the performer on the stage is equal to or less than a specified threshold value, based on the action schedule data of the performer on the stage; (b) an object-based potential map that defines a permissible travel region in which the possibility of collision with an object on the stage is equal to or less than a predetermined threshold value, based on the arrangement plan data of the object on the stage; (c) Based on the lighting control schedule data on the stage, a region other than the region where the lighting state value calculated based on the lighting irradiated on the stage is equal to or greater than a predetermined threshold value is defined as the travel-permissible region.
  • the pre-generated potential map is Areas other than areas where the possibility of collision with performers and objects on the stage is below a specified threshold and where the lighting condition value calculated based on lighting is above a predetermined threshold is defined as an allowable travel area.
  • the data processing unit In the process of synthesizing the three types of individual potential maps (a) to (c), For each of the permissible driving region, the cautionary driving region, and the prohibited driving region defined in each of the three types of individual potential maps (a) to (c) above, quantification processing is performed on a region-by-region basis. An added value is calculated by adding the numerical values of each potential map, and based on the calculated added value, the travel-permissible region, the caution-driving region, and the travel-prohibited region are reset to generate the pre-generated potential map.
  • the information processing device according to (8) or (9).
  • the data processing unit At least one of viewpoint position data of a spectator seat side viewing the stage and viewpoint position data of a camera is acquired, and based on the acquired data, the viewpoint position of the spectator or the camera is moved from the center of the stage.
  • the information processing apparatus according to any one of (1) to (10), wherein a potential map is generated that defines a position far from a line connecting positions as an allowable travel area.
  • the data processing unit (a) a performer-based potential map that defines a permissible travel region in which the possibility of collision with the performer on the stage is equal to or less than a specified threshold value, based on the action schedule data of the performer on the stage; (b) an object-based potential map that defines a permissible travel region in which the possibility of collision with an object on the stage is equal to or less than a predetermined threshold value, based on the arrangement plan data of the object on the stage; (c) Based on the lighting control schedule data on the stage, a region other than the region where the lighting state value calculated based on the lighting irradiated on the stage is equal to or greater than a predetermined threshold value is defined as the travel-permissible region.
  • the data processing unit Acquiring real-time data of at least one of behavior data of the performer and arrangement data of objects on the stage as real-time data during performance by the performer on the stage;
  • the information processing apparatus according to any one of (1) to (12), which generates a real-time data reflection potential map that defines a travel permissible region in which the possibility of collision with the above performer or object is equal to or less than a predetermined threshold value.
  • the data processing unit As real-time data during a performance by said performer on said stage, Acquire at least one data of the line-of-sight direction of the audience on the side of the audience seat looking at the stage and the shooting direction of the camera, analyze the attention area based on the acquired data, and set the area other than the attention area as the allowable travel area.
  • the information processing device As real-time data during a performance by said performer on said stage, Acquire at least one data of the line-of-sight direction of the audience on the side of the audience seat looking at the stage and the shooting direction of the camera, analyze the attention area based on the acquired data, and set the area other than the attention area as the allowable travel area.
  • the information processing device 13, which generates a defined real-time data reflection potential map.
  • the data processing unit further Acquire user comments while browsing the performance of the performer on the stage via the Internet, analyze the attention area of the Internet browsing user based on the obtained comments, and define the area other than the attention area as the allowable driving area.
  • the data processing unit The information processing apparatus according to any one of (1) to (15), which generates a route along which the imaging robot travels based on the potential map.
  • the data processing unit The information processing apparatus according to (16), which generates a travel route in which the image capturing robot is set to travel within the permissible travel region defined in the potential map.
  • the data processing unit The information processing apparatus according to any one of (1) to (17), which generates simulation data for displaying a travel route of the imaging robot generated based on the potential map on a display unit.
  • a storage unit that stores travel route information generated based on a potential map that defines an allowable travel area below a threshold; or at least one of a communication unit that acquires the travel route information from an external device, A mobile device that executes travel processing according to either the travel route information acquired from the storage unit or the travel route information acquired via the communication unit.
  • An information processing system having an image capturing robot and a server
  • the image capturing robot is an image capturing robot that moves on a stage to capture an image
  • the server is a data processing unit that generates a potential map that defines an allowable travel area of the imaging robot;
  • the data processing unit Acquiring at least one of action schedule data of the performer on the stage and arrangement schedule data of the object on the stage, and determining a probability of collision with the performer or the object on the stage based on the acquired data.
  • the imaging robot is An information processing system that travels along a travel route determined based on the potential map generated by the server.
  • the series of processes described in the specification can be executed by hardware, software, or a composite configuration of both.
  • a program recording the processing sequence is installed in the memory of a computer built into dedicated hardware and executed, or the program is loaded into a general-purpose computer capable of executing various processing. It can be installed and run.
  • the program can be pre-recorded on a recording medium.
  • the program can be received via a network such as a LAN (Local Area Network) or the Internet and installed in a recording medium such as an internal hard disk.
  • a system is a logical collective configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same housing.
  • a map for determining a safe driving route that does not collide with performers and objects on the stage is generated, and the route is determined based on the map. It is possible to run the image capturing robot according to. Specifically, for example, a potential map is generated that defines an allowable travel area for an image capturing robot that moves on a stage to capture images.
  • the data processing unit acquires the action schedule data of the performer on the stage, the arrangement schedule data of the object, and the lighting control schedule data on the stage, and based on the acquired data, an area that does not collide with the performer or the object and is not conspicuous due to lighting. is defined as the permissible travel area.
  • the robot travel route is determined and the robot travels.

Landscapes

  • Engineering & Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
  • Traffic Control Systems (AREA)
PCT/JP2021/048264 2021-01-21 2021-12-24 情報処理装置、および移動装置、並びに情報処理システム Ceased WO2022158255A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US18/260,619 US20240053169A1 (en) 2021-01-21 2021-12-24 Information processing device, mobile device, and information processing system
JP2022577067A JP7823590B2 (ja) 2021-01-21 2021-12-24 情報処理装置、および移動装置、並びに情報処理システム
CN202180090687.0A CN116783565A (zh) 2021-01-21 2021-12-24 信息处理装置、移动装置和信息处理系统

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021008183 2021-01-21
JP2021-008183 2021-01-21

Publications (1)

Publication Number Publication Date
WO2022158255A1 true WO2022158255A1 (ja) 2022-07-28

Family

ID=82548452

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/048264 Ceased WO2022158255A1 (ja) 2021-01-21 2021-12-24 情報処理装置、および移動装置、並びに情報処理システム

Country Status (4)

Country Link
US (1) US20240053169A1 (https=)
JP (1) JP7823590B2 (https=)
CN (1) CN116783565A (https=)
WO (1) WO2022158255A1 (https=)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011528627A (ja) * 2008-07-22 2011-11-24 ロボティクス テクノロジー リーダーズ ゲーエムベーハー マニュピレータアームを運搬するエアクッションプラットフォーム及び移動可能ロボット
JP2011253302A (ja) * 2010-06-01 2011-12-15 Toyota Motor Corp 車両用危険度算出装置
JP2020150297A (ja) * 2019-03-11 2020-09-17 池上通信機株式会社 リモートカメラシステム、コントロールシステム、映像出力方法、バーチャルカメラワークシステム、及びプログラム
WO2021230073A1 (ja) * 2020-05-13 2021-11-18 ソニーグループ株式会社 情報処理装置、情報処理方法、及び表示装置

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DK2952993T3 (en) * 2014-06-05 2018-07-30 Softbank Robotics Europe PROCEDURE FOR MAKING A CARD OF LIKELIHOOD FOR ONE OF THE ABSENCE OR EXISTENCE OF BARRIERS FOR AN AUTONOMOUS ROBOT
JP2019509541A (ja) 2016-01-05 2019-04-04 カーネギー−メロン ユニバーシティＣａｒｎｅｇｉｅ−Ｍｅｌｌｏｎ Ｕｎｉｖｅｒｓｉｔｙ 自律走行車両のための安全性アーキテクチャ
US10788836B2 (en) * 2016-02-29 2020-09-29 AI Incorporated Obstacle recognition method for autonomous robots

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011528627A (ja) * 2008-07-22 2011-11-24 ロボティクス テクノロジー リーダーズ ゲーエムベーハー マニュピレータアームを運搬するエアクッションプラットフォーム及び移動可能ロボット
JP2011253302A (ja) * 2010-06-01 2011-12-15 Toyota Motor Corp 車両用危険度算出装置
JP2020150297A (ja) * 2019-03-11 2020-09-17 池上通信機株式会社 リモートカメラシステム、コントロールシステム、映像出力方法、バーチャルカメラワークシステム、及びプログラム
WO2021230073A1 (ja) * 2020-05-13 2021-11-18 ソニーグループ株式会社 情報処理装置、情報処理方法、及び表示装置

Also Published As

Publication number Publication date
JPWO2022158255A1 (https=) 2022-07-28
CN116783565A (zh) 2023-09-19
US20240053169A1 (en) 2024-02-15
JP7823590B2 (ja) 2026-03-04

Similar Documents

Publication Publication Date Title
JP5972966B2 (ja) 投影された参照パターンによるオブジェクト追跡
US10262230B1 (en) Object detection and identification
US10671846B1 (en) Object recognition techniques
EP3400705B1 (en) Active speaker location detection
US10339386B2 (en) Unusual event detection in wide-angle video (based on moving object trajectories)
US9201499B1 (en) Object tracking in a 3-dimensional environment
US9137511B1 (en) 3D modeling with depth camera and surface normals
KR101566543B1 (ko) 공간 정보 증강을 이용하는 상호 인터랙션을 위한 방법 및 시스템
JP3619455B2 (ja) 自動カメラ選択システムのための方法および装置
US7796154B2 (en) Automatic multiscale image acquisition from a steerable camera
RU2729601C1 (ru) Способ и система для визуализации объекта на виртуальном виде
US20160280136A1 (en) Active-tracking vehicular-based systems and methods for generating adaptive image
JP6471387B2 (ja) 多因子画像特徴登録及び追尾のための方法、回路、装置、システム、及び、関連するコンピュータで実行可能なコード
US9756277B2 (en) System for filming a video movie
US9495936B1 (en) Image correction based on projection surface color
US9682482B2 (en) Autonomous moving device and control method of autonomous moving device
CN107439002B (zh) 深度成像
US9268520B1 (en) Altering content projection
JP2015520470A (ja) データベース生成及び更新に深度ベースのトラッキングを用いた顔認識自己学習
US20170201723A1 (en) Method of providing object image based on object tracking
US8588583B2 (en) Systems and methods for interactive video frame selection
JP5667846B2 (ja) 対象物画像判定装置
US20200322539A1 (en) Imaging system, imaging method, and not-transitory recording medium
JP2009246917A (ja) 映像表示装置及び映像処理装置
JP2012155595A (ja) 対象物画像判定装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21921356

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022577067

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 18260619

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 202180090687.0

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21921356

Country of ref document: EP

Kind code of ref document: A1