Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
• • G—PHYSICS
  • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
  • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
  • G03B15/00—Special procedures for taking photographs; Apparatus therefor
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/695—Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/71—Circuitry for evaluating the brightness variation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/72—Combination of two or more compensation controls
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/75—Circuitry for compensating brightness variation in the scene by influencing optical camera components
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/76—Circuitry for compensating brightness variation in the scene by influencing the image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/222—Studio circuitry; Studio devices; Studio equipment
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B64—AIRCRAFT; AVIATION; COSMONAUTICS
  • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
  • B64U20/00—Constructional aspects of UAVs
  • B64U20/80—Arrangement of on-board electronics, e.g. avionics systems or wiring
  • B64U20/87—Mounting of imaging devices, e.g. mounting of gimbals
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B64—AIRCRAFT; AVIATION; COSMONAUTICS
  • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
  • B64U2101/00—UAVs specially adapted for particular uses or applications
  • B64U2101/30—UAVs specially adapted for particular uses or applications for imaging, photography or videography
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B64—AIRCRAFT; AVIATION; COSMONAUTICS
  • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
  • B64U2201/00—UAVs characterised by their flight controls
  • B64U2201/10—UAVs characterised by their flight controls autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]

Definitions

• the technology of the present disclosure relates to a control device, a control method, and a program.
• JP 2021-96865A discloses an information processing device that instructs flight control of an aircraft.
• the information processing device includes a processing section.
• the processing unit acquires illuminance around the flying object detected by an illuminance sensor included in the flying object, and obtains relational information indicating a relationship between the illuminance and an upper limit value of the flight speed of the flying object.
• the processing unit derives the upper limit value of the flight speed of the aircraft corresponding to the obtained illuminance based on the related information, and controls the flight speed so that the flight speed is less than or equal to the upper limit value of the flight speed of the aircraft. instruct.
• JP 2020-50261A discloses an information processing device that instructs flight control of an aircraft.
• the information processing device includes a processing section.
• the processing unit acquires information on a shutter speed for imaging by the flying object, and acquires information on a distance in real space per pixel in a captured image taken by the flying object.
• the processing unit determines the upper limit of the flight speed of the aircraft based on the shutter speed and the distance in real space per pixel, and adjusts the flight speed so that the flight speed is less than or equal to the upper limit of the flight speed of the aircraft. Direct speed control.
• the map creation unit creates a map of a moving range, which is a range in which a moving object including an imaging device takes pictures while moving.
• the shape extraction unit extracts shapes existing within the map.
• the composition setting section sets the composition of an image to be photographed by the imaging device.
• the route determining unit determines a moving route within a moving range of the moving object based on the shape and composition.
• Japanese Translation of PCT Publication No. 2018-535487 discloses a method of controlling a movable object.
• the method includes the steps of: estimating a reference point of the movable object and one or more movement characteristics of the movable object at the reference point based on a target orientation; and generating a movement path of the movable object from the position of the movable object to the reference point based on the plurality of movement characteristics.
• PCT Publication No. 2019-507924 discloses a system that changes the autonomous flight of an unmanned aircraft.
• the system includes a first user interface and a second user interface.
• the first user interface is a first user interface configured to receive a first user input, the first user input providing one or more instructions for autonomous flight of the UAV. do.
• the second user interface is a second user interface configured to receive a second user input, the second user input generating one or more instructions for modifying autonomous flight of the UAV. provide.
• the shape generation method includes the step of acquiring information regarding a plurality of imaging positions of an aircraft having a plurality of imaging devices.
• the shape generation method includes the step of selecting an imaging device to be used for imaging for each of the plurality of imaging positions from among the plurality of imaging devices.
• the shape generation method includes the step of capturing images at each imaging position using a selected imaging device.
• the shape generation method includes the step of restoring the shape of the subject based on the captured image for each imaging device.
• the shape generation method includes a step of synthesizing shapes restored for each imaging device.
• the step of selecting an imaging device includes the step of selecting at least one imaging device from the plurality of imaging devices based on a proportion of a portion of the imaging region having a predetermined amount of light or less at the imaging position.
• JP 2019-87073 A describes a moving device for moving from a first position to a second position, a manipulator connected to the moving device, a controller that controls the moving device and the manipulator, and a controller for controlling the moving device.
• a mobile manipulator is disclosed that includes an environmental acquisition sensor that acquires environmental data in association with a position. The controller moves based on environmental data acquired by the environment acquisition sensor and associated with the destination position or any point on the movement route between the first position and the second position.
• the device is configured to control at least one of the device and the manipulator.
• one embodiment of the technology of the present disclosure provides an imaging device that uses a shutter speed that corresponds to the brightness of an imaging target area while moving the first moving body at a moving speed that corresponds to the shutter speed at a target position.
• a control device that uses a shutter speed that corresponds to the brightness of an imaging target area while moving the first moving body at a moving speed that corresponds to the shutter speed at a target position.
• a control device that uses a shutter speed that corresponds to the brightness of an imaging target area while moving the first moving body at a moving speed that corresponds to the shutter speed at a target position.
• a first aspect according to the technology of the present disclosure includes a first processor, and the first processor is configured to move the image capturing apparatus from the target position to the target position at a timing before the first moving body equipped with the image capturing apparatus reaches the target position. Obtaining the brightness of the imaging target area to be imaged, obtaining a shutter speed corresponding to the brightness and a moving speed corresponding to the shutter speed, and moving the first moving body at the moving speed at the target position, This is a control device that causes an imaging device to image a region to be imaged at a shutter speed.
• a second aspect according to the technology of the present disclosure is that in the control device according to the first aspect, the first processor adjusts brightness based on relationship information representing a relationship between brightness, shutter speed, and movement speed. This is a control device that obtains a corresponding shutter speed and a moving speed that corresponds to the shutter speed.
• a third aspect according to the technology of the present disclosure is a control device according to the second aspect, which further includes a memory storing related information.
• a fourth aspect of the technology of the present disclosure is a control device according to any one of the first to third aspects, wherein the brightness is brightness detected from a target position using a sensor. It is a certain control device.
• a fifth aspect according to the technology of the present disclosure is the control device according to the fourth aspect, in which the sensor is a control device mounted on the second moving body.
• the target position is set on the moving route, and the brightness is set at a speed higher than the moving speed of the second moving body.
• the brightness detected while moving along the moving route is the control device.
• the brightness is determined using an overhead camera that overlooks a subject including an imaging target area. This is the control device that controls the brightness detected.
• An eighth aspect according to the technology of the present disclosure is a control device according to the seventh aspect, in which the subject is a control device including a plurality of imaging target regions.
• a ninth aspect of the technology of the present disclosure is the control device according to any one of the first to eighth aspects, wherein the target position is set on a moving route, and the brightness is The control device detects the brightness at a timing before the first moving object starts moving along the moving route.
• a tenth aspect of the technology of the present disclosure is the control device according to any one of the first to eighth aspects, wherein the target position is set on a moving route, and the brightness is In the control device, the brightness is detected at a timing after the first moving body starts moving along the movement route and before the first moving body reaches the target position.
• An eleventh aspect according to the technology of the present disclosure is that in the control device according to any one of the first to tenth aspects, a plurality of target positions are set as destinations to which the first moving body moves.
• the first processor is a control device that causes the imaging device to image the imaging target area at a constant aperture value at a plurality of target positions.
• a twelfth aspect according to the technology of the present disclosure is a control device according to any one of the first to tenth aspects, in which a plurality of target positions are set as destinations to which the first moving body moves.
• the first processor is a control device that causes the imaging device to image the imaging target region at an aperture value corresponding to brightness and/or shutter speed for each target position.
• a thirteenth aspect according to the technology of the present disclosure is a control device according to any one of the first to twelfth aspects, in which a plurality of target positions are set as destinations to which the first moving body moves.
• the first processor acquires, for each target position, a composite image obtained by causing the imaging device to image the imaging target area, and the composite image is configured such that adjacent composite images partially overlap each other.
• a control device that is an image to wrap.
• a fourteenth aspect according to the technology of the present disclosure is a control device according to the thirteenth aspect, further comprising a second processor that generates a composite image by combining adjacent images for composition, and the second processor This is a control device that performs specific processing on a composite image based on the
• a fifteenth aspect according to the technology of the present disclosure is a control device according to the fourteenth aspect, in which the specific process includes a correction process for correcting the brightness of the composite image.
• a sixteenth aspect according to the technology of the present disclosure is a control device according to the fifteenth aspect, in which the correction process includes a process of correcting a difference in brightness between images for synthesis.
• a seventeenth aspect according to the technology of the present disclosure is the control device according to the fifteenth aspect or the sixteenth aspect, wherein the correction process includes a process of correcting the brightness distribution of the composite image based on the brightness distribution. It is a control device.
• An 18th aspect according to the technology of the present disclosure is that in the control device according to any one of the 14th to 17th aspects, the specifying process is such that the difference in brightness between the imaging target areas is set to a predetermined value.
• This is a control device that includes notification processing to notify when the limit is exceeded.
• a nineteenth aspect of the technology of the present disclosure is to obtain the brightness of the imaging target area imaged by the imaging device from the target position at a timing before the first moving body equipped with the imaging device reaches the target position. acquiring a shutter speed corresponding to the brightness and a movement speed corresponding to the shutter speed; and capturing an image with an imaging device at the shutter speed while moving the first moving body at the movement speed at the target position.
• This is a control method including imaging a target area.
• a 20th aspect of the technology of the present disclosure acquires the brightness of the imaging target area imaged by the imaging device from the target position at a timing before the first moving body equipped with the imaging device reaches the target position. acquiring a shutter speed corresponding to the brightness and a movement speed corresponding to the shutter speed; and capturing an image with an imaging device at the shutter speed while moving the first moving body at the movement speed at the target position.
• This is a program for causing a computer to execute processing including capturing an image of a target area.
• FIG. 2 is a front view showing an example of a plurality of imaging target areas and an imaging system according to the first embodiment.
• FIG. 1 is a block diagram illustrating an example of the hardware configuration of the flight imaging device according to the first embodiment.
• FIG. 1 is a block diagram showing a hardware configuration of an imaging device according to a first embodiment.
• FIG. 2 is a block diagram illustrating an example of a functional configuration for realizing flight imaging processing according to the first embodiment.
• FIG. 2 is an explanatory diagram illustrating an example of the operation of a data requesting unit and a brightness obtaining unit according to the first embodiment.
• FIG. 3 is an explanatory diagram illustrating an example of the operation of the brightness acquisition section and the imaging condition acquisition section according to the first embodiment.
• FIG. 1 is a block diagram illustrating an example of the hardware configuration of the flight imaging device according to the first embodiment.
• FIG. 1 is a block diagram showing a hardware configuration of an imaging device according to a first embodiment.
• FIG. 2 is a block diagram
• FIG. 2 is an explanatory diagram illustrating an example of the operation of an imaging condition acquisition unit and a flight speed control unit according to the first embodiment.
• FIG. 3 is an explanatory diagram illustrating an example of operations of an imaging condition acquisition unit, an arrival determination unit, and an imaging control unit according to the first embodiment. It is a flowchart which shows an example of the flow of flight imaging processing concerning a 1st embodiment.
• FIG. 7 is an explanatory diagram illustrating an example of the operation of a data requesting unit and a brightness obtaining unit according to the second embodiment.
• FIG. 7 is an explanatory diagram illustrating an example of the operation of a data requesting unit and a brightness acquiring unit according to a third embodiment.
• FIG. 7 is an explanatory diagram illustrating an example of the operation of a data requesting unit and a brightness acquiring unit according to a fourth embodiment.
• FIG. 7 is a block diagram showing an example of a functional configuration for realizing composite image generation processing according to a fifth embodiment.
• FIG. 12 is an explanatory diagram illustrating an example of operations of an image synthesis section, a brightness information acquisition section, and a correction processing section according to a fifth embodiment.
• 13 is a flowchart illustrating an example of the flow of composite image generation processing according to the fifth embodiment.
• I/F is an abbreviation for "Interface”.
• RAM is an abbreviation for "Random Access Memory.”
• CPU is an abbreviation for "Central Processing Unit.”
• GPU is an abbreviation for “Graphics Processing Unit.”
• HDD is an abbreviation for “Hard Disk Drive.”
• SSD is an abbreviation for “Solid State Drive.”
• DRAM is an abbreviation for "Dynamic Random Access Memory.”
• SRAM is an abbreviation for "Static Random Access Memory.”
• GNSS is an abbreviation for “Global Navigation Satellite System.”
• GPS is an abbreviation for “Global Positioning System.”
• LiDAR is an abbreviation for "light detection and ranging.”
• NVM is an abbreviation for "Non-Volatile Memory.”
• ASIC is an abbreviation for “Application Specific Integrated Circuit.”
• FPGA is an abbreviation for “Field-Programmable Gate Array.”
• CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor.”
• CCD is an abbreviation for “Charge Coupled Device”.
• RGB is an abbreviation for “Red Green Blue.”
• CIE is an abbreviation for "Commission Internationale de l'Eclairage”.
• TPU is an abbreviation for "Tensor Processing Unit”.
• USB is an abbreviation for "Universal Serial Bus.”
• SoC is an abbreviation for "System-on-a-Chip.”
• IC is an abbreviation for "Integrated Circuit.”
• the term “vertical direction” refers to not only a complete vertical direction but also a vertical direction that is generally accepted in the technical field to which the technology of the present disclosure belongs, and which is within the spirit of the technology of the present disclosure. Refers to the vertical direction, including a certain amount of error.
• the term “horizontal direction” refers to not only a completely horizontal direction but also a horizontal direction that is generally accepted in the technical field to which the technology of the present disclosure belongs, and which is within the spirit of the technology of the present disclosure. Refers to the horizontal direction, including a degree of error that does not deviate.
• a quadrilateral refers to a perfect quadrilateral as well as a quadrilateral that is generally accepted in the technical field to which the technology of the present disclosure belongs, to the extent that it does not go against the spirit of the technology of the present disclosure. Refers to a rectangle that includes the error of .
• vertical means not only completely vertical but also perpendicular to a degree that is generally accepted in the technical field to which the technology of the present disclosure belongs and does not go against the spirit of the technology of the present disclosure. It refers to vertical in the sense of including the error of.
• constant means not only a complete constant but also a constant that is generally accepted in the technical field to which the technology of the present disclosure belongs, and that does not go against the spirit of the technology of the present disclosure. It refers to a constant in the sense of including the error of.
• the imaging system S includes a flight imaging device 10 and an overhead camera 160.
• the flight imaging device 10 has a flight function and an imaging function, and images the wall surface 2A of the object 2 while flying.
• the wall surface 2A is, for example, a flat surface.
• a plane refers to a two-dimensional surface (that is, a surface along a two-dimensional direction). Furthermore, in the description of this specification, the concept of "plane" does not include the meaning of mirror surface.
• the wall surface 2A is a plane defined in the horizontal direction and the vertical direction (that is, a surface extending in the horizontal direction and the vertical direction).
• the wall surface 2A includes unevenness.
• the unevenness referred to here includes, for example, unevenness due to the material forming the wall surface 2A, as well as unevenness due to defects and/or defects.
• the object 2 having the wall surface 2A is a pier provided on a bridge.
• the piers are made of reinforced concrete, for example.
• a bridge pier is mentioned here as an example of the target object 2, the target object 2 may be an object other than a bridge pier (for example, a tunnel or a dam).
• the flight imaging device 10 includes a flying object 20 and an imaging device 60.
• the flying object 20 is, for example, an unmanned aircraft such as a drone.
• the flight function of the flight imaging device 10 is realized by the flying object 20.
• the flying object 20 has a plurality of propellers 42, and flies as the plurality of propellers 42 rotate. Flying the flying object 20 is synonymous with flying the flying imaging device 10.
• the flying object 20 is an example of a "mobile object" according to the technology of the present disclosure.
• the imaging device 60 is, for example, a digital camera or a video camera.
• the imaging function of the flight imaging device 10 is realized by the imaging device 60.
• the imaging device 60 is mounted on the flying object 20. As an example, the imaging device 60 is provided at the bottom of the flying object 20.
• the imaging device 60 is an example of an "imaging device" according to the technology of the present disclosure.
• a flight route 4 is set for the object 2.
• the flight route 4 is set on a virtual plane facing the wall surface 2A.
• the flight route 4 is set in a zigzag shape in which a horizontal route extending in the horizontal direction and a vertical route extending in the vertical direction are connected.
• the flight imaging device 10 stores flight route information 116 (see FIG. 8) indicating a flight route 4 as described later, and responds to a flight instruction signal from a transmitter (not shown) or a base station (not shown), etc. It is possible to autonomously fly the flight route 4 without depending on the vehicle.
• the flight imaging device 10 By flying along the flight route 4, the flight imaging device 10 moves in a zigzag pattern by alternately repeating movement in the horizontal direction and movement in the vertical direction. A plurality of waypoints 5 are set on the flight route 4. At each waypoint 5, the flight imaging device 10 images the wall surface 2A.
• the flight route 4 is an example of a "travel route” according to the technology of the present disclosure
• the waypoint 5 is an example of a "target position" according to the technology of the present disclosure.
• the flight imaging device 10 autonomously flies the flight route 4
• the flight imaging device 10 flies the flight route 4 based on a flight instruction signal from a transmitter, a base station, etc. You may fly 4.
• the flight imaging device 10 sequentially images a plurality of imaging target regions 3 of the wall surface 2A by imaging the wall surface 2A at each waypoint 5. Each imaging target area 3 corresponds to each waypoint 5.
• the imaging target area 3 is an area determined by the angle of view of the flight imaging device 10. In the example shown in FIG. 1, a rectangular area is shown as an example of the imaging target area 3. In the example shown in FIG. By moving the flight imaging device 10 in a zigzag pattern by alternately repeating movement in the horizontal direction and movement in the vertical direction, a plurality of imaging target areas 3 connected in a zigzag pattern are sequentially imaged.
• a plurality of images for synthesis 132 are obtained by sequentially capturing images of the plurality of imaging target regions 3 by the imaging device 60.
• a composite image 130 is generated by combining a plurality of images for composition 132.
• the plurality of images for synthesis 132 are synthesized such that adjacent images for synthesis 132 partially overlap each other.
• An example of the composite image 130 is a two-dimensional panoramic image.
• the two-dimensional panoramic image is just an example, and a three-dimensional image (for example, a three-dimensional panoramic image) is generated as the composite image 130 in the same manner as a two-dimensional panoramic image is generated as the composite image 130. You may also do so.
• the composite image 130 may be generated each time each composite image 132 from the second frame onward is obtained, or may be generated after a plurality of composite images 132 are obtained for the wall surface 2A. Further, the process of generating the composite image 130 may be executed by the flight imaging device 10, or may be executed by an external device (not shown) communicably connected to the flight imaging device 10. The composite image 130 is used, for example, to inspect or survey the wall surface 2A of the object 2.
• FIG. 1 shows a mode in which each imaging target area 3 is imaged by the imaging device 60 in a state where the optical axis OA (see FIG. 2) of the imaging device 60 is perpendicular to the wall surface 2A.
• the following description will be given on the premise that each imaging target area 3 is imaged by the imaging device 60 in a state where the optical axis OA of the imaging device 60 is perpendicular to the wall surface 2A.
• the plurality of imaging target regions 3 are imaged so that adjacent imaging target regions 3 partially overlap each other.
• the plurality of imaging target areas 3 are imaged so that the adjacent imaging target areas 3 partially overlap each other, based on the feature points included in the overlapping parts of the adjacent imaging target areas 3. This is to synthesize the synthesis image 132 corresponding to No. 3.
• partially overlapping of adjacent imaging target regions 3 and partially overlapping of adjacent compositing images 132 may be respectively referred to as "overlap.”
• the overhead camera 160 is a digital camera or video camera with a wider angle than the imaging device 60.
• the bird's-eye camera 160 is placed facing the wall surface 2A.
• the bird's-eye view camera 160 is installed at a position farther from the wall surface 2A than the flight imaging device 10.
• the bird's-eye camera 160 has an angle of view that allows it to image the entire wall surface 2A.
• the bird's-eye camera 160 includes an imaging lens, an image sensor, a processor, a storage, a RAM, and the like, although all are not shown in the drawings.
• the bird's-eye view camera 160 is an example of a "bird's-eye camera" according to the technology of the present disclosure.
• the flying object 20 includes a flight device 22, an input/output I/F 24, a computer 26, a positioning unit 28, an acceleration sensor 30, and a communication device 32.
• the computer 26 is an example of a “control device” and a “computer” according to the technology of the present disclosure.
• the computer 26 includes a processor 34, a storage 36, and a RAM 38.
• the processor 34, storage 36, and RAM 38 are interconnected via a bus 40, and the bus 40 is connected to the input/output I/F 24. Further, the positioning unit 28, the acceleration sensor 30, and the communication device 32 are also connected to the input/output I/F 24.
• the processor 34 includes, for example, a CPU, and controls the entire flight imaging device 10. Although an example in which the processor 34 includes a CPU is given here, this is just an example.
• processor 34 may include a CPU and a GPU. In this case, for example, the GPU operates under the control of the CPU and is responsible for executing image processing.
• the processor 34 is an example of a "first processor" according to the technology of the present disclosure.
• the storage 36 is a nonvolatile storage device that stores various programs, various parameters, and the like. Examples of the storage 36 include an HDD and an SSD. Note that the HDD and SSD are just examples, and flash memory, magnetoresistive memory, and/or ferroelectric memory may be used instead of or in conjunction with the HDD and/or SSD. .
• the RAM 38 is a memory in which information is temporarily stored, and is used by the processor 34 as a work memory. Examples of the RAM 38 include DRAM and/or SRAM.
• the communication device 32 is connected to the bird's-eye view camera 160 so as to be communicable by wire or wirelessly.
• the communication device 32 is in charge of exchanging information with the overhead camera 160.
• the communication device 32 transmits information in response to a request from the processor 34 to the overhead camera 160.
• the communication device 32 receives data transmitted from the overhead camera 160 and outputs the received data to the processor 34 via the bus 40.
• the flight device 22 has multiple propellers 42, multiple motors 44, and a motor driver 46.
• the motor driver 46 is connected to the processor 34 via the input/output I/F 24 and the bus 40. Motor driver 46 individually controls multiple motors 44 according to instructions from processor 34 .
• the number of multiple motors 44 is the same as the number of multiple propellers 42.
• a propeller 42 is fixed to the rotating shaft of each motor 44. Each motor 44 rotates a propeller 42 .
• the aircraft 20 flies as the plurality of propellers 42 rotate. Note that the number of the plurality of propellers 42 (in other words, the number of the plurality of motors 44) provided in the aircraft 20 is four, as an example, but this is just an example, and the number of the plurality of propellers 42 is , for example, there may be three or five or more.
• the positioning unit 28 is a device that detects the position of the flying object 20.
• the position of the aircraft 20 is detected using, for example, GNSS (eg, GPS).
• the positioning unit 28 has a GNSS receiver (not shown).
• a GNSS receiver receives, for example, radio waves transmitted from multiple satellites.
• the positioning unit 28 detects the position of the flying object 20 based on the radio waves received by the GNSS receiver, and outputs positioning data 48 (for example, data indicating latitude, longitude, and altitude) according to the detected position. .
• the acceleration sensor 30 detects the acceleration of the aircraft 20 in the pitch, yaw, and roll axis directions.
• the acceleration sensor 30 outputs acceleration data 50 corresponding to the acceleration of the flying object 20 in each axial direction.
• Processor 34 obtains the position of aircraft 20 based on positioning data 48 and/or acceleration data 50.
• the acceleration sensor 30 may be omitted.
• positioning unit 28 may be omitted.
• the processor 34 obtains the position of the flying object 20 based on the positioning data 48, the position in the absolute coordinate system is derived based on the positioning data 48.
• the processor 34 acquires the imaging position based on the acceleration data 50, the amount of change in position with respect to the reference position defined in the relative coordinate system is derived based on the acceleration data 50.
• the flying object 20 also includes other devices for detecting the position of the flying object 20 instead of or in addition to the positioning unit 28 and/or the acceleration sensor 30. You can leave it there.
• Other devices include, for example, a LiDAR scanner, a stereo camera, a magnetic compass, a barometric altimeter, or an ultrasonic sensor.
• the imaging device 60 includes an imaging lens 62, an aperture 64, an aperture actuator 66, a mechanical shutter 68, a shutter actuator 70, a controller 72, an image sensor 74, and an image sensor driver 76.
• the controller 72 and the image sensor driver 76 are connected to the processor 34 via the input/output I/F 24 and the bus 40.
• the imaging lens 62 includes, for example, an objective lens (not shown), a focus lens (not shown), and the like.
• the imaging device 60 also includes a zoom lens (not shown). The imaging lens 62 is arranged on the object side with respect to the aperture 64, and the zoom lens is arranged between the aperture 64 and the mechanical shutter 68.
• the aperture actuator 66 has a power transmission mechanism (not shown) and an aperture motor (not shown).
• the diaphragm 64 has an aperture 64A, and the size of the aperture 64A is variable.
• the opening 64A is formed by a plurality of blades (not shown).
• the plurality of blades are connected to a power transmission mechanism.
• An aperture motor is connected to the power transmission mechanism, and the power transmission mechanism transmits the power of the aperture motor to the plurality of blades.
• the plurality of blades change the size of the opening 64A by operating in response to power transmitted from the power transmission mechanism.
• the aperture 64 adjusts exposure by changing the size of the aperture 64A.
• the aperture value is defined by the size of the aperture 64A.
• Aperture actuator 66 is connected to controller 72 .
• the controller 72 is, for example, a device that includes a computer including a CPU, NVM, RAM, and the like. Note that although a computer is illustrated here, this is just one example, and a device including an ASIC, an FPGA, and/or a PLD may also be applied. Further, as the controller 72, for example, a device realized by a combination of a hardware configuration and a software configuration may be used. Controller 72 controls aperture actuator 66 according to instructions from processor 34 .
• the image sensor 74 includes a photoelectric conversion element 78 and a signal processing circuit 80.
• the image sensor 74 is, for example, a CMOS image sensor.
• a CMOS image sensor is exemplified as the image sensor 74, but the technology of the present disclosure is not limited to this.
• the image sensor 74 may be another type of image sensor such as a CCD image sensor.
• the technology of the present disclosure is realized.
• the photoelectric conversion element 78 is connected to the image sensor driver 76.
• the image sensor driver 76 controls the photoelectric conversion element 78 according to instructions from the processor 34.
• the photoelectric conversion element 78 has a light receiving surface 78A on which a plurality of pixels (not shown) are provided.
• the photoelectric conversion element 78 outputs the electrical signals output from the plurality of pixels to the signal processing circuit 80 as image data 82.
• the signal processing circuit 80 digitizes analog imaging data 82 input from the photoelectric conversion element 78.
• the signal processing circuit 80 is connected to the input/output I/F 24.
• the digitized image data 82 is image data representing the composite image 132, and is stored in the storage 36 after being subjected to various processing by the processor 34.
• the mechanical shutter 68 is, for example, a focal plane shutter, and is arranged between the aperture 64 and the light receiving surface 78A.
• the mechanical shutter 68 includes a front curtain 68A and a rear curtain 68B.
• each of the front curtain 68A and the rear curtain 68B includes a plurality of blades (not shown).
• the front curtain 68A is arranged closer to the subject than the rear curtain 68B.
• the shutter actuator 70 includes a link mechanism (not shown), a leading curtain solenoid (not shown), and a trailing curtain solenoid (not shown).
• the front curtain solenoid is a drive source for the front curtain 68A, and is mechanically connected to the front curtain 68A via a link mechanism.
• the trailing curtain solenoid is a drive source for the trailing curtain 68B, and is mechanically connected to the trailing curtain 68B via a link mechanism.
• Shutter actuator 70 is connected to controller 72 . Controller 72 controls shutter actuator 70 according to instructions from processor 34 .
• the front curtain solenoid generates power under the control of the controller 72, and selectively winds up and lowers the front curtain 68A by applying the generated power to the front curtain 68A.
• the trailing curtain solenoid generates power under the control of the controller 72, and selectively winds up and lowers the trailing curtain 68B by applying the generated power to the trailing curtain 68B.
• the opening/closing of the front curtain 68A and the opening/closing of the rear curtain 68B are controlled by the processor 34, thereby adjusting the amount of exposure to the image sensor 74.
• the shutter speed of the mechanical shutter 68 is defined by the time period during which the front curtain 68A and the rear curtain 68B are open.
• the mechanical shutter 68 may be a lens shutter.
• the explanation has been given as an example in which the shutter speed of the mechanical shutter 68 is derived and set, this is merely an example.
• the shutter speed of an electronic shutter for example, an electronic front curtain shutter or a fully electronic shutter
• a common computer 26 is used for the flying object 20 and the imaging device 60, but the computer 26 is connected to the first computer provided in the flying object 20 and the imaging device 60.
• the computer may be configured by a second computer installed in the computer.
• the computer 26 is mounted on the aircraft 20, it may also be mounted on the imaging device 60. Further, the computer 26 may be mounted on a transmitter or a base station.
• An example of the hardware configuration of the flight imaging device 10 has been described above.
• the flying object 20 on which the imaging device 60 is mounted reaches the target waypoint 5 (hereinafter referred to as "target waypoint 5")
• the imaging target area 3 corresponding to the target waypoint 5 is captured by the imaging device.
• the following technology is assumed as the technology for imaging the camera 60.
• an area corresponding to the flight position of the aircraft 20 (hereinafter referred to as , referred to as a “flight position corresponding area”) is imaged, and the brightness of the flight position corresponding area is detected based on the image obtained by being imaged. Subsequently, a shutter speed corresponding to the detected brightness and a flight speed corresponding to the shutter speed are calculated. Then, at the target waypoint 5, the imaging target area 3 is imaged by the imaging device 60 at the calculated shutter speed while the flying object 20 is flying at the calculated flight speed.
• the flight position corresponding area is imaged by the imaging device 60 at a timing before the aircraft 20 reaches the target waypoint 5, and the flight position corresponding area is imaged based on the image obtained by the imaging. Brightness is detected.
• the flight position corresponding area imaged by the imaging device 60 is an area located in front of the imaging target area 3 corresponding to the target waypoint 5. Therefore, the flight position corresponding area imaged by the imaging device 60 may have a brightness different from that of the imaging target area 3 corresponding to the target waypoint 5.
• the imaging target area 3 cannot be imaged at a shutter speed corresponding to the brightness of the imaging target area 3.
• the following method can be considered. That is, when the flying object 20 reaches the target waypoint 5, the flying object 20 is temporarily stopped at the target waypoint 5, and the imaging device 60 images the imaging target area 3 in this state.
• a possible countermeasure method is to detect the brightness of the imaging target area 3 based on the detected image and calculate the shutter speed corresponding to the detected brightness.
• the flying object 20 is temporarily stopped at the target waypoint 5
• the imaging device 60 cannot image the imaging target area 3 while the flying object 20 is flying. Therefore, in the first embodiment, while the flying object 20 is flying at the flight speed corresponding to the shutter speed at the target waypoint 5, the imaging device 60 is directed to the imaging target area at a shutter speed corresponding to the brightness of the imaging target area 3. 3 is imaged.
• the processor 34 performs flight imaging processing (see FIGS. 4 to 9) described below.
• a flight imaging program 90 is stored in the storage 36.
• the processor 34 reads the flight imaging program 90 from the storage 36 and executes the read flight imaging program 90 on the RAM 38.
• the processor 34 performs flight imaging processing for imaging the imaging target area 3 while the flight imaging device 10 is flying, according to the flight imaging program 90 executed on the RAM 38 .
• the processor 34 operates as a data requesting section 92, a brightness acquisition section 94, an imaging condition acquisition section 96, a flight speed control section 98, an arrival determination section 100, and an imaging control section 102 according to a flight imaging program 90. This is achieved by
• the bird's-eye view camera 160 may be used for multiple The entire wall surface 2A including the imaging target area 3 is imaged.
• an image 162 (hereinafter referred to as "wall surface image 162") including the entire wall surface 2A as an image is obtained.
• the overhead camera 160 derives the brightness of each imaging target area 3 corresponding to each waypoint 5 based on the wall image 162.
• the wall image 162 is an RGB image formed by the three primary colors of light
• the overhead camera 160 generates the image 164 based on the RGB image.
• the image 164 is an image in which the color space of an RGB image is converted to a color space different from RGB.
• the image 164 will be referred to as the "converted image 164.”
• the converted image 164 includes a brightness component indicating the degree of brightness. Examples of the color space of the converted image 164 include Lab color space, Lch color space, and HSV color space.
• the overhead camera 160 derives the brightness of each imaging target region 3 corresponding to each waypoint 5 based on the brightness component included in the converted image 164.
• the brightness of the imaging target area 3 may be derived based on a standard light source (for example, D50 or D65) defined by CIE. Further, the brightness of the imaging target region 3 may be a representative value (for example, the lowest value, the highest value, or the center value) of the entire brightness of the imaging target region 3, or the average brightness of the entire imaging target region 3. It can also be a value. Further, the brightness of the imaging target area 3 may be the brightness of the entire imaging target area 3 or the brightness of a part of the imaging target area 3. The brightness of a part of the imaging target area 3 may be the brightness of the center of the imaging target area 3 or may be the brightness of a part of the imaging target area 3 that is different from the central part.
• a standard light source for example, D50 or D65
• the brightness of the imaging target region 3 may be a representative value (for example, the lowest value, the highest value, or the center value) of the entire brightness of the imaging target region 3, or the average brightness of the entire imaging target region 3. It can also be a
• the overhead camera 160 generates information regarding the brightness of each imaging target area 3 corresponding to each waypoint 5 as brightness information 166, and stores the generated brightness information 166.
• the brightness information 166 is information representing the relationship between a number indicating the order of each waypoint 5 (hereinafter referred to as a "waypoint number") and the brightness of the imaging target area 3.
• FIG. 5 shows, as an example of the brightness information 166, an example of a specific value of the brightness of the imaging target area 3 corresponding to each waypoint number. Brightness is expressed, for example, by brightness, which indicates the degree of brightness.
• the data requesting unit 92 transmits a request signal 104 for requesting brightness data 168 to the overhead camera 160 via the communication device 32.
• the brightness data 168 is data indicating the brightness of the imaging target area 3 corresponding to the target waypoint 5.
• the request signal 104 includes information indicating the waypoint number corresponding to the target waypoint 5.
• the overhead camera 160 When the overhead camera 160 receives the request signal 104, it extracts the brightness corresponding to the waypoint number indicated by the request signal 104 from the brightness information 166, and sends brightness data 168 indicating the extracted brightness to the communication device 32. Send to.
• the waypoint number indicated by the request signal 104 is "NO.1”
• the brightness data 168 indicating "20” which is the degree of brightness corresponding to "NO.1”
• the brightness acquisition unit 94 acquires the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness data 168 received by the communication device 32.
• the brightness acquired by the brightness acquisition unit 94 is the brightness detected at a timing before the flying object 20 starts moving along the flight route 4.
• a table 110 is stored in the storage 36.
• the table 110 is an example of "related information" according to the technology of the present disclosure
• the storage 36 is an example of "memory” according to the technology of the present disclosure.
• Table 110 has a first table 112 and a second table 114.
• the first table 112 shows the relationship between brightness, shutter speed, and aperture value
• the second table 114 shows the relationship between shutter speed and flight speed.
• FIG. 6 shows examples of specific values of brightness, shutter speed, and aperture value for the first table 112, and examples of specific values of the shutter speed and flight speed for the second table 114.
• An example is shown.
• the unit of shutter speed is "second (s)” and the unit of flight speed is "meter per second (m/s)."
• the shutter speed is set, for example, to a value that allows image blurring that occurs during exposure on the light receiving surface 78A to fall within an allowable range.
• Image blur is defined, for example, by the moving distance of the image on the light-receiving surface 78A (for example, the moving distance corresponding to the number of pixels).
• the allowable range is set, for example, based on the image quality required for the composite image 132 (for example, image quality that allows feature points to be identified).
• the aperture value is set based on brightness and/or shutter speed.
• the flight speed is set to a value that keeps image blur within an acceptable range for each shutter speed.
• the table 110 includes a first table 112 and a second table 114, but it may also be a single table showing the relationship between brightness, shutter speed, aperture value, and flight speed.
• the first table 112 and the second table 114 are stored in the storage 36, and the first relational expression indicating the relationship between brightness, shutter speed, and aperture value is stored in the first table. 112, and a second relational expression indicating the relationship between shutter speed and flight speed may be stored instead of second table 114.
• the first table 112 stores aperture values that vary depending on brightness and shutter speed, the aperture value may be constant.
• the imaging condition acquisition unit 96 acquires imaging conditions based on the brightness acquired by the brightness acquisition unit 94.
• the imaging conditions include shutter speed, aperture value, and flight speed.
• the imaging condition acquisition unit 96 acquires the shutter speed and aperture value corresponding to the brightness acquired by the brightness acquisition unit 94 based on the first table 112. Further, the imaging condition acquisition unit 96 acquires the flight speed corresponding to the shutter speed acquired based on the first table 112 based on the second table 114.
• the flight speed control section 98 controls the plurality of motors 44 via the motor driver 46 based on the flight speed acquired by the imaging condition acquisition section 96, thereby controlling the flight object 20.
• the flight speed is set to the flight speed acquired by the imaging condition acquisition unit 96.
• the rotational speed of the plurality of propellers 42 is adjusted, and the flight speed of the flying object 20 is set to the flight speed acquired by the imaging condition acquisition unit 96.
• flight route information 116 indicating flight route 4 is stored in storage 36.
• the flight route information 116 includes position information indicating the positions of a plurality of waypoints 5 set on the flight route 4.
• the arrival determination unit 100 acquires the position of the flying object 20 based on the positioning data 48 input from the positioning unit 28 and/or the acceleration data 50 input from the acceleration sensor 30. Then, the arrival determination unit 100 determines whether the aircraft 20 has reached the target waypoint 5 based on the acquired position of the aircraft 20 and the position of the target waypoint 5 indicated by the flight route information 116. judge.
• the imaging control unit 102 sets the target waypoint to the imaging device 60 using the shutter speed and aperture value acquired by the imaging condition acquisition unit 96.
• the imaging target area 3 corresponding to No. 5 is imaged.
• the imaging control unit 102 opens and closes the front curtain 68A and the rear curtain 68B at a time corresponding to the shutter speed acquired by the imaging condition acquisition unit 96 by controlling the shutter actuator 70 via the controller 72. let Furthermore, by controlling the aperture actuator 66 via the controller 72, the size of the aperture 64A of the aperture 64 is adjusted to a size corresponding to the aperture value acquired by the imaging condition acquisition unit 96. Furthermore, the imaging control unit 102 causes the image sensor 74 to output imaging data 82 by controlling the image sensor 74 via the image sensor driver 76 .
• the imaging data 82 is image data indicating a composite image 132 obtained by imaging the imaging target region 3, and is stored in the storage 36 after being subjected to various processing by the processor 34 (see FIG. 2). .
• the flight speed of the flying object 20 is maintained at the flight speed acquired by the imaging condition acquisition unit 96 until the imaging target region 3 is imaged by the imaging control unit 102. After the imaging control unit 102 images the imaging target area 3, the flight speed of the flying object 20 remains unchanged until the flight speed is controlled again by the flight speed control unit 98 corresponding to the next target waypoint 5.
• the flight speed may be changed to a higher flight speed than the flight speed acquired by the condition acquisition unit 96.
• FIG. 9 shows an example of the flow of flight imaging processing according to the first embodiment.
• step ST10 the data requesting unit 92 sends a request signal 104 for requesting brightness data 168 indicating the brightness of the imaging target area 3 corresponding to the target waypoint 5. , is transmitted to the bird's-eye camera 160 via the communication device 32 (see FIG. 5). After the process of step ST10 is executed, the flight imaging process moves to step ST12.
• step ST12 the brightness acquisition unit 94 acquires the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness data 168 transmitted from the overhead camera 160 and received by the communication device 32. (See Figure 5). After the process of step ST12 is executed, the flight imaging process moves to step ST14.
• step ST14 the imaging condition acquisition unit 96 acquires the shutter speed and aperture value corresponding to the brightness acquired in step ST12 based on the first table 112 stored in the storage 36 (see FIG. 6). .
• the imaging condition acquisition unit 96 also acquires the flight speed corresponding to the shutter speed acquired based on the first table 112 based on the second table 114 stored in the storage 36 (see FIG. 6). After the process of step ST14 is executed, the flight imaging process moves to step ST16.
• step ST16 the flight speed control unit 98 sets the flight speed of the flying object 20 to the flight speed acquired in step ST14 by controlling the plurality of motors 44 via the motor driver 46 (see FIG. 7). ). Thereby, the rotational speed of the plurality of propellers 42 is adjusted, and the flight speed of the flying object 20 is set to the flight speed acquired in step ST14.
• step ST18 the flight imaging process moves to step ST18.
• step ST18 the arrival determination unit 100 acquires the position of the flying object 20 based on the positioning data 48 input from the positioning unit 28 and/or the acceleration data 50 input from the acceleration sensor 30 (see FIG. 8). Then, the arrival determination unit 100 determines whether the aircraft 20 has reached the target waypoint 5 based on the acquired position of the aircraft 20 and the position of the waypoint 5 indicated by the flight route information 116 stored in the storage 36. (See FIG. 8). In step ST18, if the flying object 20 has reached the target waypoint 5, the determination is affirmative and the flight imaging process moves to step ST20. In step ST18, if the flying object 20 has not reached the target waypoint 5, the determination is negative and the flight imaging process executes the process of step ST18 again.
• step ST20 the imaging control unit 102 causes the imaging device 60 to image the imaging target area 3 corresponding to the target waypoint 5 using the shutter speed and aperture value acquired in step ST14 (see FIG. 8). After the process of step ST20 is executed, the flight imaging process moves to step ST22.
• step ST22 the processor 34 determines whether a condition for terminating the flight imaging process (termination condition) is satisfied.
• termination condition is that the imaging device 60 has captured an image of the imaging target area 3 corresponding to the final waypoint 5, or that the user has given an instruction to the flight imaging device 10 to end the flight imaging process. Examples of such conditions include: In step ST22, if the termination condition is not satisfied, the determination is negative and the flight imaging process moves to step ST10. In step ST22, if the termination condition is satisfied, the determination is affirmative and the flight imaging process is terminated.
• the control method described as the operation of the flight imaging device 10 described above is an example of a "control method" according to the technology of the present disclosure.
• the processor 34 causes the imaging device 60 to capture an image from the target waypoint 5 at a timing before the flying object 20 reaches the target waypoint 5.
• the brightness of the imaging target area 3 is acquired (see FIG. 5).
• the processor 34 also obtains a shutter speed corresponding to the obtained brightness and a flight speed corresponding to the shutter speed (see FIG. 6).
• the processor 34 causes the imaging device 60 to image the imaging target area 3 at the acquired shutter speed (see FIG. 8) while flying the flying object 20 at the acquired flight speed (see FIG. 7). ). Therefore, at the target waypoint 5, it is possible to cause the imaging device 60 to image the imaging target area 3 at a shutter speed corresponding to the brightness of the imaging target area 3 while flying the flying object 20 at a flight speed corresponding to the shutter speed. can.
• the processor 34 obtains the shutter speed corresponding to the brightness and the flight speed corresponding to the shutter speed based on the table 110 representing the relationship between brightness, shutter speed, and flight speed (see FIG. 6). ). Therefore, the shutter speed and flight speed predefined by table 110 can be obtained based on brightness.
• the computer 26 also includes a storage 36 that stores the table 110 (see FIG. 6). Therefore, for example, the shutter speed and flight speed can be obtained based on the table 110 stored in the storage 36 directly connected to the processor 34.
• the brightness is the brightness detected using the overhead camera 160 (see FIG. 5). Therefore, for example, the brightness of a plurality of imaging target regions 3 can be detected from a position farther from the wall surface 2A than the flying imaging device 10.
• the bird's-eye camera 160 images the entire wall surface 2A including the plurality of imaging target areas 3 (see FIG. 5). Therefore, for example, the number of times of imaging is reduced compared to the case where the brightness of each imaging target area 3 is detected individually based on images obtained by imaging a plurality of imaging target areas 3 one by one. Therefore, the workability when detecting the brightness of a plurality of imaging target areas 3 can be improved.
• the brightness is the brightness detected at the timing before the flying object 20 starts flying on the flight route 4 (see FIG. 5).
• a shutter speed corresponding to the brightness and a flight speed corresponding to the shutter speed are acquired. Therefore, for example, compared to the case where the brightness is detected at the timing when the flight imaging device 10 reaches the target waypoint 5, it is possible to set an appropriate shutter speed and flight speed at the target waypoint 5 (for example, a shutter speed that does not cause image blur). and flight speed) are set, the flight imaging device 10 can be caused to perform imaging.
• a plurality of waypoints 5 are set on the flight route 4, and the processor 34 sets the imaging device at an aperture value (see FIG. 6) corresponding to the brightness and/or shutter speed for each waypoint 5. 60 to image the imaging target area 3. Therefore, even if the conditions regarding brightness and/or shutter speed differ for each waypoint 5, it is possible to image the imaging target area 3 with an aperture value that corresponds to the brightness and/or shutter speed.
• the imaging target area 3 is captured by the imaging device 60, thereby obtaining a composite image 132 (see FIG. 8).
• the composite image 132 is an image in which adjacent composite images 132 partially overlap. Therefore, based on the feature points included in the overlapping portions of the adjacent imaging target regions 3, it is possible to combine the images 132 for composition corresponding to the adjacent imaging target regions 3.
• the processor 34 may cause the imaging device 60 to image the imaging target area 3 at a constant aperture value at a plurality of waypoints 5. In this case, the process for operating the aperture 64 for each waypoint 5 can be omitted.
• the bird's-eye camera 160 images the entire wall surface 2A, including the plurality of imaging target areas 3, at a timing before the flying object 20 starts flying along the flight route 4.
• the timing at which the wall surface 2A is imaged by the bird's-eye camera 160 is different from that in the first embodiment. This will be explained in detail below.
• the data requesting unit 92 sends the brightness data 168 at a timing after the aircraft 20 starts moving along the flight route 4 and before the aircraft 20 reaches the target waypoint 5.
• a request signal 104 for making a request is transmitted to the overhead camera 160 via the communication device 32.
• the brightness data 168 is data indicating the brightness of the imaging target area 3 corresponding to the target waypoint 5.
• the request signal 104 includes information indicating the waypoint number corresponding to the target waypoint 5.
• the overhead camera 160 When the overhead camera 160 receives the request signal 104, it images the entire wall surface 2A including the plurality of imaging target areas 3. As a result, a wall image 162 is obtained. Further, the overhead camera 160 converts the wall image 162 into a converted image 164, and derives the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness component included in the converted image 164. The method for deriving the brightness of the imaging target area 3 is the same as in the first embodiment. Then, the overhead camera 160 transmits brightness data 168 indicating the derived brightness to the communication device 32.
• the bird's-eye camera 160 may extract an image that includes only the imaging target area 3 corresponding to the waypoint number indicated by the request signal 104 from the wall surface image 162 obtained by imaging the entire wall surface 2A. good. Then, the overhead camera 160 converts the extracted image into a converted image 164, and derives the brightness of only the imaging target area 3 corresponding to the target waypoint 5 based on the brightness component included in the converted converted image 164. You may.
• the overhead camera 160 may image only the imaging target area 3 corresponding to the waypoint number indicated by the request signal 104. Then, the bird's-eye camera 160 converts the image obtained by imaging into a converted image 164, and based on the brightness component included in the converted converted image 164, only the imaging target area 3 corresponding to the target waypoint 5 is used. You may also derive the brightness of .
• the brightness acquisition unit 94 acquires the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness data 168 received by the communication device 32.
• the brightness acquired by the brightness acquisition unit 94 is determined by the brightness acquired by the brightness acquisition unit 94 when the flying object 20 reaches the target waypoint 5 after the flying object 20 starts moving along the flight route 4. This is the brightness detected at the previous timing.
• the shutter speed corresponding to the brightness is determined based on the brightness detected at a timing after the flying object 20 starts moving along the flight route 4 and before the flying object 20 reaches the target waypoint 5. A flight speed corresponding to the shutter speed is obtained.
• the flight imaging device 10 can be caused to perform imaging.
• the changed brightness can be detected at a timing before the flying object 20 reaches the target waypoint 5. Can be done. As a result, brightness detection accuracy can be improved, for example, compared to a case where brightness is detected before the flying object 20 starts moving along the flight route 4.
• the brightness of each imaging target region 3 corresponding to each waypoint 5 is detected based on the wall image 162 obtained by imaging with the overhead camera 160.
• the third embodiment differs from the first embodiment in the method of detecting the brightness of each imaging target area 3 corresponding to each waypoint 5. This will be explained in detail below.
• a flight imaging device 210 different from the flight imaging device 10 is used instead of the bird's-eye camera 160.
• the flight imaging device 10 will be referred to as the "first flight imaging device 10" and the flight imaging device 210 will be referred to as the "second flight imaging device 210.”
• the second flight imaging device 210 has the same hardware configuration as the first flight imaging device 10. Specifically, the second flight imaging device 210 includes a flying object 220 and an imaging device 260. The flying object 220 has the same configuration as the flying object 20 included in the first flight imaging device 10 , and the imaging device 260 has the same configuration as the imaging device 60 included in the first flight imaging device 10 .
• the flying object 220 provided in the second flight imaging device 210 is an example of a "second moving object" according to the technology of the present disclosure.
• the imaging device 260 provided in the second flight imaging device 210 is an example of a "sensor" according to the technology of the present disclosure.
• the second flight imaging device 210 performs a flight at a timing before the first flight imaging device 10 reaches the target waypoint 5 (for example, a timing before the first flight imaging device 10 starts flying on the flight route 4). While flying along the route 4, each time a waypoint 5 is reached, an image of the imaging target area 3 corresponding to each waypoint 5 is captured. By imaging each imaging target area 3, an image 262 (hereinafter referred to as "imaging target area image 262”) including the imaging target area 3 as an image is obtained.
• the second flight imaging device 210 flies the flight route 4 at a flight speed higher than the flight speed when the first flight imaging device 10 flies the flight route 4.
• the flight speed may be calculated, for example, as the average value of the flight speeds from the start to the end of the flight on the flight route 4.
• the second flight imaging device 210 derives the brightness of each imaging target area 3 corresponding to each waypoint 5 based on the imaging target area image 262 obtained each time the waypoint 5 is reached.
• the imaging target area image 262 is an RGB image formed by the three primary colors of light
• the second flying imaging device 210 converts the RGB image into the image 264.
• the image 264 will be referred to as a "converted image 264.”
• the converted image 264 is similar to the converted image 164 (see FIG. 5) of the first embodiment.
• the second flight imaging device 210 derives the brightness of each imaging target region 3 corresponding to each waypoint 5 based on the brightness component included in the converted image 264.
• the method of deriving the brightness for each imaging target region 3 is the same as in the first embodiment.
• the second flight imaging device 210 generates information regarding the brightness of each imaging target area 3 corresponding to each waypoint 5 as brightness information 266, and stores the generated brightness information 266.
• the brightness information 266 is similar to the brightness information 166 (see FIG. 5) of the first embodiment.
• FIG. 11 shows, as an example of the brightness information 266, an example of a specific brightness value of the imaging target area 3 corresponding to each waypoint number.
• the data requesting unit 92 transmits a request signal 104 for requesting brightness data 268 to the second flight imaging device 210 via the communication device 32.
• the brightness data 268 is data indicating the brightness of the imaging target area 3 corresponding to the target waypoint 5.
• the request signal 104 includes information indicating the waypoint number corresponding to the target waypoint 5.
• the second flight imaging device 210 When receiving the request signal 104, the second flight imaging device 210 extracts the brightness corresponding to the waypoint number indicated by the request signal 104 from the brightness information 266, and transfers the brightness data 268 indicating the extracted brightness to the second flight imaging device 210. 1 is transmitted to the communication device 32 of the flight imaging device 10.
• the waypoint number indicated by the request signal 104 is "NO.1”
• the brightness data 268 indicating "20" which is the degree of brightness corresponding to "NO.1”
• is the waypoint number indicated by the request signal 104. 2 is transmitted by the flight imaging device 210 to the communication device 32.
• the brightness acquisition unit 94 acquires the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness data 268 received by the communication device 32.
• the brightness acquired by the brightness acquisition unit 94 is the brightness detected at the timing before the first flight imaging device 10 starts moving along the flight route 4. be.
• a shutter speed corresponding to the brightness and a flight speed corresponding to the shutter speed are acquired. Therefore, for example, compared to the case where the brightness is detected at the timing when the flight imaging device 10 reaches the target waypoint 5, it is possible to set an appropriate shutter speed and flight speed at the target waypoint 5 (for example, a shutter speed that does not cause image blur). and flight speed) are set, the flight imaging device 10 can be caused to perform imaging.
• the brightness acquired by the brightness acquisition unit 94 is the brightness detected from the target waypoint 5 using the imaging device 60 of the second flight imaging device 210. Therefore, for example, brightness detection accuracy can be improved compared to brightness detected using the imaging device 60 from a position farther from the target waypoint 5 with respect to the wall surface 2A.
• the imaging device 60 is mounted on the second flight imaging device 210. Therefore, the brightness of the imaging target region 3 can be detected at each waypoint 5 while the second flying imaging device 210 is flying.
• the second flight imaging device 210 flies on the flight route 4 at a higher flight speed than the first flight imaging device 10. Therefore, the time required for the second flight imaging device 210 to fly the flight route 4 can be made shorter than the time required for the first flight imaging device 10 to fly the flight route 4.
• the imaging device 260 is used to detect the brightness of the imaging target region 3, but an illuminance sensor (not shown) may be used instead of the imaging device 260.
• the illuminance sensor is an example of a "sensor" according to the technology of the present disclosure.
• a second flight imaging device 210 that is different from the first flight imaging device 10 is used, but the first flight imaging device 10 is used instead of the second flight imaging device 210.
• the flying object 20 provided in the first flight imaging device 10 is an example of a "first moving object” and a "second moving object” according to the technology of the present disclosure.
• the second flight imaging device 210 flies the flight route 4 at a timing before the first flight imaging device 10 starts flying on the flight route 4, and each time the second flight imaging device 210 reaches the waypoint 5. , images the imaging target area 3 corresponding to each waypoint 5.
• the timing at which the imaging target area 3 corresponding to each waypoint 5 is imaged by the second flight imaging device 210 is different from that in the third embodiment. This will be explained in detail below.
• the second flying imaging device 210 flies ahead of the first flying imaging device 10.
• the distance between the centers of the first flight imaging device 10 and the second flight imaging device 210 along the flight route 4 may be set to be shorter than the distance between the centers of adjacent waypoints 5, and The distance may be set to be longer than the distance between points 5.
• the second flight imaging device 210 flies the flight route 4 in advance of the first flight imaging device 10 and images the imaging target area 3 corresponding to each waypoint 5 each time it reaches a waypoint 5. do. By imaging each imaging target area 3, an imaging target area image 262 is obtained.
• the second flight imaging device 210 converts the imaging target area image 262 obtained each time it reaches a waypoint 5 into a converted image 264, and corresponds to each waypoint 5 based on the brightness component included in the converted image 264.
• the brightness of each imaging target area 3 is derived.
• the method of deriving the brightness for each imaging target region 3 is the same as in the third embodiment.
• the second flight imaging device 210 generates information regarding the brightness of each imaging target area 3 corresponding to each waypoint 5 as brightness information 266, and stores the generated brightness information 266.
• Brightness information 266 is the same as in the third embodiment.
• FIG. 12 shows, as an example of the brightness information 266, an example of a specific brightness value of the imaging target area 3 corresponding to each waypoint number.
• the data requesting unit 92 requests the brightness data 268 after the first flight imaging device 10 starts moving along the flight route 4 and before the first flight imaging device 10 reaches the target waypoint 5.
• a request signal 104 is transmitted to the second flight imaging device 210 via the communication device 32.
• the brightness data 268 is data indicating the brightness of the imaging target area 3 corresponding to the target waypoint 5.
• the request signal 104 includes information indicating the waypoint number corresponding to the target waypoint 5.
• the second flight imaging device 210 When receiving the request signal 104, the second flight imaging device 210 extracts the brightness corresponding to the waypoint number indicated by the request signal 104 from the brightness information 266, and transfers the brightness data 268 indicating the extracted brightness to the second flight imaging device 210. 1 is transmitted to the communication device 32 of the flight imaging device 10.
• the waypoint number indicated by the request signal 104 is "NO.3”
• the brightness data 268 indicating "60" which is the degree of brightness corresponding to "NO.3” is the waypoint number indicated by the request signal 104. 2 is transmitted by the flight imaging device 210 to the communication device 32.
• the brightness acquisition unit 94 acquires the brightness of the imaging target area 3 corresponding to the target waypoint 5 based on the brightness data 268 received by the communication device 32.
• the brightness acquired by the brightness acquisition unit 94 is determined by the brightness acquired by the brightness acquisition unit 94 after the first flight imaging device 10 starts moving along the flight route 4. This is the brightness detected before reaching waypoint 5.
• the shutter speed corresponding to the brightness is determined based on the brightness detected at a timing after the flying object 20 starts moving along the flight route 4 and before the flying object 20 reaches the target waypoint 5.
• a flight speed corresponding to the shutter speed is obtained. Therefore, for example, compared to the case where the brightness is detected at the timing when the flight imaging device 10 reaches the target waypoint 5, it is possible to set an appropriate shutter speed and flight speed at the target waypoint 5 (for example, a shutter speed that does not cause image blur). and flight speed) are set, the flight imaging device 10 can be caused to perform imaging.
• the changed brightness can be detected at a timing before the flying object 20 reaches the target waypoint 5. Can be done. As a result, brightness detection accuracy can be improved, for example, compared to a case where brightness is detected before the flying object 20 starts moving along the flight route 4.
• a processor 34 included in the flight imaging device 10 obtains a plurality of images for synthesis 132 and generates a synthesized image 130 by synthesizing adjacent images for synthesis 132.
• the processor 34 also performs correction processing on the composite image 130.
• the correction process will be specifically described below as a "specific process" according to the technology of the present disclosure.
• a composite image generation program 120 is stored in the storage 36.
• the processor 34 reads the composite image generation program 120 from the storage 36 and executes the read composite image generation program 120 on the RAM 38.
• the processor 34 performs composite image generation processing according to the composite image generation program 120 executed on the RAM 38.
• the composite image generation process is realized by the processor 34 operating as an image composition section 122, a brightness information acquisition section 124, and a correction processing section 126 according to the composite image generation program 120.
• the processor 34 is an example of a "second processor" according to the technology of the present disclosure.
• the image synthesis unit 122 combines adjacent images for synthesis 132 with respect to a plurality of images for synthesis 132 obtained by capturing each imaging target area 3.
• a composite image 130 is generated.
• the brightness information acquisition unit 124 acquires brightness information 166 generated by the overhead camera 160.
• the brightness information 166 is information generated based on the brightness component included in the converted image 164, and calculates the brightness of each imaging target area 3 corresponding to each waypoint 5. This is the information shown.
• the correction processing unit 126 performs correction processing on the composite image 130 synthesized by the image composition unit 122.
• the correction process is a process of correcting the brightness of the composite image 130 based on the brightness information 166 acquired by the brightness information acquisition unit 124.
• the brightness is corrected for areas of the composite image 130 excluding areas indicating feature points.
• a correction process a process of lowering the brightness of an area of the composite image 130 with the highest brightness (for example, an area where overexposure occurs) is executed. Further, for example, as a correction process, a process of correcting a difference in brightness between adjacent images for synthesis 132 is executed. Further, for example, as a correction process, a process of correcting the difference in brightness between the composite images 132 located at separate positions is executed. Examples of the composite images 132 located at separate positions include composite images 132 or composite images 130 in a positional relationship in which one composite image 132 is placed between composite images 132 located at separate positions. An example of this is the composition image 132 positioned at two of the four corners of the image.
• An example of the process for correcting the difference in brightness between the images for synthesis 132 is a process for bringing the brightness between the images for synthesis 132 closer together. Further, for example, as a correction process, a process of correcting the brightness distribution of the composite image 130 based on the brightness distribution of the plurality of imaging target areas 3 is executed. Examples of the process for correcting the brightness distribution of the composite image 130 include a process for correcting uneven brightness. The above correction processing is performed on the composite image 130.
• the correction process is an example of a "specific process” and a "correction process" according to the technology of the present disclosure.
• the composite image generation process may be executed, for example, after all the composite images 132 are obtained for the wall surface 2A, or may be executed every time the composite images 132 from the second frame onwards are obtained. good.
• FIG. 15 shows an example of the flow of composite image generation processing according to the fifth embodiment.
• step ST30 the image composition unit 122 selects adjacent composite images 132 for a plurality of composite images 132 obtained by capturing each imaging target region 3.
• a composite image 130 is generated by combining images 132.
• step ST32 the brightness information acquisition unit 124 acquires the brightness information 166 generated by the overhead camera 160. After the process of step ST32 is executed, the composite image generation process moves to step ST34.
• step ST34 the correction processing unit 126 corrects the brightness of the composite image 130 generated in step ST30 by executing a correction process based on the brightness information 166 acquired in step ST32. As a result, a composite image 130 whose brightness has been corrected is obtained. After the process of step ST34 is executed, the composite image generation process ends.
• the composite image 130 is generated by combining adjacent composite images 132. Then, the brightness of the composite image 130 is corrected by performing a correction process on the composite image 130 based on the brightness information 166 indicating the brightness of each imaging target area 3. Therefore, for example, it is possible to suppress the appearance of the composite image 130 from worsening due to uneven brightness caused by combining adjacent composite images 132.
• the correction process includes a process of correcting the difference in brightness between the images for synthesis 132. Therefore, it is possible to obtain a composite image 130 in which the difference in brightness between the composite images 132 is corrected based on the brightness of each imaging target region 3.
• the correction process includes a process of correcting the brightness distribution of the composite image 130 based on the brightness distribution of the plurality of imaging target areas 3. Therefore, it is possible to obtain a composite image 130 in which the brightness distribution is corrected based on the brightness distribution of the plurality of imaging target regions 3.
• the composite image generation process is executed in the flight imaging device 10, but it may be executed in an external device (not shown) that is communicably connected to the flight imaging device 10.
• the specific processing may include a notification process for notifying when the difference in brightness between the imaging target regions 3 exceeds a predetermined value.
• the default value is set, for example, to the upper limit of the difference in brightness that does not affect when inspecting or surveying the wall surface 2A based on the composite image 130.
• a process of emitting a warning sound may be executed.
• the notification process is an example of the "specific process” and the "notification process” according to the technology of the present disclosure. In this way, when the notification process is performed to notify when the difference in brightness between the imaging target areas 3 exceeds the default value, the user etc. are informed that the difference in brightness between the imaging target areas 3 exceeds the default value. It can make you realize that you have surpassed it.
• the flying imaging device 10 is illustrated as an example of a moving object, but any moving object may be used as long as it moves on a moving route.
• the moving object include a car, motorcycle, bicycle, trolley, gondola, airplane, flying object, or ship.
• the plurality of waypoints 5 refers to all the waypoints 5 set on the flight route 4, but some of the waypoints 5 set on the flight route 4 You may point to waypoint 5. Moreover, the number of waypoints 5 may be any number.
• the waypoint 5 set on the flight route 4 is used as an example of the target position, but a target position that is a position with a different concept from the waypoint 5 may also be used. .
• the imaging device 60 images the imaging target region 3 in order to obtain the composite image 132, but it may also image the imaging target region 3 for purposes other than obtaining the composite image 132. .
• the processor 34 is illustrated, but in place of the processor 34 or together with the processor 34, at least one other CPU, at least one GPU, and/or at least one TPU is used. You can do it like this.
• the flight imaging program 90 and the composite image generation program 120 are stored in the storage 36, but the technology of the present disclosure is not limited to this.
• the flight imaging program 90 and/or the composite image generation program 120 are stored on a portable, non-transitory, computer-readable storage medium (hereinafter simply referred to as a "non-transitory storage medium") such as an SSD or a USB memory. may have been done.
• the flight imaging program 90 and/or the composite image generation program 120 stored in a non-transitory storage medium may be installed on the computer 26 of the flight imaging device 10.
• the flight imaging program 90 and/or the composite image generation program 120 may be stored in a storage device such as another computer or a server device connected to the flight imaging device 10 via a network, and the flight imaging program 90 and/or the composite image generation program 120 may be stored in response to requests from the flight imaging device 10. Accordingly, the flight imaging program 90 and/or the composite image generation program 120 may be downloaded and installed on the computer 26.
• flight imaging program 90 and/or composite image generation program 120 it is not necessary to store all of the flight imaging program 90 and/or composite image generation program 120 in a storage device such as another computer or server device connected to the flight imaging device 10, or in the storage 36; Part of the program 90 and/or the composite image generation program 120 may be stored.
• the flight imaging device 10 has a built-in computer 26, the technology of the present disclosure is not limited to this, and for example, the computer 26 may be provided outside the flight imaging device 10.
• the computer 26 including the processor 34, the storage 36, and the RAM 38 is illustrated, but the technology of the present disclosure is not limited to this, and instead of the computer 26, an ASIC, an FPGA, and/or A device including a PLD may also be applied. Further, instead of the computer 26, a combination of hardware configuration and software configuration may be used.
• processors can be used as hardware resources for executing the various processes described in the above embodiments.
• the processor include a CPU, which is a general-purpose processor that functions as a hardware resource that executes various processes by executing software, that is, a program.
• the processor include a dedicated electronic circuit such as an FPGA, a PLD, or an ASIC, which is a processor having a circuit configuration specifically designed to execute a specific process.
• Each processor has a built-in memory or is connected to it, and each processor uses the memory to perform various processes.
• Hardware resources that execute various processes may be configured with one of these various processors, or a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or a CPU and FPGA). Furthermore, the hardware resource that executes various processes may be one processor.
• one processor is configured by a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes various processes.
• a and/or B has the same meaning as “at least one of A and B.” That is, “A and/or B” means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, even when three or more items are expressed by connecting them with “and/or”, the same concept as “A and/or B" is applied.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Studio Devices (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=89617607

Family Applications (1)

Country Status (4)

Citations (3)

• 2023
  • 2023-03-29 JP JP2024534927A patent/JPWO2024018691A1/ja active Pending
  • 2023-03-29 WO PCT/JP2023/012978 patent/WO2024018691A1/ja not_active Ceased
  • 2023-03-29 CN CN202380053628.5A patent/CN119563327A/zh active Pending
• 2025
  • 2025-01-14 US US19/021,107 patent/US20250159355A1/en active Pending

Patent Citations (3)

Also Published As

Similar Documents

Legal Events