Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/61—Control of cameras or camera modules based on recognised objects
• • G—PHYSICS
  • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
  • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
  • G03B15/00—Special procedures for taking photographs; Apparatus therefor
• • G—PHYSICS
  • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
  • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
  • G03B37/00—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/20—Analysis of motion
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30168—Image quality inspection

Definitions

• the technology of the present disclosure relates to an imaging control device, an imaging control method, and a program.
• Japanese Patent Application Publication No. 2018-151775 discloses a method of creating a distribution map of physical quantities that differs from place to place within a target range.
• the disclosed method includes a movement measurement step, a physical quantity setting step, an orthogonal image creation step, and a distribution map creation step.
• the moving measurement step is a step of acquiring a plurality of ground images by photographing the ground so that adjacent images overlap while moving in the target range, and measuring physical quantities.
• the physical quantity setting step is a step of assigning a representative physical quantity to each ground image based on the physical quantity obtained in the movement measurement step.
• the orthogonal image creation step is a step of creating an orthogonal image of the target range based on a plurality of ground images.
• the distribution map creation step is a step of creating a physical quantity distribution map by displaying representative physical quantities on the orthogonal image.
• JP 2020-113843A discloses an image capturing support device that supports capturing multi-view images used to restore a three-dimensional shape model of a target object.
• the image capturing support device includes a feature point extraction section that extracts feature points, a matching processing section, and a support information notification section.
• the feature point extraction unit extracts feature points in captured image data, which is image data of the target object captured immediately before, and preview image data.
• the matching processing unit detects a first corresponding point of each feature point of the captured image data and the preview image data.
• the support information notification unit displays a preview image of the preview image data on which the first corresponding points are superimposed, and notifies the preview image of support information corresponding to imaging.
• JP 2010-045587A discloses a camera device.
• the camera device includes an image capturing section, an image display section, a shake detection section, an image recording section, a relative relationship calculation section, a display control section, an overlap calculation section, a notification section, and a shooting control section.
• the image photographing unit photographs an image.
• the image display section includes at least a screen that displays images.
• the shake detection unit detects device shake when the image capture unit captures an image.
• the image recording section records information about images photographed by the image photographing section.
• the relative relationship calculation section calculates the photographing range of the first image photographed immediately before by the image photographing section and recorded in the image recording section, and the photographing range of the second image photographed by the image photographing section after the first image.
• a relative relationship degree parameter representing at least a relative positional relationship between the two is determined.
• the display control unit generates an image for clearly indicating the relative positional relationship between the shooting ranges from the relative relationship degree parameter determined by the relative relationship calculation unit, and displays the image on the screen of the image display unit together with the second image. Display.
• the overlap calculation unit calculates an overlap degree parameter that represents the degree of overlap between the shooting range of the first image and the shooting range of the second image.
• the notification unit provides a predetermined notification to the photographer according to the overlap degree parameter determined by the overlap calculation unit.
• the image capture control section activates the image capture section. to take an image.
• the imaging control device includes a wide-angle image acquisition unit, a photography information acquisition unit, a glue margin information acquisition unit, an area information acquisition unit, a photography area calculation unit, and a control unit.
• the wide-angle image acquisition unit acquires a wide-angle image in which the entire image of the object to be photographed is photographed at a wide angle.
• the photographing information acquisition unit acquires photographing information regarding the number of photographic images or the photographing angle of view of a plurality of divided images obtained by photographing a part of the entire image of the photographing object at close range with a camera of a moving object.
• the glue margin information acquisition unit acquires glue margin information regarding a glue margin when a composite image of a photographing target is generated by combining a plurality of divided images.
• the area information acquisition unit acquires imaging target area information regarding a region of the entire image of the imaging target.
• the photographing area calculation unit calculates the respective photographing areas of the divided images constituting the composite image based on the photographing information, the glue allowance information, and the photographing target area information, and calculates each photographing area of each wide-angle image in which the glue allowance is secured. Calculate the area.
• the control unit moves the mobile object, causes the camera to take close-up shots of each of the calculated shooting areas, and obtains the shot close-up images as divided images.
• the control unit compares an image corresponding to each photographing area of the acquired wide-angle image with an image photographed in close proximity by the camera, and controls the position of the moving body that causes the camera to photograph each photographing area in close proximity.
• the image alignment method includes the steps of obtaining position information from a device, obtaining first and second images from the device, and aligning a plurality of regions in the first image with a plurality of corresponding regions in a second image. identifying a plurality of corresponding regions; determining a search vector for each of the plurality of corresponding regions; and selecting from the plurality of corresponding regions only those corresponding regions having search vectors that match the position information; The method includes identifying a plurality of matching regions and registering the first and second images using the plurality of matching regions.
• One embodiment of the technology of the present disclosure provides an imaging control device, an imaging control method, and a program that can image a third imaging target area even if overlap imaging processing fails.
• a first aspect of the technology of the present disclosure includes a processor, and the processor causes an imaging device to image a first imaging target region, and in a process in which a moving object on which the imaging device is mounted moves, a second imaging target region overlaps with a part of the first imaging target area, performs overlap imaging processing that causes the imaging device to image the second imaging target area, and if the overlap imaging process fails, performs the imaging process.
• Interval imaging processing that causes the imaging device to image a third imaging target area on the condition that the moving distance traveled by the moving object from the first position where the first imaging target area was imaged by the device reaches a first predetermined moving distance.
• a second aspect of the technology of the present disclosure is that in the imaging control device according to the first aspect, a case where the overlap imaging process fails is a case where the second imaging target area is not imaged by the imaging device.
• the imaging control device includes a case where the moving distance exceeds the distance from the first position to the second position where the second imaging target area is imaged by the imaging device.
• a case where the overlap imaging process fails means that the second imaging target area is not imaged by the imaging device.
• the imaging control device includes a case where the first overlap amount to be overlapped is outside the first predetermined range.
• a fourth aspect according to the technology of the present disclosure is that in the imaging control device according to any one of the first to third aspects, a case where the overlap imaging process fails means that the imaging device
• the imaging control device includes a case where the third image obtained by imaging the imaging target area does not satisfy the predetermined image quality.
• a fifth aspect of the technology of the present disclosure is that in the imaging control device according to any one of the first to fourth aspects, a part of the third imaging target area is a part of the second imaging target area. This is an imaging control device that partially overlaps with the other parts.
• a sixth aspect of the technology of the present disclosure is that in the imaging control device according to any one of the first to fifth aspects, the first predetermined movement distance is from the first position to the third imaging target area. is the distance to a third position where a part of the second imaging target area overlaps with a part of the second imaging target area.
• a seventh aspect according to the technology of the present disclosure is that in the imaging control device according to any one of the first to sixth aspects, the first predetermined movement distance is set by the imaging device from the first position to the second position.
• the distance of the imaging target area is a natural number times 2 or more of the distance to the fourth position where the imaging target area is imaged.
• An eighth aspect according to the technology of the present disclosure is that in the imaging control device according to any one of the first to seventh aspects, the overlap imaging process is such that the first imaging target area is imaged by the imaging device.
• a second overlap amount in which a part of the fourth image obtained by imaging the second imaging target area overlaps a part of the fifth image obtained by imaging the second imaging target area is within the second predetermined range.
• a ninth aspect of the technology of the present disclosure is that in the imaging control device according to any one of the first to eighth aspects, the movement distance is determined by an acceleration mounted on the imaging device and/or the moving body. This is an imaging control device that is derived based on acceleration measured by a sensor.
• a tenth aspect of the technology of the present disclosure is that in the imaging control device according to any one of the first to ninth aspects, when the moving distance reaches the first predetermined moving distance, The image capturing control device is configured to be determined on the condition that when the image capturing device moves at a constant speed, the time elapsed from the first timing when the first image capturing target area was imaged by the image capturing device reaches the first predetermined time.
• An eleventh aspect according to the technology of the present disclosure is the imaging control device according to any one of the first to tenth aspects, in which the moving distance is a plurality of distances obtained by imaging by the imaging device.
• This is an imaging control device that derives the moving speed of the moving object based on the sixth image and the time interval when a plurality of sixth images are obtained.
• a twelfth aspect according to the technology of the present disclosure is that in the imaging control device according to any one of the first to eleventh aspects, when the processor fails in the overlap imaging process, the processor Position information regarding the position of the target area, first image information regarding the seventh image obtained by capturing the first image target area by the imaging device, and first image information regarding the seventh image obtained by capturing the third image target area by the imaging device.
• the second image information regarding the obtained eighth image is acquired, and the position information regarding the position of the second imaging target area is stored in the memory in association with image information of at least one of the first image information and the second image information. This is an imaging control device stored in the .
• a thirteenth aspect of the technology of the present disclosure is the imaging control device according to any one of the first to twelfth aspects, wherein the processor acquires the moving speed of the moving body and indicates the moving speed.
• the imaging control device outputs movement speed data and derives the movement speed based on a plurality of ninth images obtained by imaging with an imaging device.
• a fourteenth aspect of the technology of the present disclosure is to cause the imaging device to image the first imaging target region, and in the process of moving the moving object on which the imaging device is mounted, a part of the second imaging target region is transferred to the first imaging target region.
• Performing overlap imaging processing that causes the imaging device to image a second imaging target region when the region overlaps with a part of the imaging target region, and performing overlap imaging processing that causes the imaging device to image a second imaging target region when the overlap imaging processing fails Performing interval imaging processing that causes the imaging device to image a third imaging target area on the condition that the distance traveled by the moving body from the first position where the first imaging target area is imaged reaches a first predetermined moving distance.
• a fifteenth aspect of the technology of the present disclosure is to cause an imaging device to image a first imaging target region, and in a process in which a moving body on which the imaging device is mounted moves, a part of the second imaging target region is transferred to the first imaging target region.
• This is a program that causes a computer to execute processing that includes.
• FIG. 2 is a block diagram showing an example of the hardware configuration of an imaging device.
• FIG. 2 is a perspective view showing an example of a flight imaging device, a first imaging target area, a second imaging target area, and a third imaging target area.
• FIG. 2 is a block diagram illustrating an example of a functional configuration of an imaging device for executing imaging processing.
• FIG. 3 is an explanatory diagram illustrating an example of a first operation of a processor in imaging processing.
• FIG. 7 is an explanatory diagram illustrating an example of a second operation of the processor in imaging processing.
• FIG. 7 is an explanatory diagram illustrating an example of a third operation of the processor in image capturing processing.
• FIG. 3 is an explanatory diagram illustrating an example of a first operation of a processor in imaging processing.
• FIG. 7 is an explanatory diagram illustrating an example of a second operation of the processor in imaging processing.
• FIG. 7 is an explanatory diagram illustrating an example of a third operation of the processor in
• FIG. 12 is an explanatory diagram illustrating an example of a fourth operation of the processor in imaging processing.
• FIG. 12 is an explanatory diagram illustrating an example of a fifth operation of the processor in imaging processing.
• FIG. 12 is an explanatory diagram illustrating an example of a sixth operation of the processor in imaging processing. It is an explanatory view explaining an example of the 7th operation of a processor in imaging processing. It is an explanatory view explaining an example of the 8th operation of a processor in imaging processing.
• FIG. 12 is an explanatory diagram illustrating an example of a ninth operation of the processor in imaging processing.
• FIG. 2 is a block diagram illustrating an example of a functional configuration of an imaging device for executing re-imaging processing.
• FIG. 6 is an explanatory diagram illustrating an example of a first operation of a processor in re-imaging processing.
• FIG. 7 is an explanatory diagram illustrating an example of a second operation of the processor in re-imaging processing.
• FIG. 7 is an explanatory diagram illustrating an example of a third operation of the processor in re-imaging processing.
• FIG. 7 is an explanatory diagram illustrating an example of a fourth operation of the processor in re-imaging processing.
• FIG. 7 is an explanatory diagram illustrating an example of a fifth operation of the processor in re-imaging processing.
• 3 is a flowchart illustrating an example of the flow of imaging processing.
• 7 is a flowchart illustrating an example of the flow of re-imaging processing.
• I/F is an abbreviation for "Interface”.
• RAM is an abbreviation for "Random Access Memory.”
• EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory.”
• CPU is an abbreviation for "Central Processing Unit.”
• HDD is an abbreviation for “Hard Disk Drive.”
• SSD is an abbreviation for “Solid State Drive.”
• DRAM is an abbreviation for "Dynamic Random Access Memory.”
• SRAM is an abbreviation for "Static Random Access Memory.”
• CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor.”
• GPU is an abbreviation for “Graphics Processing Unit.”
• TPU is an abbreviation for “Tensor Processing Unit”.
• USB is an abbreviation for “Universal Serial Bus.”
• ASIC is an abbreviation for “Application Specific Integrated Circuit.”
• FPGA is an abbreviation for “Field-Programmable Gate Array.”
• PLD is an abbreviation for “Programmable Logic Device”.
• SoC is an abbreviation for "System-on-a-chip.”
• IC is an abbreviation for "Integrated Circuit.”
• Constant means not only a completely constant error but also an error that is generally allowed in the technical field to which the technology of the present disclosure belongs, and that does not go against the spirit of the technology of the present disclosure. It refers to a constant in the sense of including the error of.
• Vertical means not only perfectly perpendicular, but also includes errors that are generally allowed in the technical field to which the technology of the present disclosure belongs, and that do not go against the spirit of the technology of the present disclosure. Points vertically.
• the term “horizontal direction” refers to an error that is generally allowed in the technical field to which the technology of the present disclosure belongs, in addition to a completely horizontal direction, and is contrary to the spirit of the technology of the present disclosure.
• vertical direction refers to an error that is generally allowed in the technical field to which the technology of the present disclosure belongs, in addition to a perfect vertical direction, and is contrary to the spirit of the technology of the present disclosure. Refers to the vertical direction with a certain degree of error.
• the flight imaging device 1 has a flight function and an imaging function, and images the wall surface 2A of the object 2 while flying.
• the concept of "flight” includes not only the meaning that the flying imaging device 1 moves in the air, but also the meaning that the flying imaging device 1 stands still in the air.
• the wall surface 2A is, for example, a flat surface.
• a plane refers to a two-dimensional surface (that is, a surface along a two-dimensional direction). Furthermore, in the description of this specification, the concept of "plane" does not include the meaning of mirror surface.
• the wall surface 2A is a plane defined in the horizontal direction and the vertical direction (that is, a surface extending in the horizontal direction and the vertical direction).
• the wall surface 2A includes unevenness.
• the unevenness referred to here includes, for example, unevenness due to the material forming the wall surface 2A, as well as unevenness due to defects and/or defects.
• the object 2 having the wall surface 2A is a pier provided on a bridge.
• the piers are made of reinforced concrete, for example.
• a bridge pier is mentioned here as an example of the target object 2, the target object 2 may be an object other than a bridge pier (for example, a tunnel or a dam).
• the flight function (hereinafter also simply referred to as "flight function") of the flight imaging device 1 is a function in which the flight imaging device 1 flies based on a flight instruction signal.
• the flight instruction signal refers to a signal that instructs the flight imaging device 1 to fly.
• the flight instruction signal is transmitted, for example, from a transmitter 20 for controlling the flight imaging device 1.
• the transmitter 20 is operated by a user (not shown).
• the transmitter 20 includes a control section 22 for controlling the flight imaging device 1 and a display device 24 for displaying various images and/or information.
• the display device 24 is, for example, a liquid crystal display.
• the flight instruction signal may also be transmitted from a base station (not shown) or the like that sets a flight route for the flight imaging device 1. It's okay.
• the imaging function (hereinafter also simply referred to as "imaging function") of the flight imaging device 1 is a function for the flight imaging device 1 to image a subject (for example, the wall surface 2A of the object 2).
• the flight imaging device 1 includes a flying object 10 and an imaging device 30.
• the flying object 10 is, for example, an unmanned aircraft such as a drone. Flight functions are realized by the aircraft 10.
• the flying object 10 has a plurality of propellers 12, and flies when the plurality of propellers 12 rotate. Flying the flying object 10 is synonymous with flying the flying imaging device 1.
• the flying object 10 is an example of a "mobile object" according to the technology of the present disclosure.
• the imaging device 30 is, for example, a digital camera or a video camera.
• the imaging function is realized by the imaging device 30.
• the imaging device 30 is mounted on the aircraft 10. Specifically, the imaging device 30 is provided at the bottom of the flying object 10.
• the imaging device 30 is provided at the lower part of the aircraft 10, but the imaging device 30 may be provided at the upper part or the front part of the aircraft 10.
• the flight imaging device 1 sequentially images a plurality of imaging target areas 3 on the wall surface 2A.
• the imaging target area 3 is an area determined by the angle of view of the flight imaging device 1.
• a rectangular area is shown as an example of the imaging target area 3.
• a plurality of images for synthesis 92 are obtained by sequentially capturing images of the plurality of imaging target regions 3 by the imaging device 30.
• a composite image 90 is generated by combining a plurality of images 92 for composition.
• the plurality of images for synthesis 92 are synthesized so that adjacent images for synthesis 92 partially overlap each other.
• An example of the composite image 90 is a two-dimensional panoramic image.
• the two-dimensional panoramic image is just an example, and a three-dimensional image (for example, a three-dimensional panoramic image) is generated as the composite image 90 in the same manner as a two-dimensional panoramic image is generated as the composite image 90. You may also do so.
• the composite image 90 may be generated each time each composite image 92 from the second frame onward is obtained, or may be generated after a plurality of composite images 92 are obtained for the wall surface 2A. Further, the process of generating the composite image 90 may be executed by the flight imaging device 1, or may be executed by an external device (not shown) communicably connected to the flight imaging device 1. The composite image 90 is used, for example, to inspect or survey the wall surface 2A of the object 2.
• FIG. 1 shows a mode in which each imaging target area 3 is imaged by the imaging device 30 in a state where the optical axis OA of the imaging device 30 is perpendicular to the wall surface 2A.
• the following description will be given on the premise that each imaging target area 3 is imaged by the imaging device 30 in a state where the optical axis OA of the imaging device 30 is perpendicular to the wall surface 2A.
• the plurality of imaging target regions 3 are imaged so that adjacent imaging target regions 3 partially overlap each other.
• the plurality of imaging target areas 3 are imaged so that the adjacent imaging target areas 3 partially overlap each other, based on the feature points included in the overlapping parts of the adjacent imaging target areas 3. This is to synthesize a synthesis image 92 corresponding to No. 3.
• overlap the fact that adjacent imaging target regions 3 partially overlap with each other, and that adjacent compositing images 92 partially overlap with each other may be referred to as "overlap".
• the flight imaging device 1 moves in a zigzag pattern by alternately repeating horizontal movement and vertical movement. As a result, a plurality of imaging target regions 3 connected in a zigzag pattern are sequentially imaged.
• tape measures 4 are provided at both ends of the wall surface 2A in the horizontal direction. A tape measure 4 is suspended from the top of the object 2. The tape measure 4 is provided on both sides of the plurality of imaging target regions 3 in the horizontal direction. The user moves the flight imaging device 1 in the horizontal direction and the vertical direction by operating the flight imaging device 1 based on the scale marked on the tape measure 4.
• the imaging device 30 includes a computer 32, an image sensor 34, an image sensor driver 36, an imaging lens 38, and an input/output I/F 40.
• the computer 32 includes a processor 42, a storage 44, and a RAM 46.
• the computer 32 is an example of an "imaging control device” and a “computer” according to the technology of the present disclosure.
• the processor 42 is an example of a "processor” according to the technology of the present disclosure.
• the processor 42, storage 44, and RAM 46 are interconnected via a bus 48, and the bus 48 is connected to the input/output I/F 40. Further, an image sensor 34, an image sensor driver 36, and an imaging lens 38 are connected to the input/output I/F 40.
• the processor 42 includes, for example, a CPU, and controls the entire imaging device 30.
• the storage 44 is a nonvolatile storage device that stores various programs, various parameters, and the like. Examples of the storage 44 include an HDD and/or a flash memory (eg, EEPROM and/or SSD).
• the RAM 46 is a memory in which information is temporarily stored, and is used by the processor 42 as a work memory. Examples of the RAM 46 include DRAM and/or SRAM.
• the image sensor 34 is connected to an image sensor driver 36.
• Image sensor driver 36 controls image sensor 34 according to instructions from processor 42 .
• the image sensor 34 is, for example, a CMOS image sensor. Note that although a CMOS image sensor is exemplified here as the image sensor 34, the technology of the present disclosure is not limited to this, and other image sensors may be used.
• the image sensor 34 captures an image of a subject (for example, the wall surface 2A of the target object 2) under the control of the image sensor driver 36, and outputs image data obtained by capturing the image.
• the imaging lens 38 is arranged closer to the subject (object side) than the image sensor 34.
• the imaging lens 38 takes in object light that is reflected light from the object, and forms an image of the taken-in object light on the imaging surface of the image sensor 34 .
• the imaging lens 38 includes a plurality of optical elements (not shown) such as a focus lens, a zoom lens, and an aperture.
• the imaging lens 38 is connected to the computer 32 via an input/output I/F 40.
• the plurality of optical elements included in the imaging lens 38 are connected to the input/output I/F 40 via a drive mechanism (not shown) having a power source.
• a plurality of optical elements included in the imaging lens 38 operate under the control of the computer 32.
• focus, optical zoom, exposure adjustment, and the like are realized by operating a plurality of optical elements (for example, various lenses, an aperture, etc.) included in the imaging lens 38.
• FIG. 3 shows a first imaging target area 3A, a second imaging target area 3B, and a third imaging target area 3C that are connected in the horizontal direction among the plurality of imaging target areas 3.
• a part of the first imaging target area 3A overlaps with a part of the second imaging target area 3B, and a part of the second imaging target area 3B overlaps with a part of the third imaging target area 3C.
• the flight imaging device 1 performs imaging at the timing when it is determined that the predetermined imaging conditions are satisfied.
• the predetermined imaging conditions include a condition that the amount of overlap between parts of adjacent imaging target regions 3 is within a predetermined range.
• the predetermined range is set in consideration of the efficiency in sequentially capturing images of a plurality of imaging target regions 3, the number of feature points required for composing adjacent compositing images 92, and the like.
• the flight imaging device 1 When the positioning of the flight imaging device 1 is stable and the flight imaging device 1 is moving normally, a part of the second imaging target area 3B overlaps a part of the first imaging target area 3A. When the second imaging target area 3B is imaged and a part of the third imaging target area 3C overlaps a part of the second imaging target area 3B, the third imaging target area 3C is imaged. Thereby, the flight imaging device 1 sequentially images the first imaging target area 3A, the second imaging target area 3B, and the third imaging target area 3C.
• the flying imaging device 1 when the flying imaging device 1 is moving, the positioning of the flying imaging device 1 may become unstable due to disturbances such as wind acting on the flying imaging device 1, for example. In this way, when the positioning of the flight imaging device 1 becomes unstable, for example, the flight imaging device 1 performs a process (hereinafter referred to as "overload") of imaging the second imaging target area 3B after imaging the first imaging target area 3A. It is assumed that the process (referred to as "lap imaging processing”) will fail.
• An example of a failure in the overlap imaging process is, for example, the position where the second imaging target area 3B is imaged from the position where the first imaging target area 3A is imaged before it is determined that the predetermined imaging condition is satisfied. Examples include cases in which the distance traveled by the flight imaging device 1 exceeds the distance up to.
• the flight imaging device 1 it is conceivable to cause the flight imaging device 1 to image the third imaging target area 3C on the condition that the overlap imaging process for the second imaging target area 3B is successful.
• the processor 42 executes the following imaging process in order to eliminate the above-mentioned inconvenience.
• an imaging program 50 is stored in the storage 44.
• the imaging program 50 is an example of a "program" according to the technology of the present disclosure.
• the processor 42 reads the imaging program 50 from the storage 44 and executes the read imaging program 50 on the RAM 46.
• the processor 42 performs imaging processing according to an imaging program 50 executed on the RAM 46.
• the imaging process is started every time the flight imaging device 1 starts moving in the horizontal direction.
• the flight imaging device 1 starts moving in the horizontal direction when the flight imaging device 1 receives a flight instruction signal to move at a constant speed from the transmitter 20 (see FIG. 1). do.
• the processor 42 executes a first imaging control section 52, a second imaging control section 54, a first overlap determination section 56, a lost determination section 58, a third imaging control section 60, and a second overlap determination section according to the imaging program 50.
• the determination unit 62 By operating as the determination unit 62, first image storage control unit 64, interval imaging determination unit 66, fourth imaging control unit 68, image quality determination unit 70, second image storage control unit 72, and lost information storage control unit 74. Realized.
• the first imaging control unit 52 outputs a first imaging instruction signal to the image sensor 34, thereby controlling the image
• the sensor 34 is caused to image the first imaging target area 3A, which is the first imaging target area 3.
• the first imaging target area 3A is imaged under the control of the first imaging control unit 52, thereby obtaining first synthesis image data.
• the first compositing image data is image data indicating a first compositing image 92A, which is a compositing image 92 corresponding to the first imaging target area 3A.
• the first composition image data is stored in the storage 44.
• the first imaging target area 3A is an example of a "first imaging target area” according to the technology of the present disclosure.
• the first synthesis image 92A is an example of a "first image” according to the technology of the present disclosure.
• the second imaging control unit 54 causes the image sensor 34 to image the second imaging target area 3B by outputting a second imaging instruction signal to the image sensor 34 during the movement of the flight imaging device 1.
• image data for overlap determination is obtained.
• the overlap determination image data is image data indicating the overlap determination image 94.
• the overlap determination image 94 is, for example, a display image (for example, a live view image or a post-view image), and the overlap determination image data is displayed on a display device (not shown) provided in the imaging device 30 and/or Alternatively, it may be output to a display device 24 (see FIG. 1) provided in the transmitter 20.
• imaging refers to imaging for obtaining the composite image 92 unless there is an explanation that "the image was taken under the control of the second imaging control unit 54".
• the first overlap determination unit 56 determines the area of an overlap region where a part of the first synthesis image 92A and a part of the overlap determination image 94 overlap (hereinafter referred to as "first overlap amount"). ) is within a first predetermined range.
• the first predetermined range is set in consideration of the efficiency when sequentially imaging a plurality of imaging target regions 3 and the number of feature points required for compositing adjacent compositing images 92 (see FIG. 1).
• Ru for example, the upper limit value of the first predetermined range is set to a value of 50% or less of the area of the composite image 92, taking into consideration the efficiency when sequentially imaging a plurality of imaging target regions 3.
• the lower limit value of the first predetermined range is set to a value of 30% or more of the area of the compositing image 92, taking into consideration the number of feature points required for compositing adjacent compositing images 92.
• the moving speed of the aircraft 10 is determined by at least one determination by the first overlap determining unit 56 after the first overlap amount falls below the upper limit of the first predetermined range and before the first overlap amount falls below the lower limit of the first predetermined range.
• the times are set to the speed at which they are performed.
• the first overlap amount is an example of the "second overlap amount” according to the technology of the present disclosure.
• the first predetermined range is an example of the "second predetermined range” according to the technology of the present disclosure.
• the first synthesis image 92A is an example of a "fourth image” according to the technology of the present disclosure.
• the overlap determination image 94 is an example of a "fifth image” according to the technology of the present disclosure.
• FIG. 6 shows an example in which the first overlap amount exceeds the upper limit of the first predetermined range. If the first overlap amount exceeds the upper limit of the first predetermined range, the first overlap determination unit 56 determines that the first overlap amount is not within the first predetermined range.
• the lost determination unit 58 determines the amount of time that has passed since the first timing when the first imaging target area 3A was imaged. (hereinafter referred to as "elapsed time") is determined whether or not exceeds a first predetermined time. For example, when the flight imaging device 1 moves at a constant speed, the first predetermined time is the period from when the first imaging target area 3A is imaged until the first overlap amount reaches the lower limit of the first predetermined range. set to time.
• FIG. 6 shows an example in which the elapsed time does not exceed the first predetermined time.
• the lost determination unit 58 determines that the elapsed time has not exceeded the first predetermined time
• the second imaging control unit 54 outputs a second imaging instruction signal to the image sensor 34 so that the image sensor 34 to image the second imaging target area 3B.
• new image data for overlap determination is obtained.
• FIG. 7 shows an example in which the first overlap amount is within the first predetermined range.
• the third imaging control unit 60 executes overlap imaging processing when the first overlap determining unit 56 determines that the first overlap amount is within the first predetermined range. That is, the third imaging control unit 60 outputs a third imaging instruction signal to the image sensor 34, thereby causing the image sensor 34 to image the second imaging target region 3B.
• the second imaging target area 3B is imaged under the control of the third imaging control unit 60, thereby obtaining second synthesis image data.
• the second compositing image data is image data indicating a second compositing image 92B, which is the compositing image 92 corresponding to the second imaging target area 3B.
• the second imaging target area 3B is an example of a "second imaging target area” according to the technology of the present disclosure.
• the second synthesis image 92B is an example of a "second image” according to the technology of the present disclosure.
• the second overlap determination unit 62 determines the area of an overlap region (hereinafter referred to as It is determined whether or not the amount (referred to as "second overlap amount") is within a second predetermined range.
• the second predetermined range is set to, for example, the same upper and lower limit values as the first predetermined range. Note that the second predetermined range may be set to an upper limit value and a lower limit value that are different from the first predetermined range.
• the second overlap amount is an example of the "first overlap amount” according to the technology of the present disclosure.
• the second predetermined range is an example of the "first predetermined range” according to the technology of the present disclosure.
• the reason why the second overlap amount exceeds the upper limit of the second predetermined range after the determination by the first overlap determination section 56 is made is, for example, after the determination by the first overlap determination section 56 is performed.
• the second overlap amount increases as the direction of the flight imaging device 1 changes due to disturbances such as wind.
• the second overlap amount exceeds the upper limit of the second predetermined range.
• the number of pieces of composition image data stored in the storage 44 increases compared to the case where the second composition image data is stored in the storage 44 on the condition that the second composition image data is within the range. Therefore, in this embodiment, an upper limit value is set in the second predetermined range in order to suppress the number of composition image data stored in the storage 44.
• the second overlap amount falls below the lower limit of the second predetermined range, for example, the determination by the first overlap determination unit 56 is performed.
• the second overlap amount decreases as the direction of the flight imaging device 1 changes due to disturbances such as wind, or the third imaging control unit 60 issues a third imaging instruction signal to the image sensor 34.
• the second overlap amount decreases due to a delay between being output and being imaged by the image sensor 34.
• a lower limit is set in the second predetermined range in order to ensure the number of feature points required for composing adjacent compositing images 92.
• FIG. 8 shows an example in which the second overlap amount is within the second predetermined range.
• the first image storage control unit 64 outputs the second synthesis image data to the storage 44 when the second overlap determination unit 62 determines that the second overlap amount is within the second predetermined range. .
• the second composition image data is stored in the storage 44.
• the second imaging target area 3B imaged under the control of the third imaging control section 60 described above is The second image data for synthesis, which is treated as the first image capture target area 3A and obtained by capturing the second image capture target area 3B under the control of the third image capture controller 60, is used as the first image data for synthesis. be treated.
• the second imaging control unit 54 When the second image data for synthesis is stored in the storage 44 by the first image storage control unit 64, the second imaging control unit 54 outputs a second imaging instruction signal to the image sensor 34, so that the image sensor 34 to image the new second imaging target area 3B. As a result, new image data for overlap determination is obtained.
• FIG. 9 shows an example in which the elapsed time exceeds the first predetermined time.
• the first position indicates the position of the center of the flight imaging device 1 when the first imaging target area 3A is imaged.
• the second position is such that the first amount of overlap between a part of the first synthesis image 92A (see FIG. 7) and a part of the overlap determination image 94 (see FIG. 7) is within the first predetermined range.
• the position of the center of the flight imaging device 1 when the lower limit is reached is shown.
• the first position is an example of a "first position" according to the technology of the present disclosure.
• the second position is an example of a "second position" according to the technology of the present disclosure.
• the imaging device 30 has failed in the overlap imaging process. That is, this means that the second compositing image 92B corresponding to the second imaging target area 3B is lost as one of the images used in generating the composite image 90.
• the interval imaging determination unit 66 determines whether the elapsed time has reached the second predetermined time.
• the second predetermined time T2 is determined by the following equation (1), for example, when the flight imaging device 1 moves at a constant speed.
• T1 is the first predetermined time.
• the first predetermined time is set, for example, to the time from when the first imaging target area 3A is imaged until the first overlap amount reaches the lower limit of the first predetermined range.
• T3 is a third predetermined time.
• the third predetermined time is set, for example, to the same time as the time from when the first imaging target area 3A is imaged until the first overlap amount reaches the upper limit of the first predetermined range.
• FIG. 10 shows an example in which the second overlap amount is outside the second predetermined range (specifically, the second overlap amount is below the lower limit of the second predetermined range). . If the second overlap amount is less than the lower limit of the second predetermined range, the feature point included as an image in a part of the first synthesis image 92A and the feature point included as an image in a part of the second synthesis image 92B. There is a possibility that the first synthesis image 92A and the second synthesis image 92B cannot be synthesized based on the feature points. In this case, since it was not possible to obtain the second composite image 92B that could be combined with the first composite image 92A, the imaging device 30 failed in the overlap imaging process. That is, this means that the second compositing image 92B corresponding to the second imaging target area 3B is lost as one of the images used in generating the composite image 90.
• the interval imaging determination unit 66 determines whether the elapsed time has reached the second predetermined time. .
• examples of cases where the overlap imaging process fails include a case where the elapsed time exceeds the first predetermined time and a case where the second overlap amount is outside the second predetermined range.
• the imaging device 30 determines to take an image because a certain condition (for example, the condition that the imaging device 30 is out of focus in a situation where the autofocus mode is set as the operating mode) is not met.
• Other cases such as a case where the overlapping imaging process fails, or a case where the image data for synthesis is not stored normally in the storage 44 may also be included in the case where the overlap imaging process fails.
• FIG. 11 shows an example in which the elapsed time has reached the second predetermined time.
• the third position indicates the position of the center of the flight imaging device 1 when the elapsed time reaches the second predetermined time.
• the first predetermined movement distance is the distance from the first position to the third position.
• the first predetermined movement distance is twice the distance from the first position to the second position.
• the first predetermined moving distance is an example of a "first predetermined moving distance" according to the technology of the present disclosure.
• the third position is an example of a "third position" according to the technology of the present disclosure.
• the second position is an example of the "fourth position” according to the technology of the present disclosure.
• the fourth imaging control unit 68 executes interval imaging processing when the interval imaging determination unit 66 determines that the elapsed time has reached the second predetermined time. That is, the fourth imaging control unit 68 outputs a fourth imaging instruction signal to the image sensor 34, thereby causing the image sensor 34 to image the third imaging target region 3C.
• third synthesis image data is obtained.
• the third compositing image data is image data indicating a third compositing image 92C, which is a compositing image 92 corresponding to the third imaging target area 3C.
• the third imaging target area 3C is an example of a "third imaging target area" according to the technology of the present disclosure.
• the image quality determination unit 70 determines whether the third synthesis image 92C satisfies a predetermined image quality.
• the predetermined image quality is set based on, for example, the amount of blur, exposure, artifacts (eg, geometric, illuminance, or chromatic artifacts), and/or amount of blur.
• the fact that the third synthesis image 92C does not satisfy the predetermined image quality means that feature points necessary for generating the synthesis image 90 (i.e., feature points that are matched between images) are extracted from the third synthesis image 92C. means that it cannot be done. If the third composite image 92C does not meet the predetermined image quality, the imaging device 30 performs interval imaging processing because the third composite image 92C could not be obtained as one of the images necessary for generating the composite image 90. You've failed.
• the interval imaging determining unit 66 determines again whether the elapsed time has reached the second predetermined time.
• the second predetermined time T2 in the case where the inverter imaging process fails in addition to the overlap imaging process is determined by the following equation (2), for example, when the flight imaging device 1 moves at a constant speed.
• T1 is the first predetermined time.
• the first predetermined time is set, for example, to the time from when the first imaging target area 3A is imaged until the first overlap amount reaches the lower limit of the first predetermined range.
• N is a natural number indicating the number of times the overlap imaging process and the inverter imaging process have failed.
• T3 is a third predetermined time. As described above, the third predetermined time is set to, for example, the same time as the time from when the first imaging target area 3A is imaged until the first overlap amount reaches the upper limit of the first predetermined range.
• the second image storage control unit 72 outputs the third synthesis image data to the storage 44 when the image quality determination unit 70 determines that the third synthesis image 92C satisfies the predetermined image quality. As a result, the third synthesis image data is stored in the storage 44.
• the lost information storage control unit 74 stores position information regarding the position of the second imaging target area 3B (hereinafter referred to as "lost position") when the overlap imaging process or the interval imaging process fails, and the first synthesis image 92A. 1st image information regarding the third composition image 92C and second image information regarding the third composition image 92C are acquired.
• the positional information regarding the lost position is information indicating the imaging order of the second imaging target area 3B corresponding to the lost position, for example, counting from the first imaging target area 3 (see FIG. 5).
• the first image information regarding the first image for synthesis 92A is, for example, the first image for synthesis 92A corresponding to the first image capture area 3A that was imaged immediately before the second image capture area 3B corresponding to the lost position.
• the second image information regarding the third composite image 92C is, for example, identification information that allows identification of the third composite image 92C corresponding to the third imaging target area 3C captured by interval imaging processing.
• the lost information storage control unit 74 generates lost information in which the first image information and the second image information are associated with the position information, and stores the lost information in the storage 44.
• the position information may be associated with only one of the first image information and the second image information.
• the location information is an example of "location information" according to the technology of the present disclosure.
• the first image information is an example of "first image information” according to the technology of the present disclosure.
• the second image information is an example of "second image information” according to the technology of the present disclosure.
• the storage 44 is an example of a "memory” according to the technology of the present disclosure.
• the first synthesis image 92A is an example of a "seventh image” according to the technology of the present disclosure.
• the third synthesis image 92C is an example of the "eighth image” according to the technology of the present disclosure.
• the lost information is stored in the storage 44 by the lost information storage control unit 74
• the third imaging target area 3C imaged under the control of the fourth imaging control unit 68 described above is thereafter used as the first imaging target area.
• the third synthesis image data obtained by imaging the third imaging target area 3C under the control of the fourth imaging control unit 68 is handled as the first synthesis image data.
• FIG. 13 shows an example in which the overlap imaging process and the first interval imaging process fail consecutively, and the second interval imaging process succeeds.
• the overlap imaging process and the first interval imaging process fail consecutively, and then the interval imaging determination unit 66 determines that the elapsed time has reached the second predetermined time. Therefore, the second interval imaging process is executed by the fourth imaging control unit 68.
• the fourth position indicates the position of the center of the flight imaging device 1 when the elapsed time reaches the second predetermined time.
• the second predetermined movement distance is the distance from the first position to the fourth position.
• the second predetermined movement distance is three times the distance from the first position to the second position.
• the second predetermined travel distance is an example of the "first predetermined travel distance" according to the technology of the present disclosure.
• the fourth position is an example of the "third position” according to the technology of the present disclosure.
• the second position is an example of the "fourth position” according to the technology of the present disclosure.
• the second composite image data corresponding to the overlap imaging process and the third composite image data corresponding to the first interval imaging process are not obtained, and the first image capturing target First synthesis image data corresponding to the area 3A and third synthesis image data corresponding to the second interval imaging process are obtained.
• the first imaging target area 3A imaged immediately before the second imaging target area 3B corresponding to the overlap imaging process is referred to as the "first imaging target area” according to the technology of the present disclosure.
• the second imaging target area 3B corresponding to the overlap imaging process is considered as an example of the "second imaging target area” according to the technology of the present disclosure
• the third imaging target area corresponding to the first interval imaging process is It is possible to regard the region 3C as an example of a "third imaging target region" according to the technology of the present disclosure.
• the second imaging target area 3B corresponding to the overlap imaging process is considered as an example of the "first imaging target area” according to the technology of the present disclosure, and corresponds to the first interval imaging process.
• the third imaging target area 3C is considered as an example of the "second imaging target area” according to the technology of the present disclosure, and the third imaging target area 3C corresponding to the second interval imaging process is the "third imaging target area” according to the technology of the present disclosure. It is also possible to consider this as an example of "imaging target area”.
• the first interval imaging process is an example of "overlap imaging processing" according to the technology of the present disclosure.
• the reason why the first interval imaging process failed is that the third synthesis image 92C obtained by imaging the third imaging target area 3C corresponding to the first interval imaging process by the imaging device 30 is It is okay if the image quality does not meet the above requirements.
• the third synthesis image 92C is an example of a "third image" according to the technology of the present disclosure.
• the processor 42 executes the following re-imaging process in order to eliminate the above-mentioned inconvenience.
• a re-imaging program 100 is stored in the storage 44.
• the processor 42 reads the re-imaging program 100 from the storage 44 and executes the read re-imaging program 100 on the RAM 46.
• the processor 42 performs re-imaging processing according to the re-imaging program 100 executed on the RAM 46.
• the flight imaging device 1 starts moving in the horizontal direction from the same position as when it started the imaging process. Further, in the re-imaging process, the flight imaging device 1 moves at the same moving speed as the moving speed in the imaging process. The re-imaging process is started when the flight imaging device 1 starts moving in the horizontal direction.
• the processor 42 executes the first information acquisition section 102, the arrival determination section 104, the fifth imaging control section 106, the third overlap determination section 108, the sixth imaging control section 110, and the fourth imaging control section 108 according to the re-imaging program 100. This is realized by operating as an overlap determination section 112, a third image storage control section 114, a second information acquisition section 116, a fifth overlap determination section 118, and a notification control section 120.
• the first information acquisition unit 102 obtains position information regarding the position of the second imaging target area 3B corresponding to the lost position and the first synthesis image from the lost information stored in the storage 44.
• 92A is acquired. Based on the position information, for example, the order of imaging counted from the first imaging target area 3 (see FIG. 5) is specified for the second imaging target area 3B corresponding to the lost position. Further, based on the first image information regarding the first synthesis image 92A, for example, the first synthesis corresponding to the first imaging target area 3A imaged immediately before the second imaging target area 3B corresponding to the lost position is performed. image 92A (see FIG. 16) is specified.
• the arrival determination unit 104 detects a first imaging target area 3A imaged immediately before the second imaging target area 3B corresponding to the lost position (hereinafter also referred to as "first imaging target area 3A immediately before the lost position"). It is determined whether or not the flight imaging device 1 has arrived at the destination. If the time required for the flight imaging device 1 to reach the first imaging target area 3A immediately before the lost position after starting the re-imaging process is T4, the required time T4 is, for example, When the device 1 moves at a constant speed, it is determined by the following equation (3).
• T1 is the first predetermined time.
• the first predetermined time is set, for example, to the time from when the first imaging target area 3A is imaged until the first overlap amount reaches the lower limit of the first predetermined range.
• M is a natural number of 2 or more indicating the order of imaging counted from the first imaging target area 3 for the second imaging target area 3B corresponding to the lost position.
• the fifth imaging control unit 106 issues a fifth imaging instruction to the image sensor 34.
• the image sensor 34 is caused to image the second imaging target area 3B. As a result, image data for overlap determination is obtained.
• the third overlap determination unit 108 determines whether a part of the first synthesis image 92A specified by the first image information and a part of the overlap determination image 94 overlap. It is determined whether the first overlap amount is within a first predetermined range.
• the first predetermined range is the same as the first predetermined range in the imaging process.
• FIG. 16 shows an example in which the first overlap amount is within the first predetermined range.
• the sixth imaging control unit 110 executes overlap imaging processing when the third overlap determination unit 108 determines that the first overlap amount is within the first predetermined range.
• the overlap imaging process is similar to the overlap imaging process in the imaging process. That is, the sixth imaging control unit 110 outputs the sixth imaging instruction signal to the image sensor 34, thereby causing the image sensor 34 to image the second imaging target region 3B.
• second composition image data indicating the second composition image 92B corresponding to the lost position is obtained.
• the fourth overlap determination unit 112 determines that the second amount of overlap between a portion of the first compositing image 92A and a portion of the second compositing image 92B is a second amount of overlap. Determine whether it is within a predetermined range.
• the second overlap amount is the same as the second overlap amount in the imaging process
• the second predetermined range is the same as the second predetermined range in the imaging process.
• FIG. 17 shows an example in which the second overlap amount is within the second predetermined range.
• the third image storage control unit 114 outputs the second synthesis image data to the storage 44 when the fourth overlap determination unit 112 determines that the second overlap amount is within the second predetermined range. .
• the second composition image data is stored in the storage 44.
• FIG. 18 shows an example in which the second overlap amount is outside the second predetermined range (specifically, the second overlap amount is below the lower limit of the second predetermined range).
• the notification control unit 120 performs notification processing when the fourth overlap determination unit 112 determines that the second overlap amount is not within the second predetermined range.
• Examples of the notification process include a process of activating a notification device (not shown) provided in the imaging device 30 and/or the transmitter 20.
• Examples of the notification device include a speaker, a lighting device, a display device, and the like.
• the content of the notification by the notification device includes content that urges the user to perform the reimaging process again.
• the second information acquisition unit 116 acquires second image information regarding the third synthesis image 92C from the lost information stored in the storage 44. Based on the second image information regarding the third composite image 92C, for example, the third composite image 92C corresponding to the third imaging target area 3C captured by interval imaging processing is specified.
• the fifth overlap determining unit 118 determines the area of an overlap region (hereinafter referred to as "third overlap amount") where a part of the second image for synthesis 92B and a part of the third image for synthesis 92C overlap. ) is within a third predetermined range.
• the third overlap amount is the same as the second overlap amount in the imaging process
• the third predetermined range is the same as the second predetermined range in the imaging process.
• FIG. 18 shows an example in which the third overlap amount is within the third predetermined range. If the third overlap amount is within the third predetermined range, it means that the second imaging target area 3B between the first imaging target area 3A and the third imaging target area 3C has been completely imaged. Therefore, in this case, the re-imaging process ends.
• the fifth overlap determination unit 118 determines that the third overlap amount is not within the third predetermined range.
• a second imaging target area 3B that is not imaged exists between the first imaging target area 3A and the third imaging target area 3C. Therefore, in this case, the fifth imaging control section 106, the third overlap determination section 108, the sixth imaging control section 110, the fourth overlap determination section 112, the third image storage control section 114, the second information
• the processing by the acquisition unit 116 and the fifth overlap determination unit 118 is executed again.
• the fifth overlap determination unit 118 determines that the third overlap amount is not within the third predetermined range, then the second imaging target imaged under the control of the sixth imaging control unit 110 described above
• the region 3B is treated as the first imaging target region 3A, and the image data for second synthesis obtained by imaging the second imaging target region 3B under the control of the sixth imaging control unit 110 is used as the first imaging target region 3A. It is treated as image data.
• FIG. 20 shows an example of the flow of the imaging process according to the present embodiment
• FIG. 21 shows an example of the flow of the re-imaging process according to the present embodiment.
• step ST10 the first imaging control unit 52 causes the image sensor 34 to image the first imaging target area 3A, which is the first imaging target area 3 (see FIG. 5). As a result, first composition image data indicating the first composition image 92A is obtained. After the process of step ST10 is executed, the imaging process moves to step ST12.
• step ST12 the second imaging control unit 54 causes the image sensor 34 to image the second imaging target area 3B (see FIG. 5). As a result, overlap determination image data indicating the overlap determination image 94 is obtained. After the process of step ST12 is executed, the imaging process moves to step ST14.
• step ST14 the first overlap determination unit 56 determines that a part of the first synthesis image 92A obtained in step ST10 overlaps a part of the overlap determination image 94 obtained in step ST12. It is determined whether the first overlap amount is within a first predetermined range (see FIG. 5). In step ST14, if the first overlap amount is outside the first predetermined range, the determination is negative and the imaging process moves to step ST16. In step ST14, if the first overlap amount is within the first predetermined range, the determination is affirmative and the imaging process moves to step ST18.
• step ST16 the lost determination unit 58 determines whether the elapsed time that has passed since the first timing when the first imaging target area 3A was imaged in step ST10 has exceeded the first predetermined time (see FIG. 6). In step ST16, if the elapsed time has not exceeded the first predetermined time, the determination is negative and the imaging process moves to step ST12. In step ST16, if the elapsed time exceeds the first predetermined time (that is, if the overlap imaging process for the second imaging target area 3B fails), the determination is affirmative and the imaging process proceeds to step ST24. Transition.
• step ST18 the third imaging control unit 60 causes the image sensor 34 to image the second imaging target area 3B (see FIG. 7). As a result, overlap imaging processing is executed, and second synthesis image data indicating the second synthesis image 92B is obtained. After the process of step ST18 is executed, the imaging process moves to step ST20.
• step ST20 the second overlap determination unit 62 determines whether a part of the first synthesis image 92A obtained in step ST10 overlaps a part of the second synthesis image 92B obtained in step ST18. It is determined whether the second overlap amount is within a second predetermined range (see FIG. 8). In step ST20, if the second overlap amount is within the second predetermined range, the determination is affirmative and the imaging process moves to step ST22. In step ST20, if the second overlap amount is outside the second predetermined range (that is, if the overlap imaging process for the second imaging target area 3B fails), the determination is negative and the imaging process is performed as follows. The process moves to step ST24.
• step ST22 the first image storage control unit 64 stores the second synthesis image data obtained in step ST18 in the storage 44 (see FIG. 8). After the process of step ST22 is executed, the imaging process moves to step ST12.
• the second synthesis image data is stored in the storage 44 in step ST22
• the second imaging target area 3B imaged in step ST18 is thereafter treated as the first imaging target area 3A
• the second imaging target area 3B imaged in step ST18 is treated as the first imaging target area 3A.
• the second image data for synthesis is treated as the first image data for synthesis.
• step ST24 the interval imaging determination unit 66 determines whether the elapsed time has reached the second predetermined time (see FIGS. 9 and 10). In step ST24, if the elapsed time has not reached the second predetermined time, the determination is negative and the process of step ST24 is executed again. In step ST24, if the elapsed time has reached the second predetermined time, the determination is affirmative and the imaging process moves to step ST26.
• step ST26 the fourth imaging control unit 68 causes the image sensor 34 to image the third imaging target area 3C (see FIG. 11). As a result, interval imaging processing is executed, and third synthesis image data indicating the third synthesis image 92C is obtained. After the process of step ST26 is executed, the imaging process moves to step ST28.
• step ST28 the image quality determination unit 70 determines whether the third synthesis image 92C obtained in step ST26 satisfies the predetermined image quality (see FIG. 12). In step ST26, if the third synthesis image 92C satisfies the predetermined image quality, the determination is affirmative and the imaging process moves to step ST30. In step ST28, if the third synthesis image 92C does not satisfy the predetermined image quality (that is, if the interval imaging process for the third imaging target area 3C has failed), the determination is negative and the imaging process continues in step ST28. Move to ST24.
• step ST30 the second image storage control unit 72 stores the third synthesis image data obtained in step ST26 in the storage 44 (see FIG. 12). After the process of step ST30 is executed, the imaging process moves to step ST32.
• the lost information storage control unit 74 stores the position information regarding the position of the second imaging target area 3B corresponding to the lost position when the overlap imaging process or the interval imaging process fails, and the position information obtained in step ST10 or step ST18.
• the first image information regarding the first composite image 92A obtained in step ST26 and the second image information regarding the third composite image 92C obtained in step ST26 are acquired (see FIG. 12).
• lost information is generated by associating the first image information and the second image information with the position information, and the lost information is stored in the storage 44 (see FIG. 12).
• the third imaging target area 3C imaged in step ST26 is treated as the first imaging target area 3A, and the third combining target area 3C obtained in step ST26 is treated as the first imaging target area 3A.
• the image data is treated as first compositing image data.
• step ST34 the processor 42 determines whether a condition for terminating the imaging process (termination condition) is satisfied.
• the termination condition include a condition that the user has given an instruction to the imaging device 30 to terminate the imaging process, or a condition that the number of composite images 92 has reached the number specified by the user.
• the end condition is not satisfied, the determination is negative and the imaging process moves to step ST12.
• the termination condition is satisfied, the determination is affirmative and the imaging process is terminated.
• step ST40 the first information acquisition unit 102 obtains position information regarding the position of the second imaging target area 3B corresponding to the lost position from the lost information stored in the storage 44. , and the first image information regarding the first synthesis image 92A corresponding to the first imaging target area 3A immediately before the lost position (see FIG. 15).
• step ST42 the re-imaging process moves to step ST42.
• step ST42 the arrival determination unit 104 determines whether the flight imaging device 1 has reached the first imaging target area 3A immediately before the lost position (see FIG. 15). In step ST42, if the flight imaging device 1 has not reached the first imaging target area 3A immediately before the lost position, the process of step ST42 is executed again. In step ST42, when the flight imaging device 1 reaches the first imaging target area 3A immediately before the lost position, the re-imaging process moves to step ST44.
• step ST44 the fifth imaging control unit 106 causes the image sensor 34 to image the second imaging target area 3B. As a result, overlap determination image data indicating the overlap determination image 94 is obtained. After the process of step ST44 is executed, the re-imaging process moves to step ST46.
• step ST46 the third overlap determination unit 108 selects a portion of the first synthesis image 92A specified by the first image information obtained in step ST40 and the overlap determination image 94 obtained in step ST44. It is determined whether or not the first amount of overlap that overlaps with a part of is within the first predetermined range (see FIG. 16). In step ST46, if the first overlap amount is outside the first predetermined range, the determination is negative and the re-imaging process moves to step ST44. In step ST46, if the first overlap amount is within the first predetermined range, the determination is affirmative, and the re-imaging process moves to step ST48.
• step ST48 the sixth imaging control unit 110 causes the image sensor 34 to image the second imaging target area 3B (see FIG. 16). As a result, overlap imaging processing is executed, and second synthesis image data indicating the second synthesis image 92B is obtained. After the process of step ST48 is executed, the re-imaging process moves to step ST50.
• step ST50 the fourth overlap determination unit 112 selects a part of the first synthesis image 92A specified by the first image information obtained in step ST40 and the second synthesis image 92B obtained in step ST48. It is determined whether or not the second amount of overlap that overlaps with a part of is within the second predetermined range (see FIG. 17). In step ST50, if the second overlap amount is within the second predetermined range, the determination is affirmative, and the re-imaging process moves to step ST52. In step ST50, if the second overlap amount is outside the second predetermined range (that is, if the overlap imaging process for the second imaging target area 3B has failed), the determination is negative and the re-imaging process is not performed. , the process moves to step ST58.
• step ST52 the third image storage control unit 114 stores the second synthesis image data obtained in step ST48 in the storage 44 (see FIG. 17). After the process of step ST52 is executed, the re-imaging process moves to step ST54.
• step ST54 the second information acquisition unit 116 acquires second image information regarding the third synthesis image 92C from the lost information stored in the storage 44 (see FIG. 19). After the process of step ST54 is executed, the re-imaging process moves to step ST56.
• step ST56 the fifth overlap determination unit 118 selects a part of the second synthesis image 92B obtained in step ST48 and a third synthesis image 92C specified by the second image information obtained in step ST54. It is determined whether or not the third overlap amount that overlaps with a portion of is within the third predetermined range (see FIG. 19). In step ST56, if the third overlap amount is outside the third predetermined range, the determination is negative and the re-imaging process moves to step ST44.
• step ST56 If the determination in step ST56 is negative and the re-imaging process moves to step ST44, the second imaging target area 3B imaged in step ST48 will be treated as the first imaging target area 3A, and the second imaging target area 3B imaged in step ST48 will be treated as the first imaging target area 3A.
• the second composite image data thus obtained is treated as first composite image data.
• step ST50 if the third overlap amount is within the third predetermined range, the determination is affirmative and the re-imaging process ends.
• step ST58 the notification control section 120 performs notification processing (see FIG. 18). After the process of step ST58 is executed, the re-imaging process ends.
• imaging control method described as the function of the flight imaging device 1 described above is an example of the "imaging control method" according to the technology of the present disclosure.
• the processor 42 causes the imaging device 30 to image the first imaging target area 3A, and in the process of the flight imaging device 1 moving, the processor 42 When a part of the first imaging target area 3A overlaps with a part of the first imaging target area 3A, overlap imaging processing is performed to cause the imaging device 30 to image the second imaging target area 3B (see FIGS. 5 to 8). . If the overlap imaging process fails, the processor 42 determines that the distance traveled by the flight imaging device 1 from the first position where the first imaging target area 3A is imaged by the imaging device 30 reaches a first predetermined travel distance. On the condition that this is done, interval imaging processing is performed to cause the imaging device 30 to image the third imaging target area 3C (see FIGS. 9 to 11). Therefore, even if the overlap imaging process fails, the third imaging target area 3C can be imaged. That is, even if overlap imaging processing fails, imaging processing can be continued.
• a case where the overlap imaging process fails is a case where the second imaging target area 3B is not imaged by the imaging device 30 (that is, before the second imaging target area 3B is imaged), and the second imaging target area 3B is not imaged from the first position.
• a case where the overlap imaging process fails is a case where the second imaging target area 3B is imaged by the imaging device 30, and the first synthesis image obtained by imaging the first imaging target area 3A is 92A and a portion of the second synthesis image 92B obtained by capturing the second imaging target area 3B, the second overlap amount is outside the second predetermined range. (See Figure 10). Therefore, even if the overlap imaging process fails because the second overlap amount is outside the second predetermined range, the third imaging target region 3C can be imaged.
• a part of the third imaging target area 3C overlaps with a part of the second imaging target area 3B (see FIG. 11). Therefore, for example, when the second synthesis image 92B is obtained by capturing the second imaging target area 3B in the re-imaging process (see FIG. 17), the third image capturing area 3B corresponding to the third imaging target area 3C is A part of the image for synthesis 92C and a part of the second image for synthesis 92B corresponding to the second imaging target area 3B can be overlapped.
• the first predetermined moving distance is the distance from the first position to the third position where a part of the third imaging target area 3C overlaps a part of the second imaging target area 3B (see FIG. 11). Therefore, when interval imaging processing is performed on the condition that the distance traveled by the flight imaging device 1 from the first position reaches the first predetermined distance, a part of the third imaging target area 3C is The third imaging target area 3C can be imaged so as to overlap a part of the second imaging target area 3B.
• the first predetermined moving distance is a distance that is a natural number times 2 or more of the distance from the first position to the second position where the second imaging target area 3B is imaged.
• the natural number greater than or equal to 2 is 2 (see FIG. 11)
• the imaging target area 3 is imaged at the second position and at the third position (that is, a position separated from the second position by the distance from the first position to the second position). Accordingly, it is possible to obtain a plurality of images for synthesis 92 that are continuous in the moving direction of the flight imaging device 1.
• the overlap imaging process includes a portion of the first synthesis image 92A obtained by imaging the first imaging target region 3A and an overlap determination image obtained by imaging the second imaging target region 3B. This is performed on the condition that the first amount of overlap with a part of the image 94 is within the first predetermined range (see FIG. 7). Therefore, the second amount of overlap in which a portion of the second composite image 92B and a portion of the first composite image 92A obtained by the overlap imaging process overlap can be kept within the second predetermined range. .
• the fact that the moving distance has reached the first predetermined moving distance means that when the flying imaging device 1 moves at a constant speed, the time that has passed since the first timing when the first imaging target area 3A was imaged by the imaging device 30 It is determined on the condition that the first predetermined time has been reached (see FIG. 11). Therefore, interval imaging processing can be performed based on the time that has passed since the first timing.
• the processor 42 When the overlap imaging process fails, the processor 42 generates position information regarding the position of the second imaging target area 3B, first image information regarding the first composite image 92A, and second image regarding the third composite image 92C. information (see FIG. 12). Position information regarding the position of the second imaging target area 3B is stored in the storage 44 in association with the first image information and the second image information. Therefore, by executing the re-imaging process based on the position information, the first image information, and the second image information, it is possible to obtain a plurality of images for synthesis 92 that are continuous in the moving direction of the flight imaging device 1.
• the lost determination unit 58 determines whether the elapsed time from the first timing when the first imaging target area 3A was imaged exceeds the first predetermined time (see FIG. 9). However, as shown in FIG. 22 as an example, the lost determination unit 58 determines whether the distance traveled by the flight imaging device 1 from the first position corresponding to the first timing exceeds the third predetermined distance. You may judge.
• the moving distance is derived based on the moving speed and elapsed time of the flight imaging device 1.
• the moving speed of the flight imaging device 1 is derived, for example, based on the acceleration indicated by the acceleration data input to the processor 42 from the acceleration sensor 80 mounted on the imaging device 30 (that is, the acceleration measured by the acceleration sensor 80). be done.
• Acceleration sensor 80 is an example of an "acceleration sensor" according to the technology of the present disclosure.
• the third predetermined movement distance is the distance from the first position to the second position.
• the second position is determined when the amount of overlap between a part of the first synthesis image 92A and a part of the overlap determination image 94 reaches the lower limit of the first predetermined range (see FIG. 7) shows the center position of the flight imaging device 1. If the travel time exceeds the third predetermined travel distance, the imaging device 30 has failed in the overlap imaging process because the opportunity to image the second imaging target area 3B has been lost.
• the determination accuracy can be improved compared to the case where it is determined whether or not the overlap imaging process has failed without taking into account changes in the acceleration of the flight imaging device 1.
• the interval imaging determination unit 66 determines whether the elapsed time that has elapsed from the first timing when the first imaging target area 3A was imaged has reached the second predetermined time (see FIG. 11). ). However, as an example shown in FIG. 23, the interval imaging determination unit 66 determines whether the distance traveled by the flight imaging device 1 from the first position corresponding to the first timing has reached the first predetermined travel distance. It may be determined whether
• the first predetermined movement distance is the distance from the first position to the third position.
• the third position is a part of the third composite image 92C corresponding to the third imaging target region 3C, assuming that the second composite image 92B is obtained by capturing the second imaging target region 3B. This shows the position of the center of the flight imaging device 1 when the second overlap amount in which the second composite image 92B partially overlaps reaches the upper limit of the second predetermined range. If the interval imaging determination unit 66 determines that the movement distance exceeds the first predetermined movement distance, interval imaging processing is executed.
• the determination accuracy can be improved compared to the case where it is determined whether or not to perform interval imaging processing without considering changes in the acceleration of the flight imaging device 1.
• the acceleration sensor 80 is mounted on the imaging device 30, but it may also be mounted on the aircraft 10. Further, the acceleration sensors 80 may be mounted on the aircraft 10 and the imaging device 30, respectively, and the moving speed may be derived based on the average value of the accelerations measured by each acceleration sensor 80.
• the moving speed of the flight imaging device 1 is derived based on the acceleration measured by the acceleration sensor 80.
• the processor 42 operates as a moving speed deriving section 76, and the moving speed deriving section 76 performs flight flight based on the first synthesis image 92A and the overlap determination image 94.
• the moving speed of the imaging device 1 may be derived.
• the moving speed of the flight imaging device 1 is derived in the following manner. That is, first, the feature points included in common in the first synthesis image 92A and the overlap determination image 94 are moved from the position included in the first synthesis image 92A to the position included in the overlap determination image 94.
• the moving distance (hereinafter referred to as "inter-image moving distance") is derived.
• the first synthesis image 92A and the overlap determination image 94 are an example of a "sixth image" according to the technology of the present disclosure.
• the first imaging target area 3A corresponding to the first synthesis image 92A is determined.
• a movement distance (hereinafter referred to as "inter-area movement distance") that the second imaging target area 3B corresponding to the overlap determination image 94 has moved relative to the overlap determination image 94 is derived.
• a time interval (hereinafter referred to as "inter-image time interval") when the first synthesis image 92A and the overlap determination image 94 are obtained is derived.
• the moving speed of the flight imaging device 1 is derived based on the inter-region moving distance and the inter-image time interval.
• the lost determination unit 58 acquires the movement speed derived by the movement speed derivation unit 76, and derives the movement distance of the flight imaging device 1 based on the movement speed and the elapsed time. You may. Then, the lost determining unit 58 may determine whether the moving distance exceeds a third predetermined moving distance.
• the overlap imaging process is performed in consideration of changes in the acceleration of the flight imaging device 1. It is determined whether or not the process has failed. Therefore, for example, the determination accuracy can be improved compared to the case where it is determined whether or not the overlap imaging process has failed without taking into account changes in the acceleration of the flight imaging device 1.
• the interval imaging determination unit 66 acquires the movement speed derived by the movement speed derivation unit 76, and calculates the movement distance of the flight imaging device 1 based on the movement speed and the elapsed time. May be derived. Then, the interval imaging determination unit 66 may determine whether the moving distance has reached the first predetermined moving distance.
• the interval imaging process is performed in consideration of changes in the acceleration of the flight imaging device 1. It is determined whether or not to execute. Therefore, for example, the determination accuracy can be improved compared to the case where it is determined whether or not to perform interval imaging processing without considering changes in the acceleration of the flight imaging device 1.
• the processor 42 may output moving speed data indicating the moving speed to the transmitter 20. Further, the transmitter 20 may display the moving speed indicated by the moving speed data input from the imaging device 30 on the display device 24.
• the moving speed data is an example of "moving speed data" according to the technology of the present disclosure.
• the first synthesis image 92A and the overlap determination image 94 are an example of a "ninth image” according to the technology of the present disclosure.
• the flight imaging device 1 images a plurality of imaging target areas 3 while moving in a zigzag pattern by repeating horizontal movement and vertical movement alternately is used. It is assumed. However, the flight imaging device 1 may image the wall surface 2A of the object 2 while moving in any direction along the wall surface 2A.
• the imaging device 30 is mounted on the flying object 10, but the imaging device 30 may be mounted on various moving objects (for example, a gondola, an automatic transport robot, an automatic guided vehicle, or a high-speed vehicle). It may also be mounted on a vehicle for inspection.
• various moving objects for example, a gondola, an automatic transport robot, an automatic guided vehicle, or a high-speed vehicle. It may also be mounted on a vehicle for inspection.
• the synthesis image 92 is stored in the storage 44, but it may be stored in a storage medium other than the storage 44.
• the lost information is stored in the storage 44, but it may be stored in a storage medium other than the storage 44.
• the storage medium may be provided in a device other than the flight imaging device 1 (for example, a server and/or a personal computer, etc.).
• Storage media include computer-readable non-transitory storage media such as USB memory, SSD, HDD, optical disk, and magnetic tape.
• the processor 42 is illustrated, but it is also possible to use at least one other CPU, at least one GPU, and/or at least one TPU in place of the processor 42 or in addition to the processor 42. You can also do this.
• the imaging program 50 and the re-imaging program 100 are stored in the storage 44, but the technology of the present disclosure is not limited to this.
• the imaging program 50 and/or the re-imaging program 100 may be stored in a storage medium other than the storage 44.
• the imaging program 50 and/or the re-imaging program 100 stored in the storage medium may be installed in the computer 32 of the imaging device 30.
• the imaging program 50 and/or the re-imaging program 100 may be stored in a storage device such as another computer or a server device connected to the imaging device 30 via a network, and the imaging program 50 and/or the re-imaging program 100 may be programmed according to a request from the imaging device 30. 50 and/or re-imaging program 100 may be downloaded and installed on computer 32.
• imaging program 50 and/or re-imaging program 100 it is not necessary to store all of the imaging program 50 and/or re-imaging program 100 in a storage device such as another computer or server device connected to the imaging device 30, or in the storage 44; Alternatively, a part of the re-imaging program 100 may be stored.
• the computer 32 is built into the imaging device 30, the technology of the present disclosure is not limited to this, and for example, the computer 32 may be provided outside the imaging device 30.
• the computer 32 including the processor 42, the storage 44, and the RAM 46 is illustrated, but the technology of the present disclosure is not limited to this, and instead of the computer 32, an ASIC, an FPGA, and/or Alternatively, a device including a PLD may be applied. Further, instead of the computer 32, a combination of hardware configuration and software configuration may be used.
• processors can be used as hardware resources for executing the various processes described in each of the above embodiments.
• the processor include a CPU, which is a general-purpose processor that functions as a hardware resource that executes various processes by executing software, that is, a program.
• the processor include a dedicated electronic circuit such as an FPGA, a PLD, or an ASIC, which is a processor having a circuit configuration specifically designed to execute a specific process.
• Each processor has a built-in memory or is connected to it, and each processor uses the memory to perform various processes.
• Hardware resources that execute various processes may be configured with one of these various processors, or a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or a CPU and FPGA). Furthermore, the hardware resource that executes various processes may be one processor.
• one processor is configured by a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes various processes.
• a and/or B has the same meaning as “at least one of A and B.” That is, “A and/or B” means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, even when three or more items are expressed by connecting them with “and/or”, the same concept as “A and/or B" is applied.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Theoretical Computer Science (AREA)
• Studio Devices (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=88242959

Family Applications (1)

Country Status (3)

Cited By (1)

Citations (3)

• 2023
  • 2023-03-29 WO PCT/JP2023/012977 patent/WO2023195401A1/ja not_active Ceased
  • 2023-03-29 JP JP2024514246A patent/JPWO2023195401A1/ja active Pending
• 2024
  • 2024-09-30 US US18/902,823 patent/US20250024132A1/en active Pending

Patent Citations (3)

Also Published As

Similar Documents

Legal Events