WO2021176877A1 - 画像処理装置、画像処理方法、及び画像処理プログラム - Google Patents

画像処理装置、画像処理方法、及び画像処理プログラム Download PDF

Info

Publication number
WO2021176877A1
WO2021176877A1 PCT/JP2021/002200 JP2021002200W WO2021176877A1 WO 2021176877 A1 WO2021176877 A1 WO 2021176877A1 JP 2021002200 W JP2021002200 W JP 2021002200W WO 2021176877 A1 WO2021176877 A1 WO 2021176877A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
image processing
images
processor
dimensional model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/002200
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
與那覇　誠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Corp
Original Assignee
Fujifilm Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujifilm Corp filed Critical Fujifilm Corp
Priority to CN202180013765.7A priority Critical patent/CN115104128B/zh
Priority to JP2022505021A priority patent/JP7529761B2/ja
Priority to EP21765299.9A priority patent/EP4116931A4/en
Publication of WO2021176877A1 publication Critical patent/WO2021176877A1/ja
Priority to US17/819,388 priority patent/US20220383487A1/en
Anticipated expiration legal-status Critical
Priority to JP2024119366A priority patent/JP7743579B2/ja
Priority to JP2025149135A priority patent/JP2025175087A/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0635—Risk analysis of enterprise or organisation activities
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G06Q10/06395—Quality analysis or management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management
    • G06Q10/103—Workflow collaboration or project management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/20—Administration of product repair or maintenance
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/08—Construction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G06Q50/16—Real estate
    • G06Q50/163—Real estate management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G06Q50/26—Government or public services
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G06T11/60—Creating or editing images; Combining images with text
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G06T17/10—Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection

Definitions

  • the present invention relates to an image of a building, a three-dimensional model of the building, and a technique for handling damage information.
  • Patent Document 1 describes a device for creating a management drawing of a structure (building, building) such as a bridge.
  • One embodiment according to the technique of the present disclosure provides an image of a building, a three-dimensional model of the building, an image processing device capable of easily handling damage information, an image processing method, and an image processing program. do.
  • the image processing apparatus is a memory in which a processor, a plurality of images of a building, and a three-dimensional model of a structure in which a member constituting the structure is specified are stored. It is an image processing device including a memory in which a plurality of images and members are stored in association with each other, and the processor is an extraction process for extracting damage information of a building based on a plurality of images and a plurality of extraction processes. Output that selects the image corresponding to the specified member according to the specified criteria from the images of, and outputs the specified member, the selected image, and the damage information in association with each other. Process and perform.
  • the image processing apparatus constitutes a building in a generation process for generating a three-dimensional model of the building based on a plurality of images and a generated three-dimensional model.
  • a specific process for specifying a member and a storage control process for storing a three-dimensional model in a memory in association with a plurality of images and the specified member are performed.
  • the image processing apparatus performs the specifying process without the operation of the user specifying the member.
  • the image processing apparatus is one of the first to third aspects, and the processor performs a reception process for accepting the designation of the reference.
  • the image processing apparatus is in any one of the first to fourth aspects, and the processor is at least one of the type, number, magnitude, degree of damage, and time variation of the damage in the extraction process. Extract one as damage information.
  • the image processing apparatus is one of the first to fifth aspects, and the processor selects an image for each type of damage in the selection process.
  • the image processing apparatus is in any one of the first to sixth aspects, and the processor selects a specified number of images in the selection process.
  • the image processing apparatus arranges the selected image in the area designated as the image area in the document file of the specified format. Perform image placement processing.
  • the processor performs information input processing for inputting damage information into an area designated as an information area in a document file.
  • the image processing apparatus associates the three-dimensional model with the position information indicating the position of the selected image in the three-dimensional model.
  • the first display process for displaying on the display device and the second display process for displaying the selected image on the display device with respect to the designated position information among the displayed position information are performed.
  • the image processing apparatus associates the three-dimensional model with the position information indicating the position of the selected image in the three-dimensional model.
  • the first display process for displaying on the display device and the third display process for displaying the selected image on the display device with respect to the displayed position information are performed.
  • the image processing apparatus identifies and displays the position information in the tenth or eleventh aspect, and at least in the first display process, the position information is identified and displayed in the manner corresponding to the damage information.
  • the image processing apparatus is in any one of the tenth to twelfth aspects, and the processor synthesizes an image corresponding to a specified member among a plurality of images in at least the first display processing. Then, the synthesized image is mapped to the specified member and displayed on the display device.
  • the image processing apparatus is in any one of the tenth to thirteenth aspects, and the processor highlights the damage information on the three-dimensional model at least in the first display process.
  • the image processing apparatus is one of the first to the fourteenth aspects, wherein the processor is a plurality of images of a building, and the plurality of images stored in the memory and the shooting date and time.
  • the acquisition process of acquiring a plurality of images having different values, and the association process of associating the acquired plurality of images with the members of the three-dimensional model stored in the memory are performed.
  • the processor performs associative processing based on the correlation between the acquired plurality of images and the plurality of images stored in the memory.
  • the image processing method is a memory in which a processor, a plurality of images of a building, and a three-dimensional model of the building are stored, and the plurality of images and 3
  • the image processing method according to the seventeenth aspect may further have the same configuration as the second to sixteenth aspects.
  • the image processing program according to the eighteenth aspect of the present invention causes a computer to execute the image processing method according to the seventeenth aspect.
  • a non-temporary recording medium on which a computer-readable code of the image processing program according to the eighteenth aspect is recorded can also be mentioned as an aspect of the present invention.
  • FIG. 1 is a diagram showing a schematic configuration of an image processing system according to the first embodiment.
  • FIG. 2 is a diagram showing a functional configuration of the processing unit.
  • FIG. 3 is a diagram showing information stored in the storage device.
  • FIG. 4 is a flowchart showing the processing of the image processing method.
  • FIG. 5 is a diagram showing a state of acquiring an image group.
  • FIG. 6 is a diagram showing an example of three-dimensional point cloud data.
  • FIG. 7 is a diagram showing an example of a three-dimensional model.
  • FIG. 8 is a diagram showing an example of damage information.
  • FIG. 9 is a diagram showing how a selection criterion for a representative image is set.
  • FIG. 10 is a diagram showing how the output mode of the processing result is set.
  • FIG. 11 is a diagram showing a state in which a representative image and damage information are input to the two-dimensional inspection record.
  • FIG. 12 is a diagram showing a state of displaying position information on a three-dimensional model.
  • FIG. 13 is a diagram showing a state in which a representative image at a designated position is displayed.
  • FIG. 14 is a diagram showing how the representative image is displayed from the beginning.
  • FIG. 15 is a diagram showing a state in which a portion of the two-dimensional inspection record including a representative image at a designated position is displayed.
  • FIG. 16 is a diagram showing a state in which a composite image is mapped to a three-dimensional model.
  • FIG. 1 is a block diagram showing a schematic configuration of an image processing system 1 (image processing apparatus).
  • the image processing system 1 includes an image processing device 10 (image processing device) and a display device 20 (display device, monitor), extracts damage information from a plurality of images acquired by dividing and photographing a subject, and creates a three-dimensional model. It is a system that supports the creation of two-dimensional inspection records.
  • the image processing system 1 can be configured by using a device (information terminal) such as a personal computer, a tablet terminal, or a smartphone.
  • a device information terminal
  • Each element of the image processing system 1 may be housed in one housing or may be housed in an independent housing. Further, each element may be arranged at a distant place and connected via a network.
  • the image processing device 10 includes a processing unit 100, a storage device 200, and an operation unit 300, and these units are connected to each other to transmit and receive necessary information.
  • FIG. 2 is a diagram showing a configuration of a processing unit 100 (processor).
  • the processing unit 100 includes an input processing unit 102, an acquisition processing unit 103, an extraction processing unit 104, a generation processing unit 105, a specific processing unit 106, an association processing unit 107, a selection processing unit 108, a storage control processing unit 109, and a reception processing unit 109. It includes 110, an image arrangement processing unit 112, an information input unit 114, a display processing unit 116, and a communication control unit 118, and supports acquisition of captured images, creation of a three-dimensional model, creation of a two-dimensional inspection record, and the like. Details of the processing by each of these parts will be described later.
  • the function of the processing unit 100 described above can be realized by using various processors and recording media.
  • Various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions, and a GPU (Graphics Processing Unit), which is a processor specialized in image processing.
  • a programmable logic device (PLD) which is a processor whose circuit configuration can be changed after manufacturing such as an FPGA (Field Programmable Gate Array), is also included.
  • Each function may be realized by one processor, or may be realized by a plurality of processors of the same type or different types (for example, a plurality of FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Further, a plurality of functions may be realized by one processor. More specifically, the hardware structure of these various processors is an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.
  • a code readable by a computer of the software to be executed for example, various processors and electric circuits constituting the processing unit 100, and / or a combination thereof.
  • a non-temporary recording medium such as a ROM
  • the computer refers to the software.
  • the information stored in the storage device is used as needed.
  • RAM Random Access Memory
  • the processing unit 100 may be realized by a server on the network, and the image processing device 10 may input data, control communication, display the result, and the like.
  • the Application Service Provider type system is constructed including the server on the network.
  • the storage device 200 (storage device, memory) is composed of a non-temporary recording medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), a hard disk (Hard Disk), various semiconductor memories, and a control unit thereof.
  • the information shown is stored in association with each other.
  • the photographed image 202 is a plurality of images of a building, and the composite image 204 is (a set of) images corresponding to a specific member, which are synthesized from the photographed images.
  • the three-dimensional model data 206 (three-dimensional model) is a three-dimensional model of a building created based on a photographed image, and members constituting the building are specified.
  • the three-dimensional model data 206 is associated with a photographed image, a representative image, a two-dimensional record, and the like, and as will be described in detail later, the user can specify the position information on the three-dimensional model to represent the three-dimensional model data 206.
  • Images and 2D inspection records can be displayed.
  • Damage information 208 (damage information) is information indicating damage to a building extracted from a photographed image.
  • the inspection record data 210 is a two-dimensional inspection record template (document file in a designated format), or data in which representative images and damage information are arranged and input in the template (described later).
  • the template may be in a format specified by the Ministry of Land, Infrastructure, Transport and Tourism or the local government.
  • the operation unit 300 includes a keyboard 310 and a mouse 320, and the user can perform operations necessary for image processing according to the present invention by using these devices.
  • the display device 20 may be used as an operation unit.
  • the display device 20 is, for example, a device such as a liquid crystal display, and can display information such as acquired captured images, damage information, a three-dimensional model, a two-dimensional inspection record, and a representative image.
  • FIG. 4 is a flowchart showing the procedure of the image processing method according to the present invention.
  • the input processing unit 102 inputs a plurality of images of a building as a subject (step S100: input processing, input step). Buildings (buildings, structures) are, for example, bridges, roads, etc., but other structures may also be used. Further, the input processing unit 102 may input an image stored in the storage device 200 as a captured image 202, or may input an image via a recording medium or a network (not shown). These images can be taken while moving the viewpoint by a flying object such as a drone or a robot having a moving function (the user may take a picture). The image to be captured does not have to be a stereo image.
  • FIG. 5 is a diagram showing how the camera 30 captures such overlapping images.
  • the extraction processing unit 104 extracts damage information of the building based on the plurality of input images (step S110: extraction processing, extraction step).
  • the extraction processing unit 104 can extract at least one of the type, number, size, degree of damage, and time change of damage as damage information.
  • the extraction processing unit 104 can extract damage information by using various methods. For example, the crack detection method described in Japanese Patent No. 4006007 and the rust and peeling detection method described in Japanese Patent Publication No. 2010-538258 can be used. In addition, the extraction processing unit 104 can extract damage information by using a machine learning method. For example, an image with the type and size of damage as a label is given as teacher data, a learning device such as DNN (Deep Neural Network) is generated by machine learning, and the damage is detected using the generated learning device. Can be done.
  • DNN Deep Neural Network
  • the extraction processing unit 104 may extract damage information from individual captured images and combine the corresponding information into one, or may extract damage information from one image obtained by synthesizing a plurality of captured images. ..
  • the damage can be represented as a vector having a start point and an end point, and in this case, a hierarchical structure between the vectors may be considered as described in WO2017 / 110279A.
  • FIG. 8 is a diagram showing an example of extracted damage information.
  • the extraction processing unit 104 can store the extracted damage information in the storage device 200 as damage information 208.
  • the generation processing unit 105 creates a three-dimensional model of the building based on the plurality of input images (step S120: generation processing, generation step).
  • There are various models such as a 3D point cloud model, a 3D surface model created based on the 3D point cloud model, a 3D polygon model, and a texture-mapped image.
  • the generation processing unit 105 can create a three-dimensional model by using, for example, a method of SfM (Structure from Motion).
  • SfM is a method for restoring a three-dimensional shape from a multi-viewpoint image.
  • FIG. 6 is a diagram showing a point cloud 500 (an example of a point cloud).
  • the size is not calculated by SfM, for example, by installing a scaler with known dimensions on the subject and taking a picture, it is possible to associate it with the actual scale.
  • the generation processing unit 105 applies, for example, a TIN model (TIN: triangulated irregular network) to the point cloud data obtained in this way to approximate the surface of the building with a triangle, and based on the result, a solid model (solid model).
  • a solid model (three-dimensional model) can be obtained.
  • the three-dimensional shape of the building is configured as a combination of three-dimensional members such as solid blocks and blocks.
  • the generation processing unit 105 may use the result.
  • the generation processing unit 105 may automatically generate a solid model by using an algorithm such as RANSAC (RandomSampleConsensus) without any operation of the user.
  • RANSAC RandomSampleConsensus
  • the generation processing unit 105 may calculate the change of these information by using the information of the three-dimensional position, the color (R, G, B) and the brightness of the point cloud.
  • the generation processing unit 105 may read the model.
  • the acquisition processing unit 103 is a plurality of images of the building, and the shooting date and time are different from the plurality of images stored in the storage device 200 (memory).
  • a plurality of images (for example, images whose shooting date and time are newer than the stored images and which are not used for generating the three-dimensional model) can be acquired (acquisition processing, acquisition step).
  • the association processing unit 107 can associate the acquired plurality of images with the members of the three-dimensional model stored in the storage device 200 (association process, association step).
  • the association processing unit 107 can perform the association process (association step) based on the correlation between the acquired plurality of images and the plurality of images stored in the storage device 200, for example.
  • the specific processing unit 106 specifies the members constituting the building in the three-dimensional model (step S130: specific processing, specific step). That is, the specific processing unit 106 specifies "which member of the building each region of the three-dimensional model corresponds to".
  • the specifying processing unit 106 may specify the member based on the operation of the user, or may specify the member without the operation of specifying the member by the user.
  • the specific processing unit 106 may use information on the shape and dimensions of the member when specifying the member.
  • FIG. 7 is a diagram showing an example of a three-dimensional model in which members are specified.
  • the three-dimensional model 510 of the bridge is composed of each member of the floor slab 512, the wall portion 514, and the leg portion 516.
  • the memory control processing unit 109 can store the generated data indicating the three-dimensional model in the storage device 200 as the three-dimensional model data 206 (memory control processing, storage control step).
  • the above-mentioned point cloud is information indicating the position of the feature point included in the photographed image, and since the member is specified in step S130, the photographed image corresponding to each member can be specified.
  • the reception processing unit 110 receives the designation of the image selection criteria (step S140: reception processing, reception process), and the selection processing unit 108 (processor) takes an image according to the criteria.
  • An image (representative image) corresponding to the specified member is selected from the images (step S140: selection process, selection step).
  • FIG. 9 is a diagram showing how a selection criterion for a representative image is set, and the reception processing unit 110 displays such a screen on the display device 20.
  • the reception processing unit 110 sets the priority of the information included in the damage information (number of damages, magnitude, degree (degree) of damages, time change) according to the user's operation. Further, the reception processing unit 110 sets the number of representative images to be selected (one image or a plurality of images) according to the operation of the user. According to such a setting, the selection processing unit 108 selects a specified number of images according to the set priority. This allows the user to quickly and easily select a representative image.
  • the aspect shown in FIG. 9 is an example of setting the selection criteria, and the representative image may be selected based on other criteria (for example, the image quality of the captured image, the type of the member, the identification number of the member, etc.).
  • FIG. 10 is a diagram showing how the output mode of the processing result is set.
  • the user selects a radio button via the operation unit 300 and inputs a numerical value to display a two-dimensional inspection record, display a three-dimensional model, display a representative image, and have a composite image mapped. , It is possible to select whether or not to highlight the damage and display the 2D inspection record on the 3D model.
  • the image arrangement processing unit 112, the information input unit 114, and the display processing unit 116 correspond to the specified member, the selected image, and the damage information independently or in cooperation with each other. Attach and output (step S150: output processing, output process).
  • steps S110 to S150 can be repeated until the end determination is affirmed in step S160.
  • the setting of the output mode and the output in the set mode can be performed at the timing desired by the user. For example, after displaying the processing result in one mode, it can be displayed again in another mode. The specific contents of each output mode will be described below.
  • FIG. 11 shows a representative image (representative image) in an area (image area) designated as an area in which an image is arranged by the image arrangement processing unit 112 in a two-dimensional inspection record (an example of a “document file in a specified format”). This is an example in which 552,554,556,558) are arranged.
  • the representative image 552 shows the uki 552A
  • the representative image 554 shows the peeling 554A.
  • the record in this format may span multiple pages.
  • FIG. 11 shows a representative image (representative image) in an area (image area) designated as an area in which an image is arranged by the image arrangement processing unit 112 in a two-dimensional inspection record (an example of a “document file in a specified format”). This is an example in which 552,554,556,558) are arranged.
  • the representative image 552 shows the uki 552A
  • the representative image 554 shows the peeling 554A.
  • the record in this format may span multiple pages.
  • the information input unit 114 further inputs damage information such as a photo number, a member name, an element number, a type of damage, and a degree of damage to an area designated as an information area in the document file ( Information input processing, information input process).
  • damage information such as a photo number, a member name, an element number, a type of damage, and a degree of damage to an area designated as an information area in the document file ( Information input processing, information input process).
  • the content of the "memo" field may be automatically input by the information input unit 114 based on the damage information, or may be input based on the user's operation.
  • the display processing unit 116 causes the display device 20 to display such a screen when the "two-dimensional inspection record display mode" is on on the screen of FIG. Further, the display processing unit 116 stores the information corresponding to this screen as the inspection record data 210.
  • a document such as an inspection record may be in a format specified by the Ministry of Land, Infrastructure, Transport and Tourism, a local government, or the like, or may be in another format. Documents in other specified formats other than inspection records may be used.
  • the representative images may be sorted and arranged in the document by using the member as a key, or the representative images of each member may be sorted and arranged by using the type of damage as a key (for each type of damage). good.
  • the user can quickly and easily select a representative image and create an inspection record. That is, the user can easily handle an image of the building, a three-dimensional model of the building, and damage information.
  • FIG. 12 shows a state in which the 3D model 520 and the pins 532 and 534 (position information) indicating the positions of the selected image (representative image) in the 3D model are associated and displayed on the display device (first). It is a figure which shows the result of display processing).
  • the display timing of the representative image is "when the pin is specified" in FIG. 10
  • the pin is displayed at the position of the representative image in this way.
  • the three-dimensional model displays the vicinity of the wall surface of the leg of the bridge, and the display processing unit 116 enlarges or reduces the three-dimensional model according to the user's operation via the operation unit 300.
  • the viewpoint can be moved and the line-of-sight direction can be changed.
  • the display processing unit 116 may turn on or off the display of a specific member (for example, a member instructed by the user) when displaying the three-dimensional model.
  • the display processing unit 116 performs such a display when the "three-dimensional model display mode" is on in FIG.
  • the three-dimensional models without textures are shown in FIGS. 12 to 15, the same processing can be performed on the three-dimensional models with textures (see FIGS. 16 and the description related to the same figure).
  • the display processing unit 116 may highlight the damage information on the three-dimensional model. For example, it may be displayed by tracing the damage or its outline with a thick line or a line of a conspicuous color, such as cracking, peeling, or peeling.
  • the display processing unit 116 identifies and displays the pins in a manner corresponding to the damage information. Specifically, the display processing unit 116 displays by changing the pin symbol according to the type of damage, but the color or number may be changed in addition to the symbol, or it may be displayed in the form of an icon. .. Moreover, the display mode may be changed according to the degree of damage. In the example of FIG. 12, the user can select (click, etc.) a pin by moving the cursor 540 on the screen with, for example, the mouse 320.
  • FIG. 13 shows a state (result of the second display processing) in which the display processing unit 116 displays a representative image of the specified pin (position information) displayed on the screen on the display device 20. It is a figure.
  • the part (a) in FIG. 13 is a state in which the representative image 552 of "Uki” is displayed by the designation of the pin 532, and the part (b) in the figure is the representative image 554 of the "peeling" by the designation of the pin 534. It is in the displayed state.
  • the display processing unit 116 may display the representative image on a region different from the three-dimensional model, a different screen, or a different display device.
  • FIG. 14 is a diagram showing a state in which a representative image is displayed on the three-dimensional model from the beginning (results of the first display process and the third display process).
  • the display processing unit 116 performs such a display when the display of the representative image is set to "from the beginning" in FIG.
  • the pin and the representative image are connected by a leader line, the user can easily grasp the relationship between the pin and the representative image.
  • a representative image may be displayed on a balloon-shaped figure connected to a pin.
  • FIG. 15 is a diagram showing a state (result of the fourth display process) in which a portion of the two-dimensional inspection record including the representative image corresponding to the selected pin is displayed.
  • the display processing unit 116 can perform such a display when the "two-dimensional inspection record display on the three-dimensional model" is turned on in FIG.
  • the display processing unit 116 may display such a partial two-dimensional inspection record on a different area, a different screen, or a different display device from the three-dimensional model.
  • the user can quickly and easily browse the creation of the representative image and the inspection record. That is, the user can easily handle an image of the building, a three-dimensional model of the building, and damage information.
  • FIG. 16 is a diagram showing a state in which the three-dimensional model 522 in which the composite image is mapped to the three-dimensional model 520 (see FIG. 12) is displayed on the display device 20. Such a display is performed when "Mapping of composite image" is turned on in FIG. 10.
  • the image synthesizing unit 117 (processor) synthesizes an image corresponding to the specified member among the plurality of captured images, and the display processing unit 116 maps the combined image to the specified member and displays it. Display on the device 20 (fifth display process). Image composition and mapping may be performed on some members of the building or on all members. In the example shown in FIG.
  • the display processing unit 116 performs identification display of position information (pins 532, 534, 536 are displayed with symbols according to the type of damage) in the same manner as described above, and the position is displayed.
  • a representative image or a two-dimensional inspection record may be displayed depending on the selection of information.
  • the damage information may be highlighted.
  • FIG. 16 shows an example in which a separately generated composite image is mapped to a three-dimensional model having only a three-dimensional shape
  • the generation processing unit 105 uses the above-mentioned SfM or the like to create a textured three-dimensional model. May be generated directly.
  • the generation processing unit 105 may use another algorithm such as MVS (Multi-View Stereo). MVS can generate a finer three-dimensional model in that it has more information than a monocular.
  • MVS Multi-View Stereo
  • the user can easily handle the image of the building, the three-dimensional model of the building, and the damage information.
  • Image processing system 10 Image processing device 20 Display device 30 Camera 100 Processing unit 102 Input processing unit 103 Acquisition processing unit 104 Extraction processing unit 105 Generation processing unit 106 Specific processing unit 107 Correspondence processing unit 108 Selection processing unit 109 Storage control processing unit 110 Reception processing unit 112 Image placement processing unit 114 Information input unit 116 Display processing unit 117 Image composition unit 118 Communication control unit 200 Storage device 202 Captured image 204 Composite image 206 Three-dimensional model data 208 Damage information 210 Inspection record data 300 Operation unit 310 Keyboard 320 Mouse 500 Point group 510 3D model 512 Floor slab 514 Wall 516 Leg 520 3D model 522 3D model 532 pin 534 pin 536 pin 540 cursor 552 representative image 552A Uki 554 representative image 554A peeling 556 representative image 558 representative Images S100 to S160 Each step of the image processing method

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Strategic Management (AREA)
  • Economics (AREA)
  • Tourism & Hospitality (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Marketing (AREA)
  • General Business, Economics & Management (AREA)
  • Quality & Reliability (AREA)
  • Operations Research (AREA)
  • Educational Administration (AREA)
  • Development Economics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Primary Health Care (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Geometry (AREA)
  • Game Theory and Decision Science (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Data Mining & Analysis (AREA)
  • Processing Or Creating Images (AREA)
  • Bridges Or Land Bridges (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Image Processing (AREA)
PCT/JP2021/002200 2020-03-02 2021-01-22 画像処理装置、画像処理方法、及び画像処理プログラム Ceased WO2021176877A1 (ja)

Priority Applications (6)

Application Number Priority Date Filing Date Title
CN202180013765.7A CN115104128B (zh) 2020-03-02 2021-01-22 图像处理装置、图像处理方法及图像处理程序
JP2022505021A JP7529761B2 (ja) 2020-03-02 2021-01-22 画像処理装置、画像処理方法、及び画像処理プログラム
EP21765299.9A EP4116931A4 (en) 2020-03-02 2021-01-22 IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM
US17/819,388 US20220383487A1 (en) 2020-03-02 2022-08-12 Image processing apparatus, image processing method, and image processing program
JP2024119366A JP7743579B2 (ja) 2020-03-02 2024-07-25 画像処理装置、及び画像処理装置による画像処理方法
JP2025149135A JP2025175087A (ja) 2020-03-02 2025-09-09 画像処理装置、及び画像処理方法

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020035172 2020-03-02
JP2020-035172 2020-03-02

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/819,388 Continuation US20220383487A1 (en) 2020-03-02 2022-08-12 Image processing apparatus, image processing method, and image processing program

Publications (1)

Publication Number Publication Date
WO2021176877A1 true WO2021176877A1 (ja) 2021-09-10

Family

ID=77613397

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/002200 Ceased WO2021176877A1 (ja) 2020-03-02 2021-01-22 画像処理装置、画像処理方法、及び画像処理プログラム

Country Status (5)

Country Link
US (1) US20220383487A1 (https=)
EP (1) EP4116931A4 (https=)
JP (3) JP7529761B2 (https=)
CN (1) CN115104128B (https=)
WO (1) WO2021176877A1 (https=)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023126206A (ja) * 2022-02-28 2023-09-07 東芝ライテック株式会社 保守点検結果提示装置
EP4407552A4 (en) * 2021-09-22 2025-01-15 FUJIFILM Corporation INFORMATION PROCESSING DEVICE, METHOD AND PROGRAM AS WELL AS IMAGE DATA STRUCTURE
EP4411353A4 (en) * 2021-09-28 2025-02-19 FUJIFILM Corporation Information processing device, information processing method, and program

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7401478B2 (ja) * 2021-03-12 2023-12-19 キヤノン株式会社 情報処理装置、情報処理方法、及びプログラム
CN116304155A (zh) * 2023-02-22 2023-06-23 广联达科技股份有限公司 基于二维图片的三维构件检索方法、装置、设备及介质

Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4006007B2 (ja) 2004-11-10 2007-11-14 大成建設株式会社 ひび割れ検出方法
JP2010538258A (ja) 2007-09-03 2010-12-09 コリア エクスプレスウェイ コーポレイション 映像処理方法を用いた鋼橋の塗膜検査システム及びその処理方法
JP2011070632A (ja) * 2009-08-27 2011-04-07 Fujifilm Corp 検査システム、携帯端末、検査方法およびプログラム
JP2011192270A (ja) 2010-02-19 2011-09-29 West Japan Railway Co 構造物管理用図面作成装置、及びその処理プログラム
JP2014115915A (ja) * 2012-12-11 2014-06-26 Hitachi Ltd ３次元モデル生成装置、３次元モデル生成方法及び３次元モデル生成プログラム
WO2017010279A1 (ja) 2015-07-13 2017-01-19 横浜ゴム株式会社 難燃ホース用ゴム組成物及び難燃ホース
WO2017119202A1 (ja) * 2016-01-06 2017-07-13 富士フイルム株式会社 構造物の部材特定装置及び方法
JP2017167969A (ja) * 2016-03-17 2017-09-21 首都高技術株式会社 損傷抽出システム
JP2017168077A (ja) * 2016-03-09 2017-09-21 株式会社リコー 画像処理方法、表示装置および点検システム
JP2018181235A (ja) * 2017-04-21 2018-11-15 古河電気工業株式会社 報告書作成装置、風力発電設備点検システム、プログラム、及び風力発電設備の点検報告書の作成方法
JP2019082933A (ja) * 2017-10-31 2019-05-30 株式会社日本ソフト 報告書作成システム
JP2019174982A (ja) * 2018-03-27 2019-10-10 富士通株式会社 モデル表示制御方法、モデル表示制御プログラム及びモデル表示制御システム
JP2020030519A (ja) * 2018-08-21 2020-02-27 株式会社日本ソフト 工程管理システム、工程管理方法、及びプログラム

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10521960B2 (en) * 2017-05-03 2019-12-31 General Electric Company System and method for generating three-dimensional robotic inspection plan
US9156105B1 (en) * 2011-10-17 2015-10-13 Redzone Robotics, Inc. Managing infrastructure data
RU2586566C1 (ru) * 2015-03-25 2016-06-10 Общество с ограниченной ответственностью "Лаборатория 24" Способ отображения объекта
WO2017110278A1 (ja) 2015-12-25 2017-06-29 富士フイルム株式会社 情報処理装置及び情報処理方法
JP6780093B2 (ja) * 2017-03-30 2020-11-04 富士フイルム株式会社 画像処理装置及び画像処理方法
CA3099638A1 (en) * 2017-05-07 2018-11-15 Manam Applications Ltd. System and method for construction 3d modeling and analysis
EP3416131A1 (en) * 2017-06-16 2018-12-19 Episurf IP-Management AB System and method for creating a decision support material indicating damage to an anatomical joint
JP6970817B2 (ja) 2018-04-11 2021-11-24 富士フイルム株式会社 構造物管理装置、構造物管理方法、及び構造物管理プログラム
US10832476B1 (en) * 2018-04-30 2020-11-10 State Farm Mutual Automobile Insurance Company Method and system for remote virtual visualization of physical locations
US11508056B2 (en) * 2019-02-28 2022-11-22 Measure Global, Inc. Drone inspection analytics for asset defect detection
WO2020176304A1 (en) * 2019-02-28 2020-09-03 Skidmore Owings & Merrill Llp Machine learning tool for structures
JP7193728B2 (ja) * 2019-03-15 2022-12-21 富士通株式会社 情報処理装置および蓄積画像選択方法
US11222464B2 (en) * 2019-08-22 2022-01-11 The Travelers Indemnity Company Intelligent imagery
US20210110076A1 (en) * 2019-10-09 2021-04-15 Consulting Engineers, Corp. Method and system for optimization of the external and internal sheathing for the construction of a building
EP4123106A4 (en) * 2020-03-16 2023-08-16 FUJIFILM Corporation IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD AND IMAGE PROCESSING PROGRAM

Patent Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4006007B2 (ja) 2004-11-10 2007-11-14 大成建設株式会社 ひび割れ検出方法
JP2010538258A (ja) 2007-09-03 2010-12-09 コリア エクスプレスウェイ コーポレイション 映像処理方法を用いた鋼橋の塗膜検査システム及びその処理方法
JP2011070632A (ja) * 2009-08-27 2011-04-07 Fujifilm Corp 検査システム、携帯端末、検査方法およびプログラム
JP2011192270A (ja) 2010-02-19 2011-09-29 West Japan Railway Co 構造物管理用図面作成装置、及びその処理プログラム
JP2014115915A (ja) * 2012-12-11 2014-06-26 Hitachi Ltd ３次元モデル生成装置、３次元モデル生成方法及び３次元モデル生成プログラム
WO2017010279A1 (ja) 2015-07-13 2017-01-19 横浜ゴム株式会社 難燃ホース用ゴム組成物及び難燃ホース
WO2017119202A1 (ja) * 2016-01-06 2017-07-13 富士フイルム株式会社 構造物の部材特定装置及び方法
JP2017168077A (ja) * 2016-03-09 2017-09-21 株式会社リコー 画像処理方法、表示装置および点検システム
JP2017167969A (ja) * 2016-03-17 2017-09-21 首都高技術株式会社 損傷抽出システム
JP2018181235A (ja) * 2017-04-21 2018-11-15 古河電気工業株式会社 報告書作成装置、風力発電設備点検システム、プログラム、及び風力発電設備の点検報告書の作成方法
JP2019082933A (ja) * 2017-10-31 2019-05-30 株式会社日本ソフト 報告書作成システム
JP2019174982A (ja) * 2018-03-27 2019-10-10 富士通株式会社 モデル表示制御方法、モデル表示制御プログラム及びモデル表示制御システム
JP2020030519A (ja) * 2018-08-21 2020-02-27 株式会社日本ソフト 工程管理システム、工程管理方法、及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4116931A4

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4407552A4 (en) * 2021-09-22 2025-01-15 FUJIFILM Corporation INFORMATION PROCESSING DEVICE, METHOD AND PROGRAM AS WELL AS IMAGE DATA STRUCTURE
EP4411353A4 (en) * 2021-09-28 2025-02-19 FUJIFILM Corporation Information processing device, information processing method, and program
JP2023126206A (ja) * 2022-02-28 2023-09-07 東芝ライテック株式会社 保守点検結果提示装置

Also Published As

Publication number Publication date
CN115104128A (zh) 2022-09-23
JP7743579B2 (ja) 2025-09-24
EP4116931A1 (en) 2023-01-11
EP4116931A4 (en) 2023-09-06
US20220383487A1 (en) 2022-12-01
JP2024150685A (ja) 2024-10-23
JP2025175087A (ja) 2025-11-28
JP7529761B2 (ja) 2024-08-06
CN115104128B (zh) 2025-03-21
JPWO2021176877A1 (https=) 2021-09-10

Similar Documents

Publication Publication Date Title
JP7743579B2 (ja) 画像処理装置、及び画像処理装置による画像処理方法
US9208607B2 (en) Apparatus and method of producing 3D model
JP6144364B2 (ja) 作業支援用データ作成プログラム
US8487926B2 (en) Method and apparatus for generating 3D image using 2D photograph images
JP6723748B2 (ja) 画像管理装置、画像管理方法及びプログラム
CN108876706B (zh) 根据全景图像的缩略图生成
JP7480066B2 (ja) 拡張現実オーバーレイのためのシステムおよび方法
JP2018026064A (ja) 画像処理装置、画像処理方法、システム
JP6293386B2 (ja) データ処理装置、データ処理方法及びデータ処理プログラム
KR101851303B1 (ko) 3차원 공간 재구성 장치 및 방법
US20220405878A1 (en) Image processing apparatus, image processing method, and image processing program
JP2016071682A (ja) オブジェクト配置装置、オブジェクト配置方法、及び、プログラム
CN113167568A (zh) 坐标计算装置、坐标计算方法和计算机可读记录介质
JP6405539B2 (ja) 多視点画像に対するラベル情報の処理装置及びそのラベル情報の処理方法
CN111521127B (zh) 测量方法、测量装置以及记录介质
KR100943451B1 (ko) Ｔｉｆｆ 포맷을 이용한 ｄｅｍ―ｔｉｆｆ정사영상 제작 시스템 및 그 방법
JP6962242B2 (ja) 情報処理装置，重畳表示プログラム，重畳表示方法
CN112652056B (zh) 一种3d信息展示方法及装置
CN112634439B (zh) 一种3d信息展示方法及装置
JP2003087550A (ja) 画像合成装置、画像合成方法、画像合成処理プログラムを記録したコンピュータ読み取り可能な記録媒体
Saran et al. Augmented annotations: Indoor dataset generation with augmented reality
JP2021055266A (ja) 石垣配置支援プログラム、石垣配置支援方法、及び、石垣配置支援装置
JP2019175350A (ja) 構造物検査システム、構造物検査装置及び構造物検査方法
JP5925007B2 (ja) 画像処理装置、画像処理方法及びプログラム
JP2025092230A (ja) 画像処理装置、画像処理方法、及び、プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21765299

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022505021

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021765299

Country of ref document: EP

Effective date: 20221004

WWG Wipo information: grant in national office

Ref document number: 202180013765.7

Country of ref document: CN