WO2022209583A1

WO2022209583A1 - Information processing device, information processing method, and program

Info

Publication number: WO2022209583A1
Application number: PCT/JP2022/009399
Authority: WO
Inventors: 逸平難波田; 令司松本
Original assignee: パイオニア株式会社; パイオニアスマートセンシングイノベーションズ株式会社
Priority date: 2021-03-31
Filing date: 2022-03-04
Publication date: 2022-10-06

Abstract

An information processing device (10) is provided with an acquisition unit (120), a difference image generation unit (140), and an extraction unit (160). The acquisition unit (120) acquires a first image captured at a first date and time, a second image captured at a second date and time, a third image captured at a third date and time, and a fourth image captured at a fourth date and time. The difference image generation unit (140) generates a first difference image indicating a difference between the first image and the third image, and a second difference image indicating a difference between the second image and the fourth image. The extraction unit (160) extracts, as a first common component, a common component of the first difference image and the second difference image. The third date and time and the fourth date and time are both later than the first date and time, and later than the second date and time.

Description

Information processing device, information processing method, and program

The present invention relates to an information processing device, an information processing method, and a program.

　It is necessary to detect changes in images of roads and buildings for maintenance of high-precision maps. Here, a shadow or the like may appear in the image, and it is necessary to separate the changed portion to be detected from the change of the shadow or the like. For example, a photographed image of a road surface includes shadows of vehicles running on the road surface and structures on the roadside. When trying to detect changes in the paint on the road surface using such an image, there is a risk that the edge of the shadow will be recognized as the edge of the paint. And since the shadow changes depending on the time of day, season, and weather, there is a risk that it will be detected as a changed part.

Patent Document 1 describes that an area having a luminance lower than the luminance of the valley of a histogram of a luminance image is determined as a shadow area, and an area having a luminance higher than the luminance of the valley is determined as a non-shadow area. It is In the technique disclosed in Japanese Patent Application Laid-Open No. 2002-200002, the change extraction is performed after shadow removal processing is performed on the luminance image.

Japanese Patent Application Laid-Open No. 2002-200000 describes determining whether or not a partial area that constitutes an image is a shadow area by performing threshold processing on the ratio of luminance values of corresponding pixels in a plurality of images to be compared. and removing it from the image when a partial region is determined to be a shadow region.

JP-A-2004-252733 JP-A-2000-251053

However, with the techniques of Patent Documents 1 and 2, it is considered that there is a problem that shadows cannot always be determined accurately due to shooting conditions such as the weather and characteristics of other subjects. Further, in the techniques of Patent Documents 1 and 2, an intermediate image is generated from which shadows are removed prior to change detection, and it is considered that the accuracy of the intermediate image greatly affects the accuracy of change detection.

One example of the problem to be solved by the present invention is to provide a technique for detecting changes in images easily and accurately.

The invention according to claim 1,
A first image taken on a first date and time, a second image taken on a second date and time, a third image taken on a third date and time, and a fourth image taken on a fourth date and time. an acquisition unit that acquires and
a difference image generating unit that generates a first difference image indicating the difference between the first image and the third image and a second difference image indicating the difference between the second image and the fourth image;
an extraction unit that extracts a common component between the first difference image and the second difference image as a first common component,
In the information processing apparatus, both the third date and time and the fourth date and time are after the first date and time and after the second date and time.

The invention according to claim 15,
A first image taken on a first date and time, a second image taken on a second date and time, a third image taken on a third date and time, and a fourth image taken on a fourth date and time. a retrieving step of retrieving and
a difference image generating step of generating a first difference image showing the difference between the first image and the third image and a second difference image showing the difference between the second image and the fourth image;
an extraction step of extracting a common component between the first difference image and the second difference image as a first common component;
In the information processing method, both the third date and time and the fourth date and time are after the first date and time and after the second date and time.

The invention according to claim 16,
A program that causes a computer to execute each step of the information processing method according to claim 15.

1 is a block diagram illustrating the functional configuration of an information processing apparatus according to a first embodiment; FIG. 4 is a diagram for explaining processing performed by the information processing apparatus according to the first embodiment; FIG. 4 is a flowchart illustrating the flow of an information processing method according to the first embodiment; 4 is a flowchart illustrating in detail processing performed by the information processing apparatus according to the first embodiment; 7 is a flow chart showing a modification of the processing performed by the information processing apparatus according to the first embodiment; It is a figure which illustrates the computer for implement|achieving an information processing apparatus. FIG. 11 is a diagram for explaining processing performed by an information processing apparatus according to the second embodiment; FIG. 11 is a diagram for explaining processing performed by an information processing apparatus according to the second embodiment; FIG. 11 is a block diagram illustrating the functional configuration of an information processing apparatus according to a third embodiment; FIG. 11 is a flowchart illustrating the flow of an information processing method performed by an information processing apparatus according to a third embodiment; It is a figure for demonstrating the process which the information processing apparatus which concerns on 4th Embodiment performs.

Embodiments of the present invention will be described below with reference to the drawings. In addition, in all the drawings, the same constituent elements are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

In the following description, each component of the information processing apparatus 10 indicates a functional unit block, not a hardware unit configuration, unless otherwise specified. Each component of the information processing apparatus 10 includes a CPU of an arbitrary computer, a memory, a program loaded in the memory, a storage medium such as a hard disk for storing the program, an interface for network connection, and an arbitrary combination of hardware and software. is realized by a combination of There are various modifications of the implementation method and device.

(First embodiment)
FIG. 1 is a block diagram illustrating the functional configuration of an information processing device 10 according to the first embodiment. The information processing apparatus 10 according to this embodiment includes an acquisition unit 120 , a difference image generation unit 140 and an extraction unit 160 . Acquisition unit 120 obtains a first image taken on a first date and time, a second image taken on a second date and time, a third image taken on a third date and time, and a third image taken on a fourth date and time. and obtain a fourth image. The difference image generator 140 generates a first difference image indicating the difference between the first image and the third image, and a second difference image indicating the difference between the second image and the fourth image. The extraction unit 160 extracts a common component between the first difference image and the second difference image as a first common component. Both the third date and time and the fourth date and time are after the first date and time and after the second date and time. A detailed description is given below.

FIG. 2 is a diagram for explaining the processing performed by the information processing device 10 according to this embodiment. For example, image A1 corresponds to the first image, image A2 corresponds to the second image, image B1 corresponds to the third image, and image B2 corresponds to the fourth image. In the example of this figure, images A1 to A3 and images B1 to B3 are respectively orthoimages including roads 31, and white lines 30 are drawn on the roads 31. FIG. A shadow 32 is reflected in the images A1 to A3 and the images B1 to B3. The information processing apparatus 10 can detect this change in the white line 30 .

In the example of this figure, images A1 to A3 and images B1 to B3 are different images. For example, images A1 to A3 are images taken at different times on the same day, and images B1 to B3 are images taken at different times on the same day. The date on which the images B1 to B3 were taken is later than the date on which the images A1 to A3 were taken.

A difference image C1 indicates the difference between the image A1 and the image B1. A difference image C2 indicates the difference between the image A2 and the image B2. A difference image C3 indicates the difference between the image A3 and the image B3. The common component image D1 is an image showing common components of the difference image C1, the difference image C2, and the difference image C3. From this common component image D1, only the changing portion of the white line 30 is extracted. For example, the difference image C1 corresponds to the first difference image, and the difference image C2 corresponds to the second difference image. The common component image D1 corresponds to an image representing the first common component extracted by the extraction unit 160. FIG.

In other words, according to the information processing apparatus 10 according to the present embodiment, the change occurring between the time when the first image and the second image were captured and the time when the third image and the fourth image were captured is can be extracted as one common component. Also, the change between the first and second images and the change between the third and fourth images are not extracted as the first common component. Therefore, short-term changes such as shadows and long-term changes such as white lines and buildings can be separately detected.

Since the method according to the present embodiment does not use characteristics unique to shadows, it is less likely to be affected by uncertain factors related to shadow determination, and it can also deal with factors other than shadows that cause short-term changes. It is possible. It is possible to remove the influence of short-term changes without being conscious of whether the event is falling leaves, shadows, or snow. Furthermore, the method according to the present embodiment does not generate an intermediate image with shadows removed, for example. Therefore, the processing is simple and the load is low, and factors that lower the detection accuracy of the changed portion are less likely to enter. As a result, it is possible to detect the changed portion with high accuracy.

FIG. 3 is a flowchart illustrating the flow of the information processing method according to this embodiment. The information processing method according to this embodiment includes an acquisition step S10, a difference image generation step S20, and an extraction step S30. In the acquisition step S10, a first image taken on a first date and time, a second image taken on a second date and time, a third image taken on a third date and time, and a fourth image taken on a fourth date and time. A fourth image is acquired. In the difference image generating step S20, a first difference image indicating the difference between the first image and the third image and a second difference image indicating the difference between the second image and the fourth image are generated. In the extraction step S30, a common component between the first difference image and the second difference image is extracted as a first common component. Both the third date and time and the fourth date and time are after the first date and time and after the second date and time. The information processing method according to this embodiment can be executed by the information processing apparatus 10 according to this embodiment.

FIG. 4 is a flowchart illustrating in detail the processing performed by the information processing apparatus 10 according to this embodiment. Processing performed by the information processing apparatus 10 according to the present embodiment will be described in detail below with reference to FIGS. 1 and 4. FIG. A series of processes for detecting changed portions using the image acquired by the acquisition unit 120 is hereinafter referred to as “detection processing”.

The acquisition unit 120 acquires images from, for example, the storage unit 200 accessible from the acquisition unit 120 . Although FIG. 1 shows an example in which the storage unit 200 is provided outside the information processing device 10 , the storage unit 200 may be provided inside the information processing device 10 . A plurality of images are stored in advance in the storage unit 200 in association with shooting dates and shooting positions. Acquisition unit 120 can acquire an image from storage unit 200 by specifying a position and time. The shooting position of the image can be specified by latitude and longitude, or can be specified by using an identifier that indicates an individual intersection or landmark.

The image acquired by the acquisition unit 120 is not particularly limited, but may be an image of a road, a structure, or the like. Also, the image acquired by the acquisition unit 120 is not particularly limited, but is, for example, an image captured by an imaging device provided on a moving object (vehicle, two-wheeled vehicle, etc.) that moves on a road. The image acquired by the acquisition unit 120 may be an orthoimage, a top view image of a road, a structure, or the like taken from directly above, a side image, an aerial photograph, or the like. Further, when detecting a three-dimensional change, the acquisition unit 120 may acquire an image based on point cloud data acquired by LiDAR. Among them, the image acquired by the acquisition unit 120 for detection processing is preferably an orthoimage. That is, it is preferable that all of the first image, the second image, the third image, and the fourth image are orthorectified images. By doing so, it is easy to detect the changed portion with high accuracy. If the image held in the storage unit 200 is not an orthorectified image, the acquisition unit 120 may orthorectify the image read from the storage unit 200 to obtain an image used for detection processing. It is preferable that the images acquired by the acquisition unit 120 for the detection process are of the same type.

The information processing apparatus 10 uses the image acquired by the acquisition unit 120 to perform processing for detecting changed portions. The images acquired by the acquisition unit 120 for detection processing include at least a first image, a second image, a third image, and a fourth image. However, the image acquired by the acquisition unit 120 for the detection process may further include one or more images. The images acquired by the acquisition unit 120 for detection processing are preferably images including the same object. In addition, it is preferable that the images acquired by the acquisition unit 120 for the detection process have overlapping photographing areas. In addition, it is not necessary to predetermine the target.

The information processing device 10 can detect, for example, long-term changes in the target. In the example of FIG. 2, the object is the white line. Objects include, but are not limited to, structures such as buildings, signs, traffic lights, and information boards, division lines such as white lines drawn on roads, crosswalks, road markings, road surfaces, and the like. The paint drawn on the road is peeled off due to deterioration over time, road construction, etc., and is repainted anew, so it is necessary to detect these changes. In addition, it is preferable to be able to detect cracks, damage, dents, and the like on the road surface.

On the other hand, the image acquired by the acquisition unit 120 for detection processing may further include a short-term change substance. A short-term changeable substance changes temporarily, and the information processing apparatus 10 can output a detection result excluding changes due to a short-term changeable substance. The short-term change object is not particularly limited, but may be, for example, at least one of shadows, snow, puddles, earth and sand, vehicles, fallen objects, fallen leaves, and garbage.

As described above, the first image is an image taken on the first date and time, the second image is an image taken on the second date and time, and the third image is an image taken on the third date and time. and the fourth image is an image taken on the fourth date and time. Both the third date and time and the fourth date and time are after the first date and time and after the second date and time. Here, the interval between the first date and time and the second date and time is preferably narrower than the interval between the first date and time and the third date and time. The interval between the first date and time and the second date and time is preferably narrower than the interval between the second date and time and the fourth date and time. The interval between the third date and time and the fourth date and time is preferably narrower than the interval between the first date and time and the third date and time. Also, the interval between the third date and time and the fourth date and time is preferably narrower than the interval between the second date and time and the fourth date and time. In this embodiment, the first date and time are different from the second date and time, and the third date and time are different from the fourth date and time. The first date and time and the second date and time are included in the first period, and the third date and time and the fourth date and time are included in the second period. The second epoch is later than the first epoch. According to the information processing device 10, it is possible to detect a change in the target between the first period and the second period. Although the lengths of the first period and the second period are not particularly limited, they are, for example, one day.

First, in S101, the information processing apparatus 10 accepts designation of the detection target position and time. The position and time of the detection target can be input to the information processing device 10 by the user, for example. The position of the detection target may be specified, for example, by latitude and longitude, or may be specified by an intersection or landmark identifier as described above. Also, the position of the detection target may be designated by a distance or a relative position from a predetermined reference point. Landmarks include traffic lights, signs, bus stops, pedestrian crossings, and the like. The designation of the detection target period is performed by designating the first period and the second period. Each epoch can be done by specifying the start and end of the epoch, or by specifying the start and length of the epoch.

When the information processing apparatus 10 receives the designation of the detection target position and time, the acquisition unit 120 selects an image corresponding to the detection target position and time from the storage unit 200 in S102 and acquires it for the detection process. Specifically, the acquisition unit 120 acquires an image including the position designated as the detection target position. Alternatively, the acquisition unit 120 acquires an image captured from a position designated as a detection target position. When the detection target position is specified by an intersection or landmark identifier, the acquisition unit 120 acquires an image including the intersection or landmark indicated by the identifier. In the present embodiment, the acquiring unit 120 acquires two or more images captured during the first period, and acquires two or more images captured during the second period. Acquisition unit 120 acquires the first image and the second image such that the first time period includes the first date and time and the second date and time, for example. Acquisition unit 120 also acquires the third image and the fourth image such that the second time period includes the third date and time and the fourth date and time.

When the acquisition unit 120 acquires the image, the difference image generation unit 140 performs edge detection in S201. Specifically, the differential image generation unit 140 performs edge detection processing on each image acquired by the acquisition unit 120 . The edge detection process can be based, for example, on the luminance gradients of the image. However, when trying to detect a change in color, edge detection processing may be performed further based on the hue of the image. Note that the difference image generation unit 140 may perform correction processing or the like on the image acquired by the acquisition unit 120 as necessary. An edge image is obtained by edge detection processing.

Next, in S202, the difference image generation unit 140 generates a plurality of difference images. A difference image is an image showing a difference between any image taken in the first period and any image taken in the second period. Specifically, the differential image generator 140 associates the positions of the edge images obtained in S201. Then, the values of the pixels at the corresponding positions are compared between the two edge images whose positions are associated with each other. That is, the values of pixels corresponding to the same position in the real space are compared. Pixels with matching pixel values are determined as common portions, and pixels with non-matching pixel values are determined as differences. A difference image is an image in which the pixels determined to be the difference are identifiable from the other pixels.

The combination of two images used to generate the difference image is not particularly limited, but it is preferable to combine images in which the states of short-term objects such as shadows are different from each other. For example, the differential image generation unit 140 preferably derives the direction of the sun based on the shooting date and time associated with each image, and generates the differential image using two images in which the direction of the sun is separated by a predetermined angle or more.

As a method of associating the position of the edge image, for example, if the position and orientation of the image are associated with GPS information or the like, there is a method of using that position and orientation. It is also possible to match using fiducials in both images in addition to the information associated with the images. Furthermore, if there is no suitable reference object in both images, the reference object is searched over a wide range including the successively captured images. It is conceivable to estimate the position of the target image by obtaining the relative positional relationship from the found reference object and improve the accuracy of registration. The difference image can further be associated with position information based on the position information of the original image of the difference image.

Next, in S301, the extraction unit 160 extracts common components of a plurality of difference images. The extractor 160 can generate an image showing this common component. Specifically, the extraction unit 160 associates the positions of the plurality of difference images in the same manner as the positions of the edge images were associated in S202. Then, values of pixels at corresponding positions are compared in a plurality of difference images with associated positions. Then, the pixels having the same pixel values are determined as the common portion. Alternatively, among all the difference images, pixels having a ratio of matching pixel values equal to or higher than a predetermined threshold value are determined to be common portions. Thus, the image showing the pixels determined as the common portion so as to be identifiable from the other pixels is the image showing the first common component.

In the example of FIG. 1, the information processing device 10 further includes an output unit 180 that outputs an image showing the first common component. In S401, the output unit 180 outputs data for displaying the image representing the first common component generated by the extraction unit 160 on the display device. Note that if the extraction result shows that there is no common component in the difference image, the output unit 180 may output information indicating that there is no changed portion. Note that the output unit 180 may output information indicating that there is no changed portion when the ratio of the number of pixels indicating the common component in the image indicating the first common component is equal to or less than a predetermined ratio. .

Note that after the extraction unit 160 performs S301, detection processing may be performed again using a different image for the same detection target position and time. Then, when the degree of matching of the results of a plurality of detection processes is equal to or higher than a predetermined standard, the output section 180 may output the results. By doing so, more reliable results can be output.

FIG. 5 is a flowchart showing a modified example of processing performed by the information processing apparatus 10 according to this embodiment. According to this modified example, by increasing the number of images used in the detection process, it is possible to improve the detection accuracy of the changed portion. In this modified example, a predetermined number of differential images are generated. This modification is the same as the example of FIG. 4 except for the points described below.

S101 in this example is the same as S101 in FIG. When the information processing apparatus 10 receives the designation of the detection target position and time, the acquisition unit 120 selects an image corresponding to the detection target position and time from the storage unit 200 in S103 and acquires it for the detection process. In this example, the acquisition unit 120 acquires one image captured during the first period and one image captured during the second period. The combination of two images acquired by the acquisition unit 120 is not particularly limited, but a combination of images with different shadow states is preferable. For example, the acquisition unit 120 preferably derives the direction of the sun based on the date and time of photography associated with each image, and acquires two images in which the direction of the sun is separated by a predetermined angle or more.

When the acquisition unit 120 acquires the image, the difference image generation unit 140 performs edge detection in S201. S201 in this example is the same as S201 in FIG.

Next, the difference image generation unit 140 generates a difference image in S203. A difference image is an image showing a difference between an image captured in the first period and an image captured in the second period. The method for generating the differential image is the same as the method described for S202 in FIG.

Next, in S204, the differential image generation unit 140 determines whether the number of differential images generated so far has reached a predetermined number. If the number of difference images has not reached the predetermined number (N of S204), the process returns to S103, and the acquisition unit 120 acquires images again. When the number of difference images reaches a predetermined number (Y of S204), the difference image generation unit 140 outputs all the generated difference images to the extraction unit 160. FIG.

Next, in S<b>301 , the extraction unit 160 extracts common components of the plurality of difference images acquired from the difference image generation unit 140 . The method of extracting the common part is the same as S301 in FIG. In S401, the output unit 180 also outputs data for displaying the image representing the first common component generated by the extraction unit 160 on the display device.

Note that when the extraction unit 160 extracts common components using three or more differential images, the same image may be used to generate two or more differential images. However, as described in the second embodiment, the same image cannot be used to generate all the difference images unless the image does not contain short-term change substances.

The hardware configuration of the information processing device 10 will be described below. Each functional configuration unit of the information processing apparatus 10 may be implemented by hardware (eg, hardwired electronic circuit) that implements each functional configuration unit, or may be implemented by a combination of hardware and software (eg, combination of an electronic circuit and a program for controlling it, etc.). A case in which each functional configuration unit of the information processing apparatus 10 is implemented by a combination of hardware and software will be further described below.

FIG. 6 is a diagram illustrating a computer 1000 for realizing the information processing apparatus 10. FIG. Computer 1000 is any computer. For example, the computer 1000 is an SoC (System On Chip), a personal computer (PC), a server machine, a tablet terminal, a smart phone, or the like. The computer 1000 may be a dedicated computer designed to implement the information processing apparatus 10, or may be a general-purpose computer.

The computer 1000 has a bus 1020 , a processor 1040 , a memory 1060 , a storage device 1080 , an input/output interface 1100 and a network interface 1120 . The bus 1020 is a data transmission path through which the processor 1040, memory 1060, storage device 1080, input/output interface 1100, and network interface 1120 mutually transmit and receive data. However, the method of connecting processors 1040 and the like to each other is not limited to bus connection. The processor 1040 is various processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array). The memory 1060 is a main memory implemented using a RAM (Random Access Memory) or the like. The storage device 1080 is an auxiliary storage device implemented using a hard disk, SSD (Solid State Drive), memory card, ROM (Read Only Memory), or the like.

The input/output interface 1100 is an interface for connecting the computer 1000 and input/output devices. For example, the input/output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

The network interface 1120 is an interface for connecting the computer 1000 to the network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). A method for connecting the network interface 1120 to the network may be a wireless connection or a wired connection.

The storage device 1080 stores program modules that implement each functional component of the information processing apparatus 10 . The processor 1040 reads each program module into the memory 1060 and executes it, thereby realizing the function corresponding to each program module.

As described above, according to the present embodiment, the difference image generation unit 140 generates the first difference image indicating the difference between the first image and the third image, and the second difference image indicating the difference between the second image and the fourth image. Generate an image. The extraction unit 160 extracts a common component between the first difference image and the second difference image as a first common component. Therefore, image change detection can be performed easily and accurately.

(Second embodiment)
7 and 8 are diagrams for explaining the processing performed by the information processing apparatus 10 according to the second embodiment. The information processing apparatus 10 according to this embodiment is the same as the information processing apparatus 10 according to the first embodiment except for the points described below.

In this embodiment, the first image, the second image, the third image, and the fourth image are all images containing roads. The first image and the second image are the same image with no short-term changeable objects on the road, or the third image and the fourth image are the same image with no short-term changeable objects on the road. is an image of

When using images that do not contain short-term changes such as shadows, the number of images used for detection processing can be reduced. In the example of FIG. 7, image A4 corresponds to the first and second images, image B4 corresponds to the third image, and image B5 corresponds to the fourth image. That is, the first image and the second image are the same image. The first date and time and the second date and time are the same. Image A4 includes road 31 and white line 30, but shadow 32 is not included. The image B4 and the image B5 include the road 31, the white line 30, and the shadow 32. FIG. A difference image C4 indicates the difference between the image A4 and the image B4. A difference image C5 indicates the difference between the image A4 and the image B5. The common component image D2 is an image showing common components between the difference image C4 and the difference image C5. Using the first image, the second image, the third image, and the fourth image, it is possible to detect a changed portion in the same manner as in the first embodiment.

Also, in the example of FIG. 8, image A6 corresponds to the first image, image A7 corresponds to the second image, and image B6 corresponds to the third and fourth images. That is, the third image and the fourth image are the same image. The third date and time and the fourth date and time are the same. Image B6 includes road 31 and white line 30, but shadow 32 is not included. The image A6 and the image A7 include a road 31, a white line 30, and a shadow 32. FIG. A difference image C6 indicates the difference between the image A6 and the image B6. A difference image C7 indicates the difference between the image A7 and the image B6. The common component image D3 is an image showing common components of the difference image C6 and the difference image C7. Using the first image, the second image, the third image, and the fourth image, it is possible to detect a changed portion in the same manner as in the first embodiment.

In this embodiment, each image held in the storage unit 200 is associated with information indicating the presence or absence of a short-term changeable substance. Then, the acquiring unit 120 can select and acquire an image without a short-term changeable substance. When the acquisition unit 120 acquires an image without short-term change substances, the difference image generation unit 140 can use the image to generate a plurality of difference images. Further, the differential image generation unit 140 may use the image to generate all the differential images.

Examples of images with no short-term change include images taken when the sun is out or when the weather is cloudy. The images stored in the storage unit 200 can be attached with information indicating the presence/absence of short-term changeable substances by confirming each image in advance. Further, information indicating the presence or absence of a short-term changeable substance may be attached to each image based on the date and time when the image was taken.

As described above, according to this embodiment, the same actions and effects as those of the first embodiment can be obtained. In addition, the first image and the second image are the same image with no short-term changeable objects on the road, or the third image and the fourth image are with the short-term changeable objects on the road. not the same image. Therefore, the changed portion can be detected using a smaller number of images.

(Third Embodiment)
FIG. 9 is a block diagram illustrating the functional configuration of the information processing device 10 according to the third embodiment. The information processing apparatus 10 according to the present embodiment is the same as the information processing apparatus 10 according to at least one of the first and second embodiments except for the points described below.

The information processing apparatus 10 according to this embodiment further includes a determination unit 110 that determines whether or not to start detection processing. Then, when the determination unit 110 determines to start the detection process, the detection process is started. As described above, the detection process is a series of processes for detecting changed portions using the images acquired by the acquisition unit 120. Specifically, the acquisition unit 120 acquires the first image, the second image, the third image, and the In this processing, the fourth image is acquired, the difference image generation unit 140 generates the first difference image and the second difference image, and the extraction unit 160 extracts the first common component. The detection process includes an acquisition step S10, a difference image generation step S20, and an extraction step S30. A detailed description is given below.

The determination unit 110 determines the timing of performing the detection process. In particular, the determination unit 110 determines whether or not to start the detection process so that the detection process is performed at a timing when there is a relatively high possibility that the target has changed. First to fourth examples of the determination method performed by the determination unit 110 will be described below. Note that the determination method performed by the determination unit 110 is not limited to the following. Further, the information processing apparatus 10 may combine and execute a plurality of determination methods.

<First example>
In the first example, the determination unit 110 determines to start the detection process each time a predetermined period elapses. The determination unit 110 stores the date and time when the most recent detection process was performed. Then, determination unit 110 calculates the elapsed time since the most recent detection process was performed at predetermined time intervals (for example, every day). Then, when the calculated elapsed time is longer than a predetermined period, the determination unit 110 determines to start the detection process. On the other hand, if the calculated elapsed time is shorter than the predetermined period, it is determined not to start the detection process. The determination unit 110 can perform determination for each position. If it is determined to start the detection process, that position is designated as the position to be detected. Also, a predetermined period including the timing of the most recent detection process is set as the first period, and a predetermined period ending at the timing when the calculated elapsed time becomes longer than the predetermined period is set as the second period. is preferred. According to this example, detection processing can be performed periodically.

<Second example>
In the second example, the first image, the second image, the third image, and the fourth image are all images containing roads. The determination unit 110 acquires information about roads. Then, the determination unit 110 determines to start the detection process when determining that a predetermined event has occurred based on the road information.

Information about roads is, for example, road construction information provided by road management companies. Acquisition unit 120 can acquire information about roads, for example, from server 50 of a service that provides road information. A predetermined event is an event that has a high possibility of causing a change in an object, such as road construction. Since there is a high possibility that the white lines of the road will change during road construction, it is preferable that the detection process be performed to check whether there is any change before and after the road construction.

Information about roads includes the date and time or period when an event occurs and the location where the event occurs. The information processing device 10 can determine the position and time of the detection target based on the information on the road. For example, acquisition unit 120 acquires a first image and a second image captured before the occurrence of the event, and a third image and a fourth image captured after the occurrence of the event. That is, a first period of time is specified before the occurrence of the event, and a second period of time is specified after the occurrence of the event. Also, the position at which the event occurred is specified as the detection target position.

In this example, the determination unit 110 acquires information about roads at predetermined intervals (for example, every day). Then, it is checked whether or not a predetermined event has occurred after the most recent detection process. The determination unit 110 determines to start the detection process when a predetermined event occurs. On the other hand, if the predetermined event has not occurred, it is determined not to start the detection process. According to this example, it is possible to perform the detection process by detecting the occurrence of an event that is highly likely to cause a change in the object.

<Third example>
In the third example, the first image, the second image, the third image, and the fourth image are all images containing roads. The determination unit 110 acquires information about the traffic volume of the road, and determines the timing for starting the detection process based on the information about the traffic volume.

In this example, the determination unit 110 acquires information on traffic volume from the server 50 of a service that provides traffic information, for example. Information about traffic volume is, for example, information indicating traffic volume at each position on the road at each time. The determination unit 110 calculates the integrated value of the traffic volume at each position on the road using the information about the traffic volume. Note that the integrated value may be the integrated value after the most recent detection process. Then, determination unit 110 determines whether or not the calculated integrated value exceeds a predetermined value. The determination unit 110 determines to start the detection process when the integrated value exceeds a predetermined value. On the other hand, if the integrated value does not exceed the predetermined value, it is determined not to start the detection process. The determination unit 110 can perform determination for each position. Then, when it is determined to start the detection process, that position is specified as the position to be detected. Further, it is preferable that a predetermined period including the timing of the most recent detection process is set as the first period, and a predetermined period ending at the timing when the integrated value exceeds a predetermined value is set as the second period.

　The more traffic there is, the more likely it is that the paint will peel off on the road. Therefore, by determining the timing of the detection process based on the traffic volume, the change can be accurately detected.

<Fourth example>
In the fourth example, the first image, the second image, the third image, and the fourth image are all images containing roads. The determination unit 110 acquires information about the traffic flow on the road and determines the timing for starting the detection process based on the information about the traffic flow.

In this example, the determination unit 110 acquires information on traffic flow from, for example, the server 50 of a service that provides traffic information. Information about traffic flow is, for example, information indicating the state of traffic flow at each branch point (three-forked road, intersection, five-way intersection, etc.) on a road at each hour. The information on traffic flow may be information indicating the number or ratio of vehicles that have passed through a branch point within a predetermined time for each traveling direction (straight ahead, right turn, etc.). For example, at an intersection where there used to be right-turning vehicles, only straight-line vehicles from a certain period of time, there is a possibility that the traffic regulations have changed, that is, the markings on the road surface have changed. Also, the information about traffic flow may be information indicating the state of the flow of vehicles on the road. For example, if the number of convoys changed before and after a certain point in time, it is possible that the number of lanes changed, ie the markings on the road surface changed.

The determination unit 110 monitors information on traffic flow at predetermined intervals and detects changes in traffic flow. Specifically, the determination unit 110 compares the information about the traffic flow when the most recent detection process was performed and the latest information about the traffic flow. Then, the determination unit 110 determines to start the detection process when the difference between the information about the traffic flow when the most recent detection process is performed and the latest information about the traffic flow is greater than a predetermined reference value. do. On the other hand, when the difference is not larger than the predetermined reference value, it is determined not to start the detection process. Alternatively, if the determination unit 110 calculates the similarity between the information on the traffic flow when the most recent detection process was performed and the latest information on the traffic flow, and the calculated similarity is smaller than a predetermined reference value. , it is determined to start the detection process. On the other hand, when the degree of similarity is not smaller than a predetermined reference value, it is determined not to start the detection process.

More specifically, the determination unit 110 may compare the proportion of straight vehicles at each junction based on information on traffic flow. In this case, the determination unit 110 determines to start the detection process when the difference between the percentage of vehicles traveling straight when the most recent detection process was performed and the latest percentage of vehicles traveling straight is greater than a predetermined reference value. do. On the other hand, if this difference is not greater than a predetermined reference value, it is determined not to start the detection process. The determination unit 110 may also detect and compare the number of lanes at each position on the road using information on traffic flow. In this case, the determination unit 110 determines to start the detection process when the number of lanes when the most recent detection process is performed is different from the latest number of lanes. On the other hand, if these numbers are the same, it is determined not to start the detection process.

The determination unit 110 can perform determination for each position or branch point. Then, if it is determined to start the detection process, that position or branch point is designated as the position to be detected. A predetermined period including the timing of the most recent detection process is defined as the first period, and a predetermined period ending at the timing when the latest traffic flow information used for the determination is obtained is defined as the second period. It is preferable to

The hardware configuration of the computer that implements the information processing apparatus 10 according to this embodiment is represented, for example, by FIG. 6, as in the first embodiment. However, the storage device 1080 of the computer 1000 that implements the information processing apparatus 10 of this embodiment further stores a program module that implements the function of the determination unit 110 of this embodiment.

FIG. 10 is a flowchart illustrating the flow of the information processing method performed by the information processing apparatus 10 according to this embodiment. The information processing method according to this embodiment further includes a determination step S40 of determining whether or not to start the detection process. Then, when it is determined to start the detection process in determination step S40 (Y in S40), the detection process is started. On the other hand, if it is determined not to start the detection process in determination step S40 (N of S40), the detection process is not started.

As described above, according to this embodiment, the same actions and effects as those of the first embodiment can be obtained. In addition, according to the present embodiment, the detection process is started when the determination unit 110 determines to start the detection process. Therefore, detection processing can be performed at appropriate timing.

(Fourth embodiment)
FIG. 11 is a diagram for explaining processing performed by the information processing apparatus 10 according to the fourth embodiment. The information processing apparatus 10 according to this embodiment is the same as the information processing apparatus 10 according to the third embodiment except for the points described below.

In the information processing apparatus 10 according to the present embodiment, the determination unit 110 obtains three or more images captured on different dates and times, and determines a common component of each two temporally continuous images among the three or more images as a second image. Extract each as a common component. The determination unit 110 detects the presence or absence of a change in the target in the image acquired by the determination unit 110 by comparing the extracted plurality of second common components in time series. Then, the determination unit 110 determines to start the detection process when a change is detected in the object in the image acquired by the determination unit 110 . A detailed description is given below.

According to the present embodiment, the determination unit 110 determines whether or not a change has occurred in the target based on a plurality of images. Then, when a change occurs, the information processing apparatus 10 can perform detection processing and extract a specific changed portion.

In the example of FIG. 11, the determination unit 110 acquires images E1 to E6 with different shooting dates and times. Each of the images E1-E6 includes a road 31, a white line 30 and a shadow 32. FIG. In this example, the object is the white line 30 . Images E1 to E6 are images shot in this order. The images E1 to E6 may be images taken on different days, for example, but the image shooting interval is not particularly limited. The determination unit 110 extracts a common component of each two temporally consecutive images among the images E1 to E6 as a second common component. Images F1 to F5 are images showing the second common component. That is, the determination unit 110 determines the image F1 showing the common component of the image E1 and the image E2, the image F2 showing the common component of the image E2 and the image E3, the image F3 showing the common component of the image E3 and the image E4, the image E4 and the image An image F4 showing common components of E5 and an image F5 showing common components of images E5 and E6 are generated.

Next, the determination unit 110 detects whether there is a change in the target in the images E1 to E6 by comparing the images F1 to F5 in time series. In this example, there is no difference between images F1 and F2, and there is a difference between images F2 and F3 and between images F3 and F4. Also, there is no difference between the image F4 and the image F5. In this case, it can be determined that there is a change between the imaging timing of the image E3 on which the image F3 is based and the imaging timing of the image E4. The determination by the determining unit 110 is also not affected by short-term changes such as shadows.

Note that the determination unit 110 can perform determination using at least three images. For example, when the same processing is performed using three images E1 to E3, there is no difference between the images F1 and F2, so there is a change between the imaging timing of the image E1 and the imaging timing of the image E3. It can be determined that there is no Further, when the same processing is performed using three images E2 to E4, there is a difference between the image F2 and the image F3, so there is a change between the timing of capturing the image E2 and the timing of capturing the image E4. can be determined to have occurred.

When the same processing is performed using the four images E1 to E4, there is no difference between the images F1 and F2, but there is a difference between the images F2 and F3. It can be determined that a change has occurred between the photographing timing of . In this way, when four or more images are used, the change timing can be grasped more precisely. Therefore, the determination unit 110 can acquire four or more images taken at different dates and times, and detect the presence or absence of change and determine the timing of change, rather than determining the timing of change from only three images. preferable.

In addition, when a change is detected in the object in the image acquired by the determination unit 110, the determination unit 110 can further specify the change timing. Therefore, it is preferable that the acquisition unit 120 acquires the first and second images captured before the change timing and the third and fourth images captured after the change timing. That is, it is preferable to specify the first period before the change timing and the second period after the change timing. Further, the information processing apparatus 10 preferably designates the shooting position of the image acquired by the determination unit 110 and used for determination as the position to be detected.

The determination unit 110 can acquire, for example, three or more images from the storage unit 200 via the acquisition unit 120. The image acquired by the determination unit 110 is the same as the image acquired by the acquisition unit 120 described in the first embodiment. The determination unit 110 preferably acquires an image similar to the image acquired by the acquisition unit 120 for the detection process, and uses it to determine whether or not there is a change. That is, it is preferable that all the images acquired by the determination unit 110 are orthorectified images. Also, the images acquired by the determination unit 110 are preferably of the same type. It is preferable that the imaging regions of the images acquired by the determination unit 110 overlap each other. The determination unit 110 can detect the presence or absence of long-term changes in the target. In addition, it is not necessary to predetermine the target. The image acquired by the determination unit 110 may further include a short-term change substance. The three or more images acquired by the determination unit 110 are images different from each other, and are images taken at different dates and times.

After acquiring the images, the determination unit 110 performs edge detection processing on each acquired image to obtain an edge image. The edge detection processing is as described above in the description of S201. Note that the determination unit 110 may perform correction processing or the like on the acquired image as necessary.

Next, the determination unit 110 associates the positions of the two edge images obtained from the two temporally consecutive images in the same manner as described above in the description of S202. Then, the determining unit 110 compares the values of the pixels at the corresponding positions in the two edge images whose positions are associated with each other, and determines the pixels having the same pixel values as the common portion. Thus, the image showing the pixels determined as the common portion so as to be identifiable from the other pixels is the image showing the second common component.

After generating a plurality of images showing the second common component from the acquired images, the determination unit 110 compares them in time series to detect the presence or absence of change. Specifically, the determination unit 110 performs a process of extracting a difference between two temporally continuous images showing the second common component, and when the difference is extracted, there is a change between the two images. I judge. On the other hand, if no difference is extracted, it is determined that there is no change between the two images. The process of extracting the difference between the images showing the second common component can be performed in the same manner as the difference image generating section 140 generates the difference image from the edge image. Note that the determination unit 110 may determine that no difference has been extracted when the ratio of the number of pixels indicating the difference in the difference image is equal to or less than a predetermined ratio.

When it is determined that there is a change between any two images showing the second common component, the determination unit 110 determines that there is a change timing at least during the shooting period of the three or more images acquired by the determination unit 110. I can judge. When determining that there is a change timing during the shooting period of the three or more images acquired by the determination unit 110, the determination unit 110 determines to start the detection process. Then, the information processing apparatus 10 performs detection processing.

When the determination unit 110 acquires four or more images, that is, when three or more images showing the second common component are generated, the determination unit 110 may further specify the change timing in detail. By comparing a plurality of images showing the second common component, if there is an image that differs from both before and after, it can be specified that the change timing is between the two images that are the basis of that image. In addition, comparing images showing a plurality of second common components, if the first or last image is temporally different from other images, the change timing is can be identified.

As described above, according to this embodiment, the same actions and effects as those of the first embodiment can be obtained. In addition, according to this embodiment, the determination unit 110 detects whether or not there is a change in the object in the acquired image. Then, the determination unit 110 determines to start the detection process when a change is detected in the object in the image acquired by the determination unit 110 . Therefore, detection processing can be performed at appropriate timing.

Although the embodiments and examples have been described above with reference to the drawings, these are examples of the present invention, and various configurations other than those described above can be adopted.

This application claims priority based on Japanese Patent Application No. 2021-059608 filed on March 31, 2021, and the entire disclosure thereof is incorporated herein.

10 Information processing device 30 White line 31 Road 32 Shadow 50 Server 110 Judgment unit 120 Acquisition unit 140 Difference image generation unit 160 Extraction unit 180 Output unit 200 Storage unit 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input/output interface 1120 Network interface

Claims

A first image taken on a first date and time, a second image taken on a second date and time, a third image taken on a third date and time, and a fourth image taken on a fourth date and time. an acquisition unit that acquires and
a difference image generating unit that generates a first difference image indicating the difference between the first image and the third image and a second difference image indicating the difference between the second image and the fourth image;
an extraction unit that extracts a common component between the first difference image and the second difference image as a first common component,
The information processing apparatus, wherein both the third date and time and the fourth date and time are later than the first date and time and later than the second date and time.
In the information processing device according to claim 1,
The information processing apparatus, wherein the interval between the first date and time and the second date and time is narrower than the interval between the first date and time and the third date and time.
In the information processing device according to claim 1 or 2,
The information processing apparatus, wherein the interval between the third date and time and the fourth date and time is narrower than the interval between the first date and time and the third date and time.
In the information processing device according to any one of claims 1 to 3,
The information processing apparatus, wherein the first image, the second image, the third image, and the fourth image are all orthorectified images.
In the information processing device according to any one of claims 1 to 4,
each of the first image, the second image, the third image, and the fourth image is an image including a road;
The first image and the second image are the same image in which short-term changeable objects are not captured on the road, or the third image and the fourth image are the same image in which short-term changeable objects are captured on the road. An information processing device that is not the same image.
In the information processing device according to claim 5,
The information processing apparatus, wherein the short-term changeable object is at least one of shadows, snow, puddles, earth and sand, vehicles, fallen objects, fallen leaves, and garbage.
In the information processing device according to any one of claims 1 to 4,
The information processing apparatus, wherein the first date and time and the second date and time are different from each other, and the third date and time and the fourth date and time are different from each other.
In the information processing device according to any one of claims 1 to 7,
The information processing apparatus further comprising an output unit that outputs an image showing the first common component.
In the information processing device according to any one of claims 1 to 8,
The acquisition unit acquires the first image, the second image, the third image, and the fourth image, and the difference image generation unit generates the first difference image and the second difference image, The extraction unit further comprises a determination unit that determines whether or not to start detection processing for extracting the first common component,
The information processing apparatus in which the detection process starts when the determination unit determines to start the detection process.
In the information processing device according to claim 9,
The information processing apparatus, wherein the determination unit determines to start the detection process each time a predetermined period elapses.
In the information processing device according to claim 9 or 10,
each of the first image, the second image, the third image, and the fourth image is an image including a road;
The determination unit is
Get information about roads,
determining to start the detection process when it is determined that a predetermined event has occurred based on the information about the road;
The acquisition unit performs information processing for acquiring the first image and the second image captured before the occurrence of the event, and the third image and the fourth image captured after the occurrence of the event. Device.
In the information processing device according to any one of claims 9 to 11,
each of the first image, the second image, the third image, and the fourth image is an image including a road;
The determination unit is
Get information about traffic flow on roads,
An information processing device that determines a timing for starting the detection process based on the information about the traffic flow.
In the information processing device according to any one of claims 9 to 12,
The determination unit is
Acquiring three or more images with different shooting dates and times,
Extracting a common component of each two temporally consecutive images out of the three or more images as a second common component,
Detecting whether or not there is a change in the target in the image acquired by the determination unit by comparing the extracted plurality of second common components in time series,
An information processing apparatus that determines to start the detection process when a change is detected in the object in the image acquired by the determination unit.
In the information processing device according to claim 13,
The determination unit further specifies the change timing when a change is detected in the object in the image acquired by the determination unit,
The information processing device, wherein the acquisition unit acquires the first image and the second image captured before the change timing, and the third image and the fourth image captured after the change timing.
A first image taken on a first date and time, a second image taken on a second date and time, a third image taken on a third date and time, and a fourth image taken on a fourth date and time. a retrieving step of retrieving and
a difference image generating step of generating a first difference image showing the difference between the first image and the third image and a second difference image showing the difference between the second image and the fourth image;
an extraction step of extracting a common component between the first difference image and the second difference image as a first common component;
The information processing method, wherein both the third date and time and the fourth date and time are after the first date and time and after the second date and time.
A program that causes a computer to execute each step of the information processing method according to claim 15.