JP2024076389A - Information processing device, information processing method, and program - Google Patents

Abstract

[Problem] One example is to provide a technology for easily and accurately detecting changes in an image. [Solution] An information processing device 10 includes an acquisition unit 120, a difference image generation unit 140, and an extraction unit 160. The acquisition unit 120 acquires a first image taken on a first date and time, a second image taken on a second date and time, a third image taken on a third date and time, and a fourth image taken on a fourth date and time. The difference image generation unit 140 generates a first difference image showing the difference between the first image and the third image, and a second difference image showing the difference between the second image and the fourth image. The extraction unit 160 extracts a common component between the first difference image and the second difference image as a first common component. The third date and the fourth date and time are both later than the first date and time and later than the second date and time. [Selected Figure] FIG.

Description

The present invention relates to an information processing device, an information processing method, and a program.

For the maintenance of high-precision maps, it is necessary to detect changes in images of roads and buildings. However, shadows and other objects may appear in the images, and it is necessary to distinguish between the changes that are of interest and the changes in the shadows. For example, an image of a road surface may contain the shadows of vehicles traveling on the road surface and structures on the roadside. When trying to detect changes in paint on the road surface using such images, there is a risk that the edges of the shadows will be recognized as the edges of the paint. Furthermore, because shadows change depending on the time of day, season, and weather, there is a risk that they will be detected as changes.

Patent Document 1 describes a method of determining regions with a lower luminance than the luminance of the valleys in the histogram of the luminance image as shadow regions, and regions with a higher luminance than the luminance of the valleys as non-shadow regions. The technology in Patent Document 1 further performs a shadow removal process on the luminance image before extracting changes.

Patent document 2 describes a method for determining whether a partial area that constitutes an image is a shadow area by performing threshold processing on the ratio of brightness values between corresponding pixels in multiple images to be compared, and describes a method for removing the partial area from the image when it is determined that the partial area is a shadow area.

JP 2004-252733 A JP 2000-251053 A

However, the techniques of Patent Documents 1 and 2 have the problem that shadows cannot always be accurately determined due to shooting conditions such as weather and the characteristics of other subjects. In addition, the techniques of Patent Documents 1 and 2 generate intermediate images from which shadows have been removed prior to change detection, and it is believed that the accuracy of the intermediate images significantly affects the accuracy of change detection.

One example of the problem that the present invention aims to solve is to provide a technology that can easily and accurately detect changes in images.

The invention described in claim 1 is
an acquisition unit that acquires a first image captured on a first date and time, a second image captured on a second date and time, a third image captured on a third date and time, and a fourth image captured on a fourth date and time;
a difference image generating unit that generates a first difference image indicating a difference between the first image and the third image, and a second difference image indicating a difference between the second image and the fourth image;
an extracting unit that extracts a common component between the first difference image and the second difference image as a first common component,
The third date and time and the fourth date and time are both later than the first date and time and later than the second date and time for the information processing device.

The invention described in claim 15 is
an acquisition step of acquiring a first image taken at a first date and time, a second image taken at a second date and time, a third image taken at a third date and time, and a fourth image taken at a fourth date and time;
a difference image generating step of generating a first difference image indicating a difference between the first image and the third image, and a second difference image indicating a difference between the second image and the fourth image;
an extraction step of extracting a common component between the first difference image and the second difference image as a first common component,
In the information processing method, the third date and time and the fourth date and time are both later than the first date and time and later than the second date and time.

The invention described in claim 16 is
A program for causing a computer to execute each step of the information processing method according to claim 15.

1 is a block diagram illustrating a functional configuration of an information processing apparatus according to a first embodiment. FIG. 2 is a diagram for explaining a process performed by the information processing device according to the first embodiment. 4 is a flowchart illustrating the flow of an information processing method according to the first embodiment. 4 is a flowchart illustrating in detail a process performed by the information processing apparatus according to the first embodiment. 10 is a flowchart showing a modified example of processing performed by the information processing device according to the first embodiment. FIG. 1 is a diagram illustrating a computer for implementing an information processing device. FIG. 11 is a diagram for explaining a process performed by an information processing device according to a second embodiment. FIG. 11 is a diagram for explaining a process performed by an information processing device according to a second embodiment. FIG. 13 is a block diagram illustrating a functional configuration of an information processing device according to a third embodiment. 13 is a flowchart illustrating the flow of an information processing method performed by an information processing apparatus according to a third embodiment. FIG. 13 is a diagram for explaining a process performed by an information processing device according to a fourth embodiment.

The following describes an embodiment of the present invention with reference to the drawings. Note that in all drawings, similar components are given similar reference numerals and descriptions will be omitted where appropriate.

In the following explanation, unless otherwise specified, each component of the information processing device 10 is shown as a functional block rather than a hardware-based configuration. Each component of the information processing device 10 is realized by any combination of hardware and software, centered on the CPU of any computer, memory, a program loaded into the memory, a storage medium such as a hard disk that stores the program, and an interface for network connection. There are various variations in the method and device for realizing this.

First Embodiment
FIG. 1 is a block diagram illustrating a functional configuration of an information processing device 10 according to a first embodiment. The information processing device 10 according to this embodiment includes an acquisition unit 120, a difference image generation unit 140, and an extraction unit 160. The acquisition unit 120 acquires a first image taken at a first date and time, a second image taken at a second date and time, a third image taken at a third date and time, and a fourth image taken at a fourth date and time. The difference image generation unit 140 generates a first difference image showing the difference between the first image and the third image, and a second difference image showing the difference between the second image and the fourth image. The extraction unit 160 extracts a common component between the first difference image and the second difference image as a first common component. The third date and the fourth date and time are both later than the first date and time and later than the second date and time. This will be described in detail below.

Figure 2 is a diagram for explaining the processing performed by the information processing device 10 according to this embodiment. For example, image A1 corresponds to the first image, image A2 corresponds to the second image, image B1 corresponds to the third image, and image B2 corresponds to the fourth image. In the example shown in this figure, images A1 to A3 and images B1 to B3 are each orthoimages including a road 31, on which a white line 30 is drawn. Furthermore, a shadow 32 appears in images A1 to A3 and images B1 to B3. The information processing device 10 can detect changes in this white line 30.

In the example shown in this figure, images A1 to A3 and images B1 to B3 are different images. For example, images A1 to A3 are images taken at different times on the same day, and images B1 to B3 are images taken at different times on the same day. Images B1 to B3 were taken on a day after images A1 to A3 were taken.

Difference image C1 shows the difference between image A1 and image B1. Difference image C2 shows the difference between image A2 and image B2. Difference image C3 shows the difference between image A3 and image B3. And common component image D1 is an image showing the common component between difference image C1, difference image C2 and difference image C3. This common component image D1 is an image in which only the changed portion of the white line 30 is extracted. For example, difference image C1 corresponds to the first difference image, and difference image C2 corresponds to the second difference image. And common component image D1 corresponds to an image showing the first common component extracted by extraction unit 160.

In other words, according to the information processing device 10 of this embodiment, the changes that occurred between the time when the first image and the second image were taken and the time when the third image and the fourth image were taken can be extracted as the first common component. Furthermore, the changes that occurred between the first image and the second image and the changes that occurred between the third image and the fourth image are not extracted as the first common component. Therefore, it is possible to detect short-term changes such as shadows separately from long-term changes in white lines, buildings, etc.

The method according to this embodiment does not use characteristics specific to shadows, so it is not easily affected by uncertain factors related to shadow determination, and can also handle factors other than shadows that cause short-term changes. It is possible to remove the effects of short-term changes without being aware of attributes such as whether the phenomenon causing the change is fallen leaves, shadows, or snow. Furthermore, the method according to this embodiment does not generate intermediate images such as those with shadows removed. Therefore, the processing is simple and low-load, and factors that reduce the accuracy of detecting changed parts are unlikely to be introduced. As a result, it is possible to detect changed parts with high accuracy.

FIG. 3 is a flowchart illustrating the flow of the information processing method according to this embodiment. The information processing method according to this embodiment includes an acquisition step S10, a difference image generation step S20, and an extraction step S30. In the acquisition step S10, a first image taken at a first date and time, a second image taken at a second date and time, a third image taken at a third date and time, and a fourth image taken at a fourth date and time are acquired. In the difference image generation step S20, a first difference image showing the difference between the first image and the third image, and a second difference image showing the difference between the second image and the fourth image are generated. In the extraction step S30, a common component between the first difference image and the second difference image is extracted as a first common component. The third date and the fourth date and time are both later than the first date and time and later than the second date and time. The information processing method according to this embodiment can be executed by the information processing device 10 according to this embodiment.

Figure 4 is a flowchart illustrating in detail the processing performed by the information processing device 10 according to this embodiment. The processing performed by the information processing device 10 according to this embodiment will be described in detail below with reference to Figures 1 and 4. The series of processes for detecting changed portions using the image acquired by the acquisition unit 120 will be referred to as "detection processing" below.

The acquisition unit 120 acquires images, for example, from a storage unit 200 accessible from the acquisition unit 120. Although FIG. 1 shows an example in which the storage unit 200 is provided outside the information processing device 10, the storage unit 200 may be provided inside the information processing device 10. A plurality of images are stored in advance in the storage unit 200 in association with the shooting date and time and the shooting location. The acquisition unit 120 can acquire images from the storage unit 200 by specifying the location and time. The shooting location of an image can be specified by latitude and longitude, or it may be specified using an identifier indicating an individual intersection or landmark.

The image acquired by the acquisition unit 120 is not particularly limited, but may be an image of a road, a structure, etc. Also, the image acquired by the acquisition unit 120 is not particularly limited, but may be, for example, an image captured by an imaging device provided on a moving body (vehicle, motorcycle, etc.) moving on a road. The image acquired by the acquisition unit 120 may be an orthoimage, a top view image of a road or a structure taken from directly above, a side image, an aerial photograph, etc. Also, when detecting a three-dimensional change, the acquisition unit 120 may acquire an image based on point cloud data acquired by LiDAR. Among them, the image acquired by the acquisition unit 120 for the detection process is preferably an orthoimage. That is, it is preferable that the first image, the second image, the third image, and the fourth image are all orthoimages. In this way, it is easy to detect the changed part with high accuracy. If the image held in the storage unit 200 is not an orthoimage, the acquisition unit 120 may obtain an image to be used in the detection process by orthorectifying the image read out from the storage unit 200. It is preferable that the images acquired by the acquisition unit 120 for the detection process are images of the same type.

The information processing device 10 performs a process of detecting changed parts using images acquired by the acquisition unit 120. The images acquired by the acquisition unit 120 for the detection process include at least a first image, a second image, a third image, and a fourth image. However, the images acquired by the acquisition unit 120 for the detection process may further include one or more images. It is preferable that the images acquired by the acquisition unit 120 for the detection process are images that include the same object. It is also preferable that the images acquired by the acquisition unit 120 for the detection process have mutually overlapping shooting areas. It is not necessary to determine the object in advance.

The information processing device 10 can detect long-term changes in an object, for example. In the example of FIG. 2, the object is a white line. Objects are not particularly limited, but include structures such as buildings, signs, traffic lights, and guide boards, as well as white lines and other dividing lines painted on roads, pedestrian crossings, road markings, and road surfaces. Paint on roads peels off or is repainted due to deterioration over time or road construction, and it is necessary to detect these changes. It is also preferable to be able to detect cracks, damage, dents, and the like in the road surface.

On the other hand, the image acquired by the acquisition unit 120 for the detection process may further include short-term changes. Short-term changes are temporary changes, and the information processing device 10 can output detection results excluding changes due to short-term changes. The short-term changes are not particularly limited, but may be, for example, at least one of shadows, snow, puddles, soil, vehicles, fallen objects, fallen leaves, and garbage.

As described above, the first image is an image taken at the first date and time, the second image is an image taken at the second date and time, the third image is an image taken at the third date and time, and the fourth image is an image taken at the fourth date and time. The third date and time and the fourth date and time are both later than the first date and time and later than the second date and time. Here, it is preferable that the interval between the first date and time and the second date and time is narrower than the interval between the first date and time and the third date and time. It is preferable that the interval between the first date and time and the second date and time is narrower than the interval between the second date and time and the fourth date and time. It is also preferable that the interval between the third date and time and the fourth date and time is narrower than the interval between the second date and time and the fourth date and time. In this embodiment, the first date and time and the second date and time are different from each other, and the third date and time and the fourth date and time are different from each other. The first date and time and the second date and time are included in the first period, and the third date and time and the fourth date and time are included in the second period. The second period is later than the first period. According to the information processing device 10, it is possible to detect a change in the object between the first period and the second period. The length of each of the first period and the second period is not particularly limited, but is, for example, one day.

First, in S101, the information processing device 10 accepts the designation of the position and time of the detection target. The position and time of the detection target can be input to the information processing device 10, for example, by the user. The position of the detection target may be designated, for example, by latitude and longitude, or by an identifier of an intersection or landmark as described above. The position of the detection target may also be designated by a distance or relative position from a predetermined reference point. Examples of landmarks include traffic lights, signs, bus stops, and crosswalks. The time of the detection target is designated by designating a first time period and a second time period. Each time period can be designated by designating the start point and end point of that time period, or the start point and length of that time period.

When the information processing device 10 accepts the designation of the position and time of the detection target, the acquisition unit 120 selects an image corresponding to the position and time of the detection target from the storage unit 200 in S102 and acquires it for detection processing. Specifically, the acquisition unit 120 acquires an image including the position designated as the position of the detection target. Alternatively, the acquisition unit 120 acquires an image taken from the position designated as the position of the detection target. When the position of the detection target is designated by an identifier of an intersection or landmark, the acquisition unit 120 acquires an image including the intersection or landmark represented by the identifier. In this embodiment, the acquisition unit 120 acquires two or more images taken in a first period, and acquires two or more images taken in a second period. For example, the acquisition unit 120 acquires the first image and the second image so that the first period includes the first date and time and the second date and time. The acquisition unit 120 also acquires the third image and the fourth image so that the second period includes the third date and time and the fourth date and time.

When the acquisition unit 120 acquires an image, the difference image generation unit 140 performs edge detection in S201. Specifically, the difference image generation unit 140 performs edge detection processing on each image acquired by the acquisition unit 120. The edge detection processing can be performed, for example, based on the luminance gradient of the image. However, when attempting to detect a color change, the edge detection processing may also be performed based on the hue of the image. Note that the difference image generation unit 140 may perform correction processing, etc., as necessary, on the image acquired by the acquisition unit 120. An edge image is obtained by the edge detection processing.

Next, in S202, the difference image generating unit 140 generates multiple difference images. A difference image is an image that shows the difference between any image captured at a first time and any image captured at a second time. Specifically, the difference image generating unit 140 matches the positions of the edge images obtained in S201. Then, it compares the pixel values at corresponding positions in the two edge images whose positions are matched. That is, it compares the pixel values that correspond to the same position in real space. Then, it determines that pixels whose pixel values match are common parts, and determines that pixels whose pixel values do not match are differences. In this way, an image in which the pixels determined to be differences are displayed so that they can be distinguished from the other pixels is the difference image.

The combination of two images used to generate a difference image is not particularly limited, but it is preferable to use a combination of images in which the states of short-term changes such as shadows are different from each other. For example, it is preferable for the difference image generating unit 140 to derive the direction of the sun based on the shooting date and time associated with each image, and generate a difference image using two images in which the directions of the sun are apart by a predetermined angle or more.

One method for matching the positions of edge images is to use the position and orientation when the position and orientation at which the image was taken are associated with each other using GPS information or the like. It is also possible to match the images using reference objects in both images in addition to the information associated with the images. Furthermore, if there is no suitable reference object in both images, it is possible to search for the reference object over a wide range, including images taken in succession, and estimate the position of the target image by determining the relative positional relationship from the reference object found, thereby improving the accuracy of alignment. The difference image can also be associated with location information based on the location information of the image that is the basis of the difference image.

Next, in S301, the extraction unit 160 extracts a common component from the multiple difference images. The extraction unit 160 can generate an image that shows this common component. Specifically, the extraction unit 160 matches the positions of the multiple difference images in the same way as it matched the positions of the edge images in S202. Then, it compares pixel values at corresponding positions in the multiple difference images whose positions are matched. It then determines that pixels whose pixel values all match are a common part. Alternatively, it determines that pixels whose pixel values match at a rate equal to or greater than a predetermined threshold value are a common part among all the difference images. An image in which the pixels determined to be a common part are thus displayed so as to be distinguishable from the other pixels is an image that shows the first common component.

In the example of FIG. 1, the information processing device 10 further includes an output unit 180 that outputs an image showing the first common component. In S401, the output unit 180 outputs data for displaying the image showing the first common component generated by the extraction unit 160 on a display device. Note that, if there is no common component in the difference image as a result of the extraction, the output unit 180 may output information indicating that there is no changed portion. Note that, the output unit 180 may output information indicating that there is no changed portion if the ratio of the number of pixels showing the common component in the image showing the first common component is equal to or less than a predetermined ratio.

After the extraction unit 160 performs S301, the detection process may be performed again using a different image for the same position and time of the detection target. Then, if the degree of agreement between the results of the multiple detection processes is equal to or greater than a predetermined standard, the output unit 180 may output the result. By doing so, it is possible to output a more reliable result.

Figure 5 is a flowchart showing a modified example of the process performed by the information processing device 10 according to this embodiment. According to this modified example, the accuracy of detecting changed parts can be improved by increasing the number of images used in the detection process. In this modified example, a predetermined number of difference images are generated. This modified example is the same as the example in Figure 4, except for the points described below.

S101 in this example is the same as S101 in FIG. 4. When the information processing device 10 accepts the designation of the position and time of the detection target, the acquisition unit 120 selects an image corresponding to the position and time of the detection target from the storage unit 200 in S103 and acquires it for detection processing. In this example, the acquisition unit 120 acquires one image taken in a first time period and one image taken in a second time period. The combination of two images acquired by the acquisition unit 120 is not particularly limited, but it is preferable that the combination be images with different shadow conditions. For example, it is preferable that the acquisition unit 120 derives the direction of the sun based on the shooting date and time associated with each image, and acquires two images in which the directions of the sun are apart by more than a predetermined angle.

When the acquisition unit 120 acquires an image, the difference image generation unit 140 performs edge detection in S201. S201 in this example is the same as S201 in FIG. 4.

Then, in S203, the difference image generating unit 140 generates a difference image. The difference image is an image that shows the difference between the image captured at the first time point and the image captured at the second time point. The method for generating the difference image is the same as the method described for S202 in FIG. 4.

Next, in S204, the difference image generation unit 140 determines whether the number of difference images generated up to that point has reached a predetermined number. If the number of difference images has not reached the predetermined number (N in S204), the process returns to S103, and the acquisition unit 120 acquires images again. If the number of difference images has reached the predetermined number (Y in S204), the difference image generation unit 140 outputs all of the generated difference images to the extraction unit 160.

Next, in S301, the extraction unit 160 extracts common components of the multiple difference images obtained from the difference image generation unit 140. The method for extracting the common parts is the same as in S301 of FIG. 4. In addition, in S401, the output unit 180 outputs data for displaying an image showing the first common component generated by the extraction unit 160 on a display device.

When the extraction unit 160 extracts common components using three or more difference images, the same image may be used to generate two or more difference images. However, as described in the second embodiment, the same image cannot be used to generate all the difference images unless the image does not contain short-term changes.

The hardware configuration of the information processing device 10 is described below. Each functional component of the information processing device 10 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, further explanation is given of the case where each functional component of the information processing device 10 is realized by a combination of hardware and software.

FIG. 6 is a diagram illustrating a computer 1000 for realizing the information processing device 10. The computer 1000 is any computer. For example, the computer 1000 is a SoC (System On Chip), a Personal Computer (PC), a server machine, a tablet terminal, or a smartphone. The computer 1000 may be a dedicated computer designed to realize the information processing device 10, or may be a general-purpose computer.

The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input/output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input/output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection. The processor 1040 is various processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array). The memory 1060 is a main storage device realized using a RAM (Random Access Memory) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, an SSD (Solid State Drive), a memory card, or a ROM (Read Only Memory), or the like.

The input/output interface 1100 is an interface for connecting the computer 1000 to an input/output device. For example, an input device such as a keyboard and an output device such as a display device are connected to the input/output interface 1100.

The network interface 1120 is an interface for connecting the computer 1000 to a network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method for connecting the network interface 1120 to the network may be a wireless connection or a wired connection.

The storage device 1080 stores program modules that realize each functional component of the information processing device 10. The processor 1040 reads each of these program modules into the memory 1060 and executes them to realize the function corresponding to each program module.

As described above, according to this embodiment, the difference image generating unit 140 generates a first difference image showing the difference between the first image and the third image, and a second difference image showing the difference between the second image and the fourth image. The extracting unit 160 extracts the common component between the first difference image and the second difference image as the first common component. Therefore, it is possible to easily and accurately detect changes in the image.

Second Embodiment
7 and 8 are diagrams for explaining the processing performed by the information processing device 10 according to the second embodiment. The information processing device 10 according to this embodiment is the same as the information processing device 10 according to the first embodiment, except for the points described below.

In this embodiment, the first image, the second image, the third image, and the fourth image are all images that include a road. The first image and the second image are the same image that does not show any short-term changes on the road, or the third image and the fourth image are the same image that does not show any short-term changes on the road.

When images that do not include short-term changes such as shadows are used, the number of images used in the detection process can be reduced. In the example of FIG. 7, image A4 corresponds to the first and second images, image B4 corresponds to the third image, and image B5 corresponds to the fourth image. That is, the first and second images are the same image. The first date and time are the same as the second date and time. Image A4 includes the road 31 and the white line 30, but does not include the shadow 32. Images B4 and B5 include the road 31, the white line 30, and the shadow 32. Difference image C4 shows the difference between image A4 and image B4. Difference image C5 shows the difference between image A4 and image B5. And common component image D2 is an image showing the common component between difference image C4 and difference image C5. Even if such first image, second image, third image, and fourth image are used, the change part can be detected in the same manner as in the first embodiment.

In the example of FIG. 8, image A6 corresponds to the first image, image A7 corresponds to the second image, and image B6 corresponds to the third and fourth images. That is, the third and fourth images are the same image. The third date and time are the same as the fourth date and time. Image B6 includes the road 31 and the white line 30, but does not include the shadow 32. Images A6 and A7 include the road 31, the white line 30, and the shadow 32. Difference image C6 shows the difference between image A6 and image B6. Difference image C7 shows the difference between image A7 and image B6. And common component image D3 is an image showing the common component between difference image C6 and difference image C7. Even if such first, second, third, and fourth images are used, the change portion can be detected in the same manner as in the first embodiment.

In this embodiment, each image stored in the memory unit 200 is associated with information indicating the presence or absence of short-term changes. The acquisition unit 120 can select and acquire an image that does not have a short-term change. When the acquisition unit 120 acquires an image that does not have a short-term change, the difference image generation unit 140 can use that image to generate multiple difference images. Furthermore, the difference image generation unit 140 may use that image to generate all difference images.

Examples of images without short-term changes include images taken when the sun is not out or on a cloudy day. Images stored in the memory unit 200 can be checked in advance and have information added to them indicating the presence or absence of short-term changes. Each image may also be added with information indicating the presence or absence of short-term changes based on the date and time it was taken.

As described above, according to this embodiment, the same actions and effects as those of the first embodiment can be obtained. In addition, the first image and the second image are the same image that does not show any short-term changes on the road, or the third image and the fourth image are the same image that does not show any short-term changes on the road. Therefore, it is possible to detect the changed portion using fewer images.

Third Embodiment
9 is a block diagram illustrating a functional configuration of an information processing device 10 according to a third embodiment. The information processing device 10 according to this embodiment is the same as the information processing device 10 according to at least one of the first and second embodiments, except for the points described below.

The information processing device 10 according to this embodiment further includes a determination unit 110 that determines whether or not to start the detection process. When the determination unit 110 determines to start the detection process, the detection process is started. As described above, the detection process is a series of processes for detecting a change portion using the images acquired by the acquisition unit 120. Specifically, the acquisition unit 120 acquires a first image, a second image, a third image, and a fourth image, the difference image generation unit 140 generates a first difference image and a second difference image, and the extraction unit 160 extracts the first common component. The detection process includes an acquisition step S10, a difference image generation step S20, and an extraction step S30. This will be described in detail below.

The determination unit 110 determines the timing for performing the detection process. In particular, the determination unit 110 determines whether or not to start the detection process so that the detection process is performed at a timing when there is a relatively high possibility that a change has occurred in the target. Below, first to fourth examples of the determination method performed by the determination unit 110 are described. Note that the determination method performed by the determination unit 110 is not limited to the following. Also, the information processing device 10 may execute a combination of multiple determination methods.

<First Example>
In the first example, the determination unit 110 determines that the detection process is to be started each time a predetermined period has elapsed. The determination unit 110 stores the date and time when the detection process was most recently performed. Then, the determination unit 110 calculates the elapsed time since the detection process was most recently performed every predetermined time (for example, every day). Then, if the calculated elapsed time is longer than the predetermined period, the determination unit 110 determines that the detection process is to be started. On the other hand, if the calculated elapsed time is shorter than the predetermined period, the determination unit 110 determines that the detection process is not to be started. The determination unit 110 can make a determination for each position. If it is determined that the detection process is to be started, the position is designated as the position of the detection target. In addition, it is preferable that a predetermined period including the timing of the most recent detection process is set as the first period, and a predetermined period ending at the timing when the calculated elapsed time becomes longer than the predetermined period is set as the second period. According to this example, the detection process can be performed periodically.

<Second Example>
In the second example, the first image, the second image, the third image, and the fourth image are all images including a road. The determination unit 110 acquires information about the road. Then, when the determination unit 110 determines that a predetermined event has occurred based on the information about the road, the determination unit 110 determines to start the detection process.

The information relating to the road is, for example, road construction information provided by a road management company. The acquisition unit 120 can acquire the information relating to the road, for example, from a server 50 of a service that provides road information. The predetermined event is an event that is highly likely to cause a change in the target, for example, road construction. Since road construction is highly likely to cause changes in the white lines on the road, it is preferable to perform a detection process to check whether any changes have occurred before and after the road construction.

The information about the road includes the date and time or period when the event occurs, and the location where the event occurs. The information processing device 10 can determine the location and time of the detection target based on the information about the road. For example, the acquisition unit 120 acquires a first image and a second image taken before the event occurs, and a third image and a fourth image taken after the event occurs. That is, the first time is specified before the event occurs, and the second time is specified after the event occurs. In addition, the location where the event occurs is specified as the location of the detection target.

In this example, the determination unit 110 acquires information about roads at predetermined time intervals (for example, once a day). Then, it checks whether a predetermined event has occurred after the most recent detection process. If a predetermined event has occurred, the determination unit 110 determines that the detection process should be started. On the other hand, if a predetermined event has not occurred, it determines that the detection process should not be started. According to this example, it is possible to detect the occurrence of an event that is likely to cause a change in the target and perform the detection process.

<Third Example>
In the third example, the first image, the second image, the third image, and the fourth image are all images including a road. The determination unit 110 obtains information about the traffic volume of the road, and determines the timing to start the detection process based on the information about the traffic volume.

In this example, the determination unit 110 obtains information about traffic volume from, for example, a server 50 of a service that provides traffic information. The information about traffic volume is, for example, information indicating traffic volume at each position on the road at each time. The determination unit 110 uses the information about traffic volume to calculate an integrated value of traffic volume at each position on the road. The integrated value may be an integrated value after the most recent detection process. The determination unit 110 then determines whether the calculated integrated value has exceeded a predetermined value. If the integrated value has exceeded the predetermined value, the determination unit 110 determines that the detection process should be started. On the other hand, if the integrated value has not exceeded the predetermined value, the determination unit 110 determines that the detection process should not be started. The determination unit 110 can make a determination for each position. Then, if it is determined that the detection process should be started, the position is designated as the position to be detected. In addition, it is preferable that a predetermined period including the timing of the most recent detection process is set as the first period, and a predetermined period ending at the timing when the integrated value has exceeded the predetermined value is set as the second period.

The greater the traffic volume, the greater the likelihood that paint on the road will peel off. Therefore, by determining the timing of the detection process based on traffic volume, changes can be detected accurately.

<Fourth Example>
In the fourth example, the first image, the second image, the third image, and the fourth image are all images including a road. The determination unit 110 obtains information about a traffic flow on the road, and determines the timing to start the detection process based on the information about the traffic flow.

In this example, the determination unit 110 obtains information about traffic flow from, for example, a server 50 of a service that provides traffic information. The information about traffic flow is, for example, information indicating the state of traffic flow at each branch point (three-way intersection, intersection, five-way intersection, etc.) on the road at each time. The information about traffic flow can be information indicating the number or percentage of vehicles that pass a branch point within a specified time for each traveling direction (straight ahead, right turn, etc.). For example, if an intersection where right-turning vehicles once existed becomes a place where only straight-line vehicles can pass at a certain time, it is possible that the traffic regulations have changed, that is, the markings on the road surface have changed. The information about traffic flow can also be information indicating the state of the flow of vehicles on the road. For example, if the number of vehicles in a vehicle line changes before and after a certain point in time, it is possible that the number of lanes has changed, that is, the markings on the road surface have changed.

The determination unit 110 monitors information about traffic flow at a predetermined interval and detects changes in traffic flow. Specifically, the determination unit 110 compares information about traffic flow when the most recent detection process was performed with information about the latest traffic flow. The determination unit 110 then determines to start the detection process when the difference between the information about traffic flow when the most recent detection process was performed and the information about the latest traffic flow is greater than a predetermined reference value. On the other hand, if the difference is not greater than the predetermined reference value, the determination unit 110 determines not to start the detection process. Alternatively, the determination unit 110 calculates the similarity between the information about traffic flow when the most recent detection process was performed and the information about the latest traffic flow, and determines to start the detection process when the calculated similarity is smaller than a predetermined reference value. On the other hand, if the similarity is not smaller than the predetermined reference value, the determination unit 110 determines not to start the detection process.

More specifically, the determination unit 110 may compare the proportion of straight-moving vehicles at each branch point based on information about traffic flow. In this case, the determination unit 110 determines to start the detection process if the difference between the proportion of straight-moving vehicles when the most recent detection process was performed and the latest proportion of straight-moving vehicles is greater than a predetermined reference value. On the other hand, if this difference is not greater than the predetermined reference value, the determination unit 110 determines not to start the detection process. The determination unit 110 may also detect and compare the number of lanes at each position on the road using information about traffic flow. In this case, the determination unit 110 determines to start the detection process if the number of lanes when the most recent detection process was performed is different from the latest number of lanes. On the other hand, if these numbers are the same, the determination unit 110 determines not to start the detection process.

The determination unit 110 can perform the determination for each position or branch point. Then, when it is determined that the detection process should be started, the position or branch point is designated as the position to be detected. In addition, it is preferable to set a predetermined period including the timing of the most recent detection process as the first time period, and set a predetermined period ending at the timing when the latest traffic flow information used in the determination was obtained as the second time period.

The hardware configuration of the computer that realizes the information processing device 10 according to this embodiment is shown, for example, in FIG. 6, similar to the first embodiment. However, the storage device 1080 of the computer 1000 that realizes the information processing device 10 according to this embodiment further stores a program module that realizes the functions of the determination unit 110 according to this embodiment.

Figure 10 is a flowchart illustrating the flow of an information processing method performed by the information processing device 10 according to this embodiment. The information processing method according to this embodiment further includes a judgment step S40 in which it is judged whether or not to start the detection process. Then, if it is judged in judgment step S40 that the detection process is to be started (Y in S40), the detection process is started. On the other hand, if it is not judged in judgment step S40 that the detection process is to be started (N in S40), the detection process is not started.

As described above, according to this embodiment, the same actions and effects as those of the first embodiment can be obtained. In addition, according to this embodiment, when the determination unit 110 determines that the detection process should be started, the detection process is started. Therefore, the detection process can be performed at an appropriate timing.

Fourth Embodiment
11 is a diagram for explaining the process performed by the information processing device 10 according to the fourth embodiment. The information processing device 10 according to this embodiment is the same as the information processing device 10 according to the third embodiment except for the points described below.

In the information processing device 10 according to this embodiment, the determination unit 110 acquires three or more images with different shooting dates and times, and extracts a common component between two temporally consecutive images among the three or more images as a second common component. The determination unit 110 detects whether or not there is a change in the object in the image acquired by the determination unit 110 by comparing the extracted second common components in chronological order. Then, when the determination unit 110 detects a change in the object in the image acquired by the determination unit 110, it determines that a detection process should be started. This is explained in detail below.

According to this embodiment, the determination unit 110 determines whether a change has occurred in the target based on multiple images. Then, if a change has occurred, the information processing device 10 performs a detection process and can extract the specific changed part.

In the example of FIG. 11, the determination unit 110 acquires images E1 to E6 that are taken at different dates and times. Each of the images E1 to E6 includes a road 31, a white line 30, and a shadow 32. In this example, the target is the white line 30. Images E1 to E6 are taken in this order. Images E1 to E6 may be taken on different days, for example, but the interval between images is not particularly limited. The determination unit 110 extracts the common component of each of two temporally consecutive images from images E1 to E6 as the second common component. Images F1 to F5 are images that show the second common component. That is, the determination unit 110 generates image F1 showing the common component between images E1 and E2, image F2 showing the common component between images E2 and E3, image F3 showing the common component between images E3 and E4, image F4 showing the common component between images E4 and E5, and image F5 showing the common component between images E5 and E6.

Then, the determination unit 110 detects whether or not there is a change in the object in images E1 to E6 by comparing images F1 to F5 in chronological order. In this example, there is no difference between images F1 and F2, but there is a difference between images F2 and F3 and between images F3 and F4. There is also no difference between images F4 and F5. In this case, it can be determined that a change occurred between the capture timing of image E3, which is the basis of image F3, and the capture timing of image E4. The determination by the determination unit 110 is also not affected by short-term changes such as shadows.

The determination unit 110 can make a determination using a minimum of three images. For example, when a similar process is performed using three images, images E1 to E3, it can be determined that no change has occurred between the shooting timing of image E1 and the shooting timing of image E3, since there is no difference between images F1 and F2. Furthermore, when a similar process is performed using three images, images E2 to E4, it can be determined that a change has occurred between the shooting timing of image E2 and the shooting timing of image E4, since there is a difference between images F2 and F3.

When the same process is performed using the four images E1 to E4, it can be determined that a change occurred between the capture timing of image E3 and the capture timing of image E4, since there is no difference between image F1 and image F2, but there is a difference between image F2 and image F3. In this way, when four or more images are used, the timing of the change can be grasped in more detail. Therefore, it is preferable for the determination unit 110 to obtain four or more images with different capture dates and times, and detect the presence or absence of a change and determine the timing of the change, rather than determining the timing of the change from only three images.

Furthermore, in this manner, when a change is detected in an object in an image acquired by the determination unit 110, the determination unit 110 can further identify the timing of the change. Therefore, it is preferable that the acquisition unit 120 acquires a first image and a second image taken before the timing of the change, and a third image and a fourth image taken after the timing of the change. In other words, it is preferable to specify the first time period before the timing of the change, and the second time period after the timing of the change. Furthermore, it is preferable that the information processing device 10 designates the shooting position of the image acquired by the determination unit 110 and used for the determination as the position of the detection object.

The determination unit 110 can obtain, for example, three or more images from the storage unit 200 via the acquisition unit 120. The images obtained by the determination unit 110 are similar to the images obtained by the acquisition unit 120 described in the first embodiment. It is preferable that the determination unit 110 obtains images similar to the images obtained by the acquisition unit 120 for the detection process and uses them to determine the presence or absence of change. That is, it is preferable that all images obtained by the determination unit 110 are orthoimages. It is also preferable that the images obtained by the determination unit 110 are images of the same type. It is preferable that the images obtained by the determination unit 110 have overlapping shooting areas. The determination unit 110 can detect the presence or absence of long-term changes in the target. Note that the target does not need to be determined in advance. The images obtained by the determination unit 110 may further include short-term changes. The three or more images obtained by the determination unit 110 are different from each other and are images taken at different dates and times.

When the determination unit 110 acquires images, it performs edge detection processing on each acquired image to obtain an edge image. The edge detection processing is as described above in the explanation of S201. Note that the determination unit 110 may perform correction processing, etc. on the acquired images as necessary.

Then, the determination unit 110 matches the positions of the two edge images obtained from the two temporally consecutive images in the same manner as described above in the explanation of S202. The determination unit 110 then compares the pixel values of the pixels at corresponding positions in the two edge images whose positions have been matched, and determines that pixels whose pixel values match are in a common portion. In this way, an image in which the pixels determined to be in a common portion are displayed so that they can be distinguished from the other pixels is an image that shows the second common component.

When the determination unit 110 generates multiple images showing the second common component from the acquired images, it compares them in chronological order to detect the presence or absence of a change. Specifically, the determination unit 110 performs a process to extract the difference between two images showing the second common component that are consecutive in time, and if a difference is extracted, it determines that there is a change between the two images. On the other hand, if no difference is extracted, it determines that there is no change between the two images. The process of extracting the difference between images showing the second common component can be performed in the same way that the difference image generation unit 140 generates a difference image from an edge image. Note that the determination unit 110 may determine that no difference has been extracted if the proportion of the number of pixels showing a difference in the difference image is equal to or less than a predetermined proportion.

When it is determined that there is a change between any two images that show the second common component, the determination unit 110 can determine that there is a timing of the change at least within the shooting period of the three or more images acquired by the determination unit 110. When the determination unit 110 determines that there is a timing of the change within the shooting period of the three or more images acquired by the determination unit 110, it determines to start the detection process. Then, the detection process is performed by the information processing device 10.

When the determination unit 110 acquires four or more images, i.e., when three or more images showing the second common component are generated, the determination unit 110 may further specify the timing of the change in detail. When images showing multiple second common components are compared and there is an image that is different from both the previous and next images, it is possible to specify that the timing of the change is between the two images that are the basis of that image. Also, when images showing multiple second common components are compared and the first or last image is different from the other images in terms of time, it is possible to specify that the timing of the change is between the two images that are the basis of that first or last image.

As described above, according to this embodiment, the same actions and effects as those of the first embodiment can be obtained. In addition, according to this embodiment, the determination unit 110 detects whether or not there is a change in the object in the image acquired. Then, when the determination unit 110 detects a change in the object in the image acquired by the determination unit 110, the determination unit 110 determines that the detection process should be started. Therefore, the detection process can be performed at an appropriate timing.

The above describes the embodiments and examples with reference to the drawings, but these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

REFERENCE SIGNS LIST 10 Information processing device 30 White line 31 Road 32 Shadow 50 Server 110 Determination unit 120 Acquisition unit 140 Difference image generation unit 160 Extraction unit 180 Output unit 200 Storage unit 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input/output interface 1120 Network interface

Claims (16)

an acquisition unit that acquires a first image captured on a first date and time, a second image captured on a second date and time, a third image captured on a third date and time, and a fourth image captured on a fourth date and time;
a difference image generating unit that generates a first difference image indicating a difference between the first image and the third image, and a second difference image indicating a difference between the second image and the fourth image;
an extracting unit that extracts a common component between the first difference image and the second difference image as a first common component,
The information processing device, wherein the third date and time and the fourth date and time are both later than the first date and time and later than the second date and time. 2. The information processing device according to claim 1,
An information processing device, wherein an interval between the first date and time and the second date and time is narrower than an interval between the first date and time and the third date and time. 3. The information processing device according to claim 1,
An information processing device in which an interval between the third date and time and the fourth date and time is narrower than an interval between the first date and time and the third date and time. In the information processing device according to any one of claims 1 to 3,
An information processing device, wherein the first image, the second image, the third image, and the fourth image are all orthoimages. In the information processing device according to any one of claims 1 to 4,
the first image, the second image, the third image, and the fourth image are all images including a road;
An information processing device, wherein the first image and the second image are the same image that does not show short-term changing objects on the road, or the third image and the fourth image are the same image that does not show short-term changing objects on the road. 6. The information processing device according to claim 5,
The short-term changing object is at least one of a shadow, snow, a puddle, soil, a vehicle, a fallen object, fallen leaves, and garbage. In the information processing device according to any one of claims 1 to 4,
The information processing device, wherein the first date and time and the second date and time are different from each other, and the third date and time and the fourth date and time are different from each other. In the information processing device according to any one of claims 1 to 7,
The information processing device further includes an output unit that outputs an image indicating the first common component. In the information processing device according to any one of claims 1 to 8,
a determination unit that determines whether or not to start a detection process in which the acquisition unit acquires the first image, the second image, the third image, and the fourth image, the difference image generation unit generates the first difference image and the second difference image, and the extraction unit extracts the first common component,
The information processing device starts the detection process when the determination unit determines to start the detection process. 10. The information processing device according to claim 9,
The information processing device, wherein the determination unit determines to start the detection process every time a predetermined period elapses. 11. The information processing device according to claim 9,
the first image, the second image, the third image, and the fourth image are all images including a road;
The determination unit is
Get information about roads,
determining, when it is determined that a predetermined event has occurred based on the information about the road, to start the detection process;
The acquisition unit is an information processing device that acquires the first image and the second image taken before the occurrence of the event, and the third image and the fourth image taken after the occurrence of the event. In the information processing device according to any one of claims 9 to 11,
the first image, the second image, the third image, and the fourth image are all images including a road;
The determination unit is
Obtaining information about road traffic flow,
An information processing device that determines the timing to start the detection process based on information about the traffic flow. In the information processing device according to any one of claims 9 to 12,
The determination unit is
Obtain three or more images with different shooting dates and times;
extracting a common component of each of two temporally consecutive images from among the three or more images as a second common component;
The determination unit detects whether or not there is a change in the object in the acquired image by comparing the extracted second common components in a time series manner;
The information processing device determines to start the detection process when a change is detected in the object in the image acquired by the determination unit. 14. The information processing device according to claim 13,
When a change is detected in the object in the image acquired by the determination unit, the determination unit further identifies a timing of the change;
The acquisition unit is an information processing device that acquires the first image and the second image captured before the timing of the change, and the third image and the fourth image captured after the timing of the change. an acquisition step of acquiring a first image taken at a first date and time, a second image taken at a second date and time, a third image taken at a third date and time, and a fourth image taken at a fourth date and time;
a difference image generating step of generating a first difference image indicating a difference between the first image and the third image, and a second difference image indicating a difference between the second image and the fourth image;
an extraction step of extracting a common component between the first difference image and the second difference image as a first common component,
The information processing method, wherein the third date and time and the fourth date and time are both later than the first date and time and later than the second date and time. A program for causing a computer to execute each step of the information processing method described in claim 15.

Images