JP2022154584A - Information processing device and information processing method - Google Patents

Abstract

To provide a method for obtaining information about a plurality of sorts of buildings and the like using a captured image.SOLUTION: An information processing device comprises: an acquisition unit that acquires an image captured by a camera mounted on a vehicle; a specifying unit that specifies an object to be processed based on the captured image; and an extracting unit that extracts a change point to be a target for detecting a change in the object to be processed from the captured image, wherein the extraction unit is configured to be able to execute a plurality of pieces of extraction processing, and extracts the change point of the object to be processed by performing the extraction processing corresponding to the object to be processed specified by the specifying unit.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information processing device and an information processing method.

2. Description of the Related Art Conventionally, in order to obtain information about structures such as buildings, there has been known a method of using a photographed image obtained by photographing a wide range. For example, Patent Literature 1 discloses a method of determining whether or not a building is an empty house by using a photographed image such as an aerial photograph image or a satellite image and map data.

International publication 2019/225597

The method of obtaining information about a subject using a photographed image may be used for various purposes, not limited to determining whether or not a building is an empty house. However, in order to obtain useful information, the process of analyzing the captured image must be specialized for the subject. For this reason, a device suitable for a specific purpose had to be utilized, as described in US Pat.
SUMMARY OF THE INVENTION It is an object of the present invention to provide a method of obtaining information about a plurality of types of buildings using photographed images.

One aspect for achieving the above object is an acquisition unit that acquires an image captured by a camera mounted on a vehicle, an identification unit that identifies an object to be processed based on the captured image, and an object to be processed from the captured image. an extraction unit for extracting change points from which change in an object is to be detected, wherein the extraction unit is configured to be capable of executing a plurality of extraction processes, and corresponds to the processing object specified by the specifying unit. The information processing apparatus extracts a change point of the object to be processed by performing the extraction process.

In the above information processing system, the extraction unit is capable of executing a plurality of the extraction processes in which the change point in the object to be processed and the mode of change detected for the change point are different, and the extraction process includes: A configuration may be adopted in which a mode of change of the change point in the processing target is detected by specifying and executing the extraction process corresponding to the processing target specified by the specifying unit.

In the above information processing system, in the extraction process, the extraction unit extracts the change point extracted from the first captured image and the second captured image captured at a different time from the first captured image. A configuration may be employed in which a change at the change point is detected by comparing the change point extracted from the image.

In the above information processing system, when the identifying unit identifies a house as the object to be processed, the extracting unit extracts a change point from an image of a side surface of the house included in the photographed image, thereby extracting the house. may be configured to determine whether the house is inhabited or vacant.

In the above information processing system, the extracting unit extracts the change point extracted from the side image of the house included in the photographed image taken at night and the house contained in the photographed image taken during the daytime. It may be determined whether the house is inhabited or vacant by comparing with the change point extracted from the side image.

In the above information processing system, the extracting unit extracts from the image of the side of the house the presence or absence of people inside or near the house, the lighting of the house based on the change point extracted from the photographed image. At least one of a lighting state, a state of the laundry being hung in the residence, an open/closed state of the windows of the residence, a temperature change in a part or the entirety of the residence, and a state of mail delivered to the residence. It may be determined whether the house is inhabited or vacant by detecting .

In the above information processing system, when the specifying unit specifies a traffic light as the object to be processed, the extracting unit extracts a change point from the image of the traffic light included in the captured image to determine the operation of the traffic light. It may be configured to determine at least one of whether or not the state is normal and whether or not a foreign object is attached to the traffic signal.

In the above information processing system, when a road is specified as the object to be processed by the specifying unit, the extracting unit extracts a change point from the image of the road included in the captured image, thereby extracting a road surface of the road. A configuration for detecting an abnormality in the state may be employed.

In the above information processing system, the acquisition unit acquires a moving image captured by the camera and position information of the vehicle during capturing of the moving image, and cuts out a still image from the moving image to obtain the captured image. is obtained, position information corresponding to the still image is generated based on the position information of the vehicle, and processing is performed to associate the generated position information with the captured image, and the extraction unit extracts from the captured image the The position information associated with the captured image may be output together with the result of the process of extracting the change point.

Another aspect for achieving the above object is to obtain an image captured by a camera mounted on a vehicle, identify a processing target based on the captured image, and detect changes in the processing target from the captured image. By performing the extraction process corresponding to the identified processing object among a plurality of extraction processes set to be executable, the extraction process extracting a change point to be detected is performed. This is an information processing method for extracting change points.

According to the present invention, by performing extraction processing corresponding to the processing target specified in the captured image, it is possible to obtain information on a plurality of types of processing target based on the captured image of the camera mounted on the vehicle. can.

1 is an explanatory diagram showing an overview of an information processing system; FIG. 1 is a configuration diagram of an information processing system; FIG. 4 is a flowchart showing the operation of the information processing device; 4 is a flowchart showing the operation of the information processing device; 4 is a flowchart showing the operation of the information processing device; 4 is a flowchart showing the operation of the information processing device; 4 is a flowchart showing the operation of the information processing device; 4 is a flowchart showing the operation of the information processing device; 4 is a flowchart showing the operation of the information processing device;

[1. Information processing system overview]
FIG. 1 is an explanatory diagram showing an outline of an information processing system 1 of this embodiment.
The information processing system 1 is a system that obtains information on the state of the road on which the vehicle V travels and the structures and the like located around the road, based on the images captured by the camera 205 mounted on the vehicle V. The information processing system 1 includes an information processing server 100 and an in-vehicle device 200 mounted on a vehicle V. FIG. The information processing server 100 corresponds to an example of an information processing device.

The in-vehicle device 200 has a camera 205 . The camera 205 is a digital camera that captures an angle of view including the road while the vehicle V is running, and may be a video camera that captures moving images or a still camera. Also, the in-vehicle device 200 may be equipped with an infrared camera 206 . The infrared camera 206 is an infrared thermography camera, and an image captured by the infrared camera 206 indicates the surface temperature of the subject.

The angle of view of the camera 205 and the infrared camera 206, that is, the imaging range, includes, for example, a road 530, a house 501 built beside the road 530, a traffic light 510 installed on the road 530, a sidewalk 520, and a vehicle parked on the sidewalk 520. Included are wheeled bicycles 540 and the like. These buildings, objects, people, etc. can be subjects of the camera 205 and the infrared camera 206 .

The in-vehicle device 200 has a wireless communication function such as cellular communication, and transmits photographed data D including images photographed by the camera 205 and/or the infrared camera 206 to the information processing server 100 . The information processing system 1 and the information processing server 100 are connected, for example, through a communication network NW so as to be able to communicate data with each other.

The number of vehicles V and in-vehicle devices 200 included in the information processing system 1 is not limited. Although one vehicle V is shown in FIG. 1 , this is an example, and the information processing server 100 can be configured to acquire captured images from a plurality of in-vehicle devices 200 . Further, although an example in which the vehicle V is a four-wheeled vehicle will be described below, the type of the vehicle V is not particularly limited, and may be a large vehicle, a commercial vehicle, a two-wheeled vehicle, a three-wheeled vehicle, or the like.

The communication network NW may be any data communication network, such as a wide area communication network including a public line network, a LAN (Local Area Network), and the like.

The information processing server 100 receives the photographed data D transmitted by the in-vehicle device 200 and analyzes the photographed images included in the photographed data D to obtain information regarding the house 501, the traffic light 510, the sidewalk 520, the road 530, and the like. . In the present embodiment, the information processing server 100 determines whether or not the house 501 is vacant, whether the traffic light 510 is operating or has an external appearance abnormality, and whether the sidewalk Whether or not there is a bicycle 540 left on the road 520, whether or not there is an abnormality on the road 530, and the like are determined.

The information processing server 100 of the present embodiment is connected to the distribution server 300 via the communication network NW so as to be capable of data communication. The information processing server 100 transmits the determination result based on the captured image to the distribution server 300 .

The distribution server 300 distributes information to the user terminal 250 by communicating with the user terminal 250 . The user terminal 250 is a mobile device or a portable computer called a handheld computer, such as a smart phone.

The distribution server 300 uses functions of an application program (hereinafter referred to as an application) installed in the user terminal 250 to provide information to the user terminal 250 through the communication network NW. Applications installed in the user terminal 250 are, for example, local applications intended for municipalities or narrower areas and used for information exchange in local communities and information exchange with administrative agencies. The local application is installed in the user terminal 250 personally used by local residents and in a computer (not shown) used by an administrative agency. The distribution server 300 acquires information transmitted from the information processing server 100, and distributes the information to the user terminal 250 or computer in which the local application is installed.

The information processing server 100 may be composed of one server computer, may be composed of a plurality of computers, or may be a cloud-type server. The distribution server 300 is also the same. Also, the information processing server 100 is not limited to a server as long as it can acquire a captured image from the in-vehicle device 200 . It may be a computer such as a desktop computer or a notebook computer.

[2. Configuration of information processing system]
FIG. 2 is a configuration diagram of the information processing system 1, and shows configurations of the information processing server 100 and the in-vehicle device 200 that configure the information processing system 1. As shown in FIG.

First, the configuration of the in-vehicle device 200 will be described.
The in-vehicle device 200 has a processor 201 , a storage unit 202 , a communication unit 203 , a touch panel 204 , a camera 205 and a GNSS 207 .

The processor 201 is an information processing device such as a CPU (Central Processing Unit) or a microcontroller. The storage unit 202 may be a non-volatile storage device configured using a semiconductor memory device such as a flash memory. The storage unit 202 may have a volatile storage area and may be configured to form a work area for the processor 201 . The storage unit 202 stores programs executed by the processor 201 and data processed by the processor 201 .
The processor 201 executes programs stored in the storage unit 202 and controls the in-vehicle device 200 .

The communication unit 203 is a communication device that includes an antenna, an RF circuit, etc., and executes cellular communication through a mobile communication line. The communication unit 203 connects to the communication network NW and performs data communication with the information processing server 100 under the control of the processor 201 .

A touch panel 204 displays various images under the control of the processor 201 . The touch panel 204 detects a touch operation on the touch panel 204 and outputs operation data indicating the detected operation to the processor 201 . The touch panel 204 may be a touch panel of a car navigation system mounted on the vehicle V, for example.

The camera 205 is installed, for example, on the windshield of the vehicle V or in the vicinity thereof, and photographs the front of the vehicle V in the traveling direction. Also, the camera 205 may be a camera that captures the rear of the vehicle V. FIG. The camera 205 may also include a camera that captures the front of the vehicle V and a camera that captures the rear of the vehicle. The camera 205 performs photographing under the control of the processor 201 and outputs the photographed image to the processor 201 .

The infrared camera 206 performs photographing under the control of the processor 201 and outputs a thermographic image to the processor 201 . As described above, the in-vehicle device 200 may be configured without the infrared camera 206 .

A GNSS (Global Navigation Satellite System) 207 detects the position of the in-vehicle device 200 under the control of the processor 201 .

Processor 201 performs photography with camera 205 and infrared camera 206 . The processor 201 adds position information at the time of photography and information indicating the date and time of photography to the image captured by the camera 205 and the image captured by the infrared camera 206 to generate the image data D. FIG.

As shown in FIG. 2, the photographed data D includes a photographed image D1, a thermographic image D2, position information D3, and photographing date and time D4. The captured image D1 is data of a still image or moving image (video) captured by the camera 205 . A thermographic image D2 is thermography image data captured by the infrared camera 206 . In a configuration in which the in-vehicle device 200 does not include the infrared camera 206, the captured data D does not include the thermo image D2. The position information D3 is position information of the vehicle V when the in-vehicle device 200 captures the captured image D1, and is position information measured by the GNSS 207 . When the captured image D1 is moving image data, the position information D3 is position information at the start and/or the end of shooting the moving image, or a plurality of positions measured at predetermined intervals during the shooting of the moving image. Contains information. The shooting date and time D4 is the shooting date and time of the camera 205 . When the photographed image D1 is data of a moving image, the photographing date and time D4 is the date and time when the photographing of the moving image is started and/or when the photographing is finished, or a plurality of pieces of time information specified at a predetermined cycle during the photographing of the moving image. including.

The processor 201 transmits the photographed data D to the information processing server 100 through the communication unit 203 . The processor 201 performs photography with the camera 205 and the infrared camera 206 while the in-vehicle device 200 is activated or while the vehicle V is running, for example. Specifically, the camera 205 and the infrared camera 206 may shoot moving images all the time, or may shoot still images at preset intervals. In this case, the processor 201 temporarily stores data of images captured by the camera 205 and the infrared camera 206 , position information measured by the GNSS 207 , and the date and time of capturing in the storage unit 202 . The processor 201 generates the imaging data D and transmits it through the communication unit 203 at a preset cycle or when the in-vehicle device 200 stops operating.

The information processing server 100 includes a processor 10 , a storage section 20 and a communication section 30 . The processor 10 is an information processing device such as a CPU or microcontroller. The storage unit 20 stores programs executed by the processor 10 and data processed by the processor 10 . The storage unit 20 may be a nonvolatile storage device configured using a semiconductor memory device such as a flash memory, a magnetic storage device, an optical storage device, or the like. Moreover, the storage unit 20 may include a volatile storage area configured by a semiconductor memory device, and the work area of the processor 10 may be formed by this volatile storage area. Further, when the processor 10 is configured as an integrated device in which a volatile memory is integrated, the storage unit 20 is configured as a non-volatile storage device.

The storage unit 20 stores a control program 21 , moving image data 22 , photographed image data 23 , thermographic image data 24 , setting data 25 and processing result data 26 . The control program 21 is a program executed by the processor 10 to control the information processing server 100 .

The processor 10 implements the functions of the information processing server 100 by executing the control program 21 . As functional units configured by the processor 10, an acquisition unit 11, an identification unit 12, an extraction unit 13, and an output unit 14 are shown. Each of these functional units is realized by cooperation of software and hardware when the processor 10 executes the control program 21 .

The acquisition unit 11 acquires the photographed data D from the in-vehicle device 200 . When the photographed image D1 included in the photographed data D is data of a still image, the acquisition unit 11 associates the photographed image D1, the position information D3, and the photographing date and time D4, and stores the photographed image data 23 in the storage unit 20. . When the captured image D1 is moving image data, the acquisition unit 11 associates the captured image D1, the position information D3, and the shooting date and time D4, and stores them in the storage unit 20 as moving image data 22 . That is, the moving image data 22 and the photographed image data 23 each include the data of the photographed image of the camera 205, the position information, and the photographing date and time in association with each other. The acquisition unit 11 also causes the storage unit 20 to store the thermo image D2 as thermo image data 24 in association with the position information D3 and the photographing date and time D4.

The acquisition unit 11 also has a function of extracting data of a still image from the moving image data 22 and storing it as captured image data 23 .
The storage unit 20 can store captured image data 23 acquired from a plurality of in-vehicle devices 200 . That is, it is possible to store captured images captured by a plurality of vehicles V in the storage unit 20 . The acquisition unit 11 can acquire the photographed data D from the plurality of in-vehicle devices 200 and store the photographed image data 23 in the storage unit 20 .

Here, when acquiring the photographed data D, the acquiring unit 11 may acquire identification information for identifying the in-vehicle device 200 that has transmitted the photographed data D. FIG. This identification information may be given to the in-vehicle device 200 in advance and included in the photographed data D, or may be given by the acquisition unit 11 . In this case, the acquiring unit 11 associates the moving image data 22, the photographed image data 23, or the thermographic image data 24 with the identification information of the in-vehicle device 200, and causes the storage unit 20 to store them. Thereby, when the photographed image data 23 or the thermographic image data 24 are processed, the in-vehicle device 200 that photographed them can be specified.

The specifying unit 12 analyzes the photographed image data 23 and detects a processing object appearing as a subject in the photographed image data 23 . In this embodiment, the identification unit 12 detects the house 501, the traffic light 510, the sidewalk 520, and the road 530 from the photographed image data 23 as objects to be processed. The identifying unit 12 identifies which of the house 501, the traffic light 510, the sidewalk 520, and the road 530 is the detected object to be processed.

The extraction unit 13 extracts change points of the processing object from the image of the processing object specified by the specifying unit 12 . A change point is a point, a portion, or an area for which a change in a processing object is to be detected, and may be the entire processing object. In other words, the change point is a focus point focused on in the process of detecting change in the processing object. Here, the change in the object to be processed is, for example, change over time. Changes over time are not limited to mere deterioration over time, but include changes caused by natural events, artificial changes, other sudden changes, and the like. For example, changes caused by the behavior of residents of the house 501, adhesion of foreign matter to the traffic light 510, failure of the traffic light 510, objects left on the sidewalk 520, damage to the road surface of the road 530, and damage to the road 530. Flooding of the road surface is an example of a change in the object to be processed. The extraction unit 13 detects a change mode at a change point of the object to be processed. For example, when the object to be processed is a house 501, the extraction unit 13 detects a change in the window 502 of the house 501 as a change point. The extraction unit 13 determines whether or not the illumination of the house 501 has been turned on by detecting a change in the window 502 . Since the lighting of the house 501 has turned on, it can be determined that the house 501 is not vacant.

The change point of the object to be processed and the mode of change detected for the change point differ for each object to be processed. In other words, the extracting unit 13 detects the mode of change corresponding to the type of the processing object for the change point of the processing object set according to the type of the processing object.
For example, if the object to be processed is a house 501, the points of change are the exterior including the sides of the house 501, the mailbox of the house 501, and the window 502 of the house 501. The mode of change detected by the extraction unit 13 is the presence or absence of a person P around the house 501, the presence or absence of mail, the lighting state of the lighting appearing in the window 502, the presence or absence of laundry appearing in the window 502, and the opening/closing of the window 502. state.

The extraction unit 13 extracts a change point for each object to be processed and detects the mode of change. use. The first processing unit 131, the second processing unit 132, the third processing unit 133, and the fourth processing unit 134 are processing units corresponding to different processing objects. In each of these processing units, the change point corresponding to the processing object and the mode of change to be detected are defined. These processing units represent detection logic executed by the processor 10 through the functions of the control program 21 . The extraction unit 13 can select and execute extraction processing corresponding to the first processing unit 131 , the second processing unit 132 , the third processing unit 133 and the fourth processing unit 134 .

The extraction unit 13 selects a processing unit corresponding to the processing object identified by the acquisition unit 11 from among the first processing unit 131, the second processing unit 132, the third processing unit 133, and the fourth processing unit 134, and performs processing. to run. This corresponds to switching and executing the detection logic in which the change point and the mode of change to be detected are set according to the processing object specified by the acquisition unit 11 .

The processing unit of the extraction unit 13 uses the setting data 25 in the process of detecting the mode of change of the change point. The setting data 25 is, for example, image data indicating the normal state of the change point, or a threshold value for determination.
The extraction unit 13 causes the storage unit 20 to store the result of the processing performed by the processing unit as the processing result data 26 .

The output unit 14 outputs the result of processing by the extraction unit 13 . The output unit 14 transmits the processing result data 26 to the distribution server 300 via the communication unit 30, for example.

[3. Operation of information processing system]
FIG. 3 is a flowchart showing the operation of the information processing server 100. As shown in FIG. The operation of FIG. 3 is executed by the acquisition unit 11. FIG.
The acquisition unit 11 acquires the photographed data D from the in-vehicle device 200 (step S11). The acquisition unit 11 determines whether or not the captured image D1 is moving image data (step S12).

When the acquisition unit 11 determines that the captured image D1 is not moving image data, that is, is still image data (step S12; NO), the acquisition unit 11 associates the captured image D1 with the position information D3 and the shooting date and time D4. are stored as photographed image data 23 (step S13).

When the acquisition unit 11 determines that the captured image D1 is moving image data (step S12; YES), the acquisition unit 11 cuts out still image data from the captured image D1 (step S14). In step S1, the acquisition unit 11 extracts, for example, part or all of the frames that constitute the moving image data, and uses them as still image data.

Based on the position information D3 and the shooting date and time D4 added to the shot image D1, the obtaining unit 11 generates the position information and the shooting date and time corresponding to the still image cut out in step S14 (step S15). The acquisition unit 11 associates the data of the still image cut out in step S14, the position information, and the shooting date and time, and stores them in the storage unit 20 as the captured image data 23 (step S16).

The obtaining unit 11 also performs the operation of FIG. 3 for the thermo image D2. As a result, the thermo image D2 is stored in the storage unit 20 as the thermo image data 24. FIG.

By the operation of FIG. 3, even when the photographed data D transmitted by the in-vehicle device 200 includes moving image data, the identifying unit 12 and the extracting unit 13 can perform processing using still image data. Become. Here, the operation of FIG. 3 may be configured to be executed by the in-vehicle device 200 . That is, the processor 201 may generate still image data from moving images captured by the camera 205 and the infrared camera 206, generate captured data D including the still image data, and transmit the captured data D to the information processing server 100. . In this case, the photographed data D transmitted by the in-vehicle device 200 is data including the photographed image D1 and the thermographic image D2, which are still images.

4, 5, 6, 7, 8 and 9 are flowcharts showing the operation of the information processing server 100. FIG. Steps S21 to S23 in FIG. 4 are executed by the identification unit 12, steps S24 and S25 are executed by the extraction unit 13, and steps S26 and S27 are executed by the output unit 14. FIG.

As shown in FIG. 4, the information processing server 100 acquires the photographed image data 23 stored in the storage unit 20 (step S21). The information processing server 100 extracts an image of the subject from the captured image data 23 (step S22). In step S<b>22 , subjects that can be recognized as buildings, objects, or people in the photographed image data 23 are extracted regardless of whether they are objects to be processed by the extraction unit 13 .

The information processing server 100 identifies the object to be processed from among the subjects extracted in step S22 (step S23). The information processing server 100 selects the processing unit corresponding to the processing object identified in step S23 from among the first processing unit 131, the second processing unit 132, the third processing unit 133, and the fourth processing unit 134. (Step S24). The processing of step S24 corresponds to selecting the logic of the extraction processing corresponding to the processing object specified in step S23.

The information processing server 100 executes the extraction process by the processing unit selected in step S23 (step S25). The information processing server 100 determines whether or not the result of the extraction process should be output (step S26). When the information processing server 100 determines that the result should be output (step S26; YES), the information processing server 100 outputs the processing result data 26 to the distribution server 300 (step S27), and distributes it from the distribution server 300 to the user terminal 250 and the like. . When the information processing server 100 determines that the processing result data 26 is not the result to be output (step S26; NO), the processing ends without performing step S27.

5-9 are flow charts showing in detail step S23 of FIG. FIG. 5 shows the operation when the first processing unit 131 is selected in step S24. That is, the operation of FIG. 5 is executed by the first processing unit 131 .

The first processing unit 131 is selected when the object to be processed specified by the specifying unit 12 is the traffic light 510 .
The first processing unit 131 sets the change point in the image of the traffic light 510 to the lighting portion of the traffic light 510 (step S31). The lighting portion of the traffic light 510 is a portion that emits light in red, blue, and yellow by light from an LED (Light Emitting Diode) built in the traffic light 510 or light from a light bulb.

The first processing unit 131 extracts the image of the change point from the photographed image data 23 (step S32). The first processing unit 131 determines the lighting state (operating state) of the lighting unit by analyzing the image of the change point (step S33). Specifically, in step S33, the first processing unit 131 determines whether or not the lighting unit is normally lit. For example, when the image of the traffic light 510 when it is normally lit is stored in advance in the storage unit 20 as the setting data 25, the first processing unit 131 compares this image with the image extracted in step S32. By doing so, the determination in step S33 is performed. In step S<b>33 , the first processing unit 131 causes the storage unit 20 to store the determination result in the processing result data 26 .

Traffic light 510 usually has a plurality of lighting units. The first processing unit 131 may set a plurality of lighting portions as changing points in step S31, and determine the plurality of lighting portions in step S33. Alternatively, the first processing unit 131 may set any one of the plurality of lighting units as the point of change in step S31, and determine the point of change in step S33. In this case, the first processing unit 131 may repeat the operations of steps S31 to S33 by the number of lighting units.

The first processing unit 131 sets the change point in the image of the traffic light 510 to the lower end of the traffic light 510 (step S34). Here, the lower end portion of the traffic signal 510 is the lowest portion of the portion that does not include the pole pointing to the traffic signal 510, and is, for example, the lower portion of the lighting portion of the traffic signal 510. FIG.

The first processing unit 131 extracts the image of the change point from the photographed image data 23 (step S35). The first processing unit 131 determines whether or not a foreign object is attached to the traffic signal 510 by analyzing the image of the change point (step S36). For example, when an image of the traffic light 510 when there is no foreign object is stored in the storage unit 20 in advance as the setting data 25, the first processing unit 131 compares this image with the image extracted in step S35. , the determination of step S36 is performed. For example, the first processing unit 131 determines whether or not an icicle adheres to the traffic light 510 in step S33. In step S<b>36 , the first processing unit 131 stores the determination result in the processing result data 26 in the storage unit 20 .

As shown in FIG. 5, the extracting unit 13 can set a plurality of change points for one processing object, and can analyze the captured image for each change point and perform determination.

6 and 7 show the operation when the second processing unit 132 is selected in step S24. That is, the operations of FIGS. 6 and 7 are executed by the second processing unit 132 .

The second processing unit 132 is selected when the object to be processed specified by the specifying unit 12 is the house 501 .
The second processing unit 132 acquires a plurality of captured image data 23 (step S41). If the captured image data 23 acquired in step S21 is one captured image, the second processing unit 132 acquires the captured image data 23 captured at a date and time different from that of this captured image. Moreover, when the photographed image data 23 obtained in step S21 includes a plurality of photographed images, these photographed images may be obtained in step S41.

In the following description, one of the captured images acquired in step S41 will be referred to as the first image, and one of the other captured images will be referred to as the second image. It is preferable that the first image and the second image are images taken on different dates and times. More preferably, the date and time of photography, that is, the timing of photography, are different by several hours or more. More preferably, the first image is an image taken at night and the second image is an image taken during daytime. . Also, the first image and the second image are images of the same house 501 photographed. The second processing unit 132 acquires a plurality of photographed images of the same house 501 based on the position information included in the photographed image data 23, for example. The first image and the second image may be images captured by cameras 205 mounted on different vehicles V. FIG. The first image corresponds to an example of a first captured image, and the second image corresponds to an example of a second captured image.

The second processing unit 132 sets the change point in the image of the house 501 to the appearance including the side surface of the house 501 (step S42). Since the photographed image data 23 is an image photographed by the camera 205 mounted on the vehicle V, it is an image photographed of the exterior including the sides of the house 501, and is not an image photographed from above such as an aerial image or a satellite image. is different.

The second processing unit 132 extracts images of change points from a plurality of captured images (step S43). The second processing unit 132 determines whether or not there is a person P near the house 501 by comparing the change points extracted from the first image and the change points extracted from the second image in step S43 (step S44). ).

The second processing unit 132 sets the change point in the image of the house 501 to the mailbox of the house 501 (step S45). The second processing unit 132 extracts images of change points from the plurality of captured images (step S46). The second processing unit 132 compares the points of change extracted from the first image and the points of change extracted from the second image in step S46, thereby determining whether or not mail is accumulated in the mailbox of the residence 501. (Step S47). For example, if the number of mail items in the mailbox increases over time, it can be determined that the mail items delivered to the residence 501 are accumulated. Also, if the number of mail items in the mailbox decreases or disappears over time, it can be determined that the mail items delivered to the house 501 have been collected by the residents of the house 501, that is, the mail items have not accumulated.

The second processing unit 132 sets the change point in the image of the house 501 to the window 502 (step S48). The second processing unit 132 extracts images of change points from the plurality of captured images (step S49). The second processing unit 132 determines the lighting state of the lighting of the house 501 by comparing the change points extracted from the first image and the change points extracted from the second image in step S49 (step S50). If either the first image or the second image is an image taken at night, the determination can be made more reliably.

Further, the second processing unit 132 determines the presence or absence of laundry hanging in the house 501 by comparing the change points extracted from the first image and the change points extracted from the second image in step S49. (Step S51). If either the first image or the second image is an image taken in the daytime, the determination can be made more reliably.

Further, the second processing unit 132 determines the open/closed state of the window 502 by comparing the change point extracted from the first image in step S49 and the change point extracted from the second image (step S52). The open/closed state of the window 502 indicates, for example, whether the window 502 is opened or closed.

The second processing unit 132 determines whether or not the house 501 is vacant based on the determination results of steps S44, S47, S50, S51, and S52, and stores the determination result in the storage unit 20 as the processing result data 26. (step S53).
In the above process, the second processing unit 132 may determine whether or not there is a person P inside the house 501 appearing through the window 502 when the change point is set to the window 502 . In this case, the determination result of whether or not the person P is present in the room of the house 501 may be used for determination in step S53.

As shown in FIG. 6, the extraction unit 13 can perform a plurality of determinations based on one change point set for one processing object. Also, it can be determined whether the house 501 is vacant or inhabited based on the results of a plurality of determinations.

FIG. 7 shows the operation when the second processing unit 132 extracts the change points from the thermo-image data 24 .
The second processing unit 132 acquires a plurality of thermo-image data 24 by the same process as in step S61 (step S61). In the following, one of the captured images acquired in step S61 will be referred to as a first thermo image, and one of the other captured images will be referred to as a second thermo image. It is preferable that the first thermo image and the second thermo image are images taken on different dates and times. More preferably, the date and time of photography, that is, the timing of photography, are different by several hours or more. More preferably, the first thermo-image is an image taken at night, and the second thermo-image is an image taken during the day. is. Also, the first thermo image and the second thermo image are images of the same house 501 photographed. The second processing unit 132 acquires a plurality of photographed images of the same house 501 based on the position information included in the photographed image data 23, for example. The first thermo image and the second thermo image may be images captured by cameras 205 mounted on different vehicles V. FIG. The first thermo image corresponds to an example of the first captured image, and the second thermo image corresponds to an example of the second captured image.

The second processing unit 132 sets the change point in the image of the house 501 to the window 502 (step S62). The second processing unit 132 extracts images of change points from a plurality of captured images (step S63). The second processing unit 132 determines the temperature change of the window 502 by comparing the change points extracted from the first thermo image in step S63 and the change points extracted from the second thermo image (step S64). The second processing unit 132 determines whether or not the house 501 is vacant based on the determination result of step S64, and stores the determination result as the processing result data 26 in the storage unit 20 (step S65). For example, if the temperature change in the window 502 differs from the temperature change in the area of the house 501, it is the basis for determining that the house 501 is not an empty house. Specifically, when the second processing unit 132 determines in step S64 that the temperature of the window 502 is higher at night than during the day, it can be determined that the house 501 is not vacant.

In step S65, the second processing unit 132 determines whether or not the house 501 is an empty house by combining the determination results of steps S44, S47, S50, S51, and S52 in FIG. 6 and the determination result of step S65. You may For example, in step S61, if the second processing unit 132 obtains thermographic image data 24 of the same house 501 as the photographed image data 23 obtained in step S41 based on the position information included in the thermographic image data 24, good. In this case, by combining the photographed image data 23 and the thermographic image data 24, it can be determined more accurately whether or not the house 501 is vacant.
Further, the second processing unit 132 may determine the temperature change with the change point as the entire appearance including the side surface of the house 501 in the process of FIG. 7 . Also in this case, it is possible to determine whether or not the house 501 is vacant based on the temperature change of the house 501 .

FIG. 8 shows the operation when the third processing unit 133 is selected in step S24. That is, the operation of FIG. 8 is executed by the third processing unit 133 .
The third processing unit 133 is selected when the processing object specified by the specifying unit 12 is the road 530 .

The third processing unit 133 sets the change point in the captured image of the road to the road surface of the road 530 (step S71). The third processing unit 133 extracts the image of the change point from the photographed image data 23 (step S72). The third processing unit 133 determines whether or not there is a depression in the road surface by analyzing the image of the change point (step S73). For example, when an image of a road surface with depressions is stored in advance in the storage unit 20 as the setting data 25, the third processing unit 133 compares this image with the image extracted in step S72 to A determination is made in step S73. In step S<b>73 , the third processing unit 133 stores the determination result in the processing result data 26 in the storage unit 20 .

The third processing unit 133 determines whether or not the road 530 is flooded by analyzing the image of the change point (step S74). For example, when an image of a flooded road surface is stored in advance in the storage unit 20 as the setting data 25, the third processing unit 133 compares this image with the image extracted in step S72. , the determination of step S74 is performed. In step S<b>74 , the third processing unit 133 causes the storage unit 20 to store the determination result in the processing result data 26 .
In this manner, the third processing unit 133 can determine the road surface condition of the road 530 using the captured image data 23 .

FIG. 9 shows the operation when the fourth processing unit 134 is selected in step S24. That is, the operation of FIG. 9 is executed by the fourth processing unit 134 .
The fourth processing unit 134 is selected when the object to be processed specified by the specifying unit 12 is the sidewalk 520 .

The fourth processing unit 134 acquires a plurality of captured image data 23 (step S81). When the captured image data 23 acquired in step S21 is one captured image, the fourth processing unit 134 acquires the captured image data 23 captured at a different date and time from this captured image. Moreover, when the photographed image data 23 acquired in step S21 includes a plurality of photographed images, these photographed images may be acquired in step S81.

Here, one of the captured images acquired in step S81 is defined as the first image, and one of the other captured images is defined as the second image. It is preferable that the first image and the second image are images taken on different dates and times. More preferably, the first image and the second image are images taken on different days. More preferably, the first image is an image taken at night and the second image is an image taken during the day. Also, the first image and the second image are images taken at the same place on the sidewalk 520 . The fourth processing unit 134 acquires a plurality of photographed images of the sidewalk 520 at the same location based on the position information included in the photographed image data 23, for example. The first image and the second image may be images captured by cameras 205 mounted on different vehicles V. FIG.

The fourth processing unit 134 sets the change point in the image of the sidewalk 520 to the sidewalk 520 itself (step S82). The fourth processing unit 134 extracts the image of the change point from the photographed image data 23 (step S83).

The fourth processing unit 134 determines the presence or absence of the bicycle 540 on the sidewalk 520 by analyzing the image of the change point (step S84). When the bicycle 540 is shown in both the first image and the second image, the fourth processing unit 134 determines that the bicycle 540 is present (step S84; YES). In this case, the fourth processing unit 134 sets the point of change to the bicycle 540 (step S85), and extracts the image of the point of change from each of the first image and the second image (step S86).

The fourth processing unit 134 compares the change points extracted from the first image and the change points extracted from the second image in step S86 to determine whether or not the bicycle 540 is left unattended. It is determined whether or not it is a bicycle (step S87). In step S<b>87 , the fourth processing unit 134 stores the determination result in the processing result data 26 in the storage unit 20 .

Further, in step S84, if the bicycle 540 is not shown in either the first image or the second image, the fourth processing unit 134 determines that the bicycle 540 is not present (step S84; NO). In this case, the fourth processing unit 134 determines that there is no parked bicycle (step S88), and stores the determination result in the processing result data 26 in the storage unit 20. FIG.

[4. Other embodiments]
The above embodiment shows a specific example to which the present invention is applied, and does not limit the form to which the invention is applied.

For example, in the above-described embodiment, the determination results of steps S44, S47, S50, S51, and S52 are used to determine whether the house 501 is occupied or vacant. Not required. The extraction unit 13 may make a determination in step S53 using at least one of the determination results of steps S44, S47, S50, S51, and S52.

For example, in the above-described embodiment, an example in which the photographed data D is transmitted from the in-vehicle device 200 to the information processing server 100 through cellular communication or the like has been described, but the in-vehicle device 200 does not have to have a cellular communication function. For example, the in-vehicle device 200 may store the photographic data D in a portable information storage medium connectable to the in-vehicle device 200, and the information processing server 100 may read the photographic data D from the information storage medium.

FIG. 2 is a schematic diagram showing the functional configurations of the information processing server 100 and the in-vehicle device 200, divided according to main processing contents, to facilitate understanding of the present invention. 200 configuration is not limiting. For example, the processing of the components included in the processor 10 may be executed by one hardware unit or may be executed by a plurality of hardware units. Also, each process shown in FIGS. 3 to 9 may be executed by one program or may be executed by a plurality of programs.

The control program 21 can also be realized by recording the control program 21 on a portable information recording medium. Examples of information recording media include magnetic recording media such as hard disks, optical recording media such as CDs, and semiconductor storage devices such as USB (Universal Serial Bus) memories and SSD (Solid State Drives). It is also possible to use The processor 10 may read and execute the control program 21 from the information recording medium.

[5. Configuration supported by the above embodiment]
The above embodiment supports the following configurations.

(Configuration 1) an acquisition unit that acquires an image captured by a camera mounted on a vehicle, an identification unit that identifies a processing target based on the captured image, and a target that detects changes in the processing target from the captured image and an extraction unit configured to be capable of executing a plurality of extraction processes, and performing the extraction process corresponding to the object to be processed specified by the specifying unit An information processing device that extracts a change point of the object to be processed by.
According to the information processing apparatus of configuration 1, by performing extraction processing corresponding to the processing target specified in the captured image, information on a plurality of types of processing targets is obtained based on the captured image of the camera mounted on the vehicle. can be obtained.

(Arrangement 2) The extraction unit is capable of executing a plurality of extraction processes in which the change point in the object to be processed and the mode of change detected for the change point are different, and among the plurality of extraction processes, the The information processing apparatus according to configuration 1, wherein a mode of change of the change point in the processing target is detected by specifying and executing the extraction process corresponding to the processing target specified by the specifying unit.
According to the information processing apparatus of the configuration 2, it is possible to detect the mode of change at the change point peculiar to the object to be processed by the extraction process corresponding to the object to be processed. Accordingly, it is possible to obtain information specific to the type of the object to be processed by using the captured image of the camera mounted on the vehicle.

(Configuration 3) In the extraction process, the extraction unit extracts the change point extracted from the first captured image and the second captured image captured at a different time from the first captured image. The information processing apparatus according to configuration 1 or 2, wherein a change at the change point is detected by comparing the extracted change point.
According to the information processing apparatus of configuration 3, detailed information regarding the object to be processed can be obtained by comparing a plurality of images.

(Configuration 4) When the identifying unit identifies a house as the object to be processed, the extracting unit extracts a change point from an image of the side surface of the house included in the photographed image to determine whether the house is habitable. The information processing apparatus according to any one of configurations 1 to 3, which determines whether the house is inside or vacant.
According to the information processing apparatus of configuration 4, by using the image of the side of the house, it is possible to more accurately determine whether the house is inhabited or vacant, compared to the case of using an image taken from above, for example. can.

(Arrangement 5) The extracting unit extracts the change point extracted from the image of the side of the house included in the captured image taken at night and the side of the house included in the captured image taken during the daytime. The information processing apparatus according to configuration 4, wherein it is determined whether the house is inhabited or vacant by comparing with the change point extracted from the image.
According to the information processing apparatus of configuration 5, since the photographed image photographed at night and the photographed image photographed during the daytime are compared, it can be determined more accurately whether the house is occupied or vacant.

(Configuration 6) The extracting unit extracts the presence or absence of a person inside or near the house from the image of the side of the house based on the change point extracted from the photographed image, and the lighting state of the lighting of the house. , the state of the laundry hanging to dry in the residence, the open/closed state of the windows of the residence, the temperature change in a part or the whole of the residence, and the state of the mail delivered to the residence. The information processing apparatus according to configuration 4 or 5, wherein it is determined whether the house is inhabited or vacant by doing so.
According to the information processing apparatus of configuration 6, it is possible to more accurately determine whether a house is inhabited or vacant by making a determination on an event that serves as an indicator of the living condition of the house.

(Arrangement 7) When the specifying unit specifies a traffic light as the object to be processed, the extracting unit extracts a change point from the image of the traffic light included in the captured image, thereby determining the operation state of the traffic light. 3. The information processing apparatus according to any one of configurations 1 to 3, wherein it is determined at least one of whether the signal is normal and whether a foreign object is attached to the traffic signal.
According to the information processing device of configuration 7, it is possible to obtain information about the state of the traffic light based on the image captured by the camera mounted on the vehicle.

(Arrangement 8) When the specifying unit specifies a road as the object to be processed, the extracting unit extracts a change point from the image of the road included in the photographed image to obtain the road surface condition of the road. The information processing device according to any one of Configurations 1 to 3, which detects an abnormality.
According to the information processing device of configuration 8, information regarding the state of the road surface can be obtained based on the image captured by the camera mounted on the vehicle.

(Arrangement 9) The acquisition unit acquires the moving image captured by the camera and the position information of the vehicle during the capturing of the moving image, and acquires the captured image by extracting a still image from the moving image. and generating position information corresponding to the still image based on the position information of the vehicle, and performing processing for associating the generated position information with the captured image. 9. The information processing apparatus according to any one of configurations 1 to 8, wherein position information associated with the photographed image is output together with a result of the process of extracting .
According to the information processing apparatus of configuration 9, it is possible to obtain information on a plurality of types of processing objects by using moving images captured by a camera mounted on a vehicle.

(Arrangement 10) Acquiring an image captured by a camera mounted on a vehicle, identifying a processing target based on the captured image, and extracting a change point from which a change in the processing target is detected from the captured image. an information processing method for extracting a change point of the processing object by performing the extraction processing corresponding to the specified processing object among a plurality of extraction processes set to be executable. .
According to the information processing method of configuration 10, by performing extraction processing corresponding to the processing target specified in the captured image, information on a plurality of types of processing targets is obtained based on the captured image of the camera mounted on the vehicle. can be obtained.

DESCRIPTION OF SYMBOLS 1... Information processing system 10... Processor 11... Acquisition part 12... Identification part 13... Extraction part 14... Output part 20... Storage part 21... Control program 22... Moving image data 23... Shooting Image data 24 Thermographic image data 25 Setting data 26 Processing result data 30 Communication unit 100 Information processing server (information processing device) 131 First processing unit 132 Second processing unit DESCRIPTION OF SYMBOLS 133... 3rd process part, 134... 4th process part, 200... Vehicle-mounted apparatus, 201... Processor, 202... Storage part, 203... Communication part, 204... Touch panel, 205... Camera, 206... Infrared camera, 207... GNSS, 250 User terminal 300 Distribution server D Photographed data D1 Photographed image D2 Thermal image D3 Location information D4 Photographed date and time NW Communication network V Vehicle.

Claims (10)

an acquisition unit that acquires an image captured by a camera mounted on a vehicle;
a specifying unit that specifies a processing target based on the captured image;
an extracting unit that extracts a change point from the captured image to detect a change in the object to be processed;
The extraction unit is configured to be capable of executing a plurality of extraction processes, and extracts a change point of the processing object by performing the extraction process corresponding to the processing object specified by the specifying unit. processing equipment. The extracting unit is capable of executing a plurality of the extraction processes that differ in the change point in the object to be processed and the mode of change detected for the change point,
identifying and executing the extraction processing corresponding to the processing object identified by the identifying unit from among the plurality of extraction processing, thereby detecting a change mode of the change point in the processing object; Item 1. The information processing apparatus according to item 1. In the extraction process, the extraction unit extracts the change point extracted from the first captured image and the change extracted from the second captured image captured at a different time from the first captured image. 3. The information processing apparatus according to claim 1, wherein a change at said change point is detected by comparing a point and a point. When the identifying unit identifies a house as the object to be processed, the extracting unit extracts a point of change from an image of a side surface of the house included in the photographed image to determine whether the house is inhabited. 4. The information processing apparatus according to any one of claims 1 to 3, which determines whether the house is vacant. The extraction unit extracts from the change point extracted from the image of the side of the house included in the captured image taken at night and the image of the side of the house included in the captured image taken during the daytime. 5. The information processing apparatus according to claim 4, which determines whether said house is inhabited or vacant by comparing with said change point. Based on the change points extracted from the photographed image, the extracting unit extracts from the image of the side of the house the presence or absence of people inside or near the house, the lighting state of the lighting of the house, By detecting at least one of the state of the laundry being hung, the open/closed state of the windows of the residence, the temperature change in part or the entirety of the residence, and the state of the mail delivered to the residence, 6. The information processing apparatus according to claim 4, which determines whether said house is inhabited or vacant. When the identification unit identifies a traffic light as the object to be processed, the extraction unit extracts a change point from the image of the traffic light included in the captured image to determine whether the operation state of the traffic light is normal. 4. The information processing apparatus according to any one of claims 1 to 3, which determines at least one of whether or not a foreign object is attached to said traffic signal. When a road is specified as the object to be processed by the specifying unit, the extraction unit detects an abnormality in the road surface condition of the road by extracting a change point from the image of the road included in the captured image. 4. The information processing apparatus according to any one of claims 1 to 3. The acquisition unit acquires a moving image captured by the camera and position information of the vehicle during capturing of the moving image, obtains the captured image by extracting a still image from the moving image, and obtains the captured image by extracting a still image from the moving image. performing a process of generating position information corresponding to the still image based on the position information of and associating the generated position information with the captured image;
9. The information processing apparatus according to any one of claims 1 to 8, wherein the extraction unit outputs position information associated with the captured image together with a result of processing for extracting the change point from the captured image. . Acquire images captured by the camera mounted on the vehicle,
Identifying an object to be processed based on the captured image,
A process for extracting a change point from the photographed image as a target for detecting a change in the object to be processed, which corresponds to the specified object to be processed, among a plurality of extraction processes set to be executable. An information processing method for extracting a change point of the object to be processed by performing an extraction process.

Images