JP2020013556A - Information processing device, information processing method, program, and application program - Google Patents

Abstract

To provide an information processing device, an information processing method, and a program that can obtain necessary information while reducing traffic.SOLUTION: In an information processing system, an imaging control unit 28 of an imaging device 20A causes an imaging unit 24 to capture an image and causes a storage unit 40A to store the captured image together with positional information from a position identifying unit 26 as correspondence information 42. Next, a determination unit 29 inputs the captured image into a determination model 45, and determines whether the image is a transmission object on the basis of a result output by the determination model 45. If it is determined that the image is a transmission object, a transmission control unit 30 transmits the image input into the determination model 45 to an information processing device.SELECTED DRAWING: Figure 12

Description

  The present invention relates to an information processing device, an information processing method, a program, and an application program.

  Conventionally, the price information extracted from the image is associated with the position information of the point where the price information is obtained, and the price information transmitting device that transmits the price information to the price information receiving device, and indicates the lowest price among the received price information. Price information reception that extracts the lowest price information and the lowest position information associated with the lowest price information, and notifies the lowest price information when the own device passes near the point indicated by the lowest price information. A price information providing system including a device is disclosed (for example, see Patent Document 1).

JP 2014-21516 A

  However, in the conventional technique, if the image itself is transmitted to the price information receiving apparatus, if the image is transmitted without any restrictions, the communication volume increases, which may be undesirable for the user or the administrator.

  The present invention has been made in view of such circumstances, and provides an information processing apparatus, an information processing method, a program, and an application program capable of acquiring necessary information while suppressing the amount of communication. Is one of the objectives.

  One embodiment of the present invention provides a communication unit that communicates with another device, an imaging control unit that causes an imaging unit that captures a landscape to capture an image, a position acquisition unit that acquires a position, and transmission based on the image and model information. An acquisition unit that acquires an image in which the landscape associated with the position information is captured from a terminal device including a determination unit that determines an image, and a representation of the captured landscape from the image acquired by the acquisition unit. An information processing apparatus comprising: a recognition unit that recognizes a meaning; and a control unit that transmits the model information to the terminal device and causes the terminal device to transmit the image.

  According to one embodiment of the present invention, necessary information can be acquired while suppressing traffic.

FIG. 2 is a diagram illustrating an example of a functional configuration of the information processing system 1. FIG. 4 is a sequence diagram illustrating a flow of a process performed by the imaging device 20 and the information processing device 100. FIG. 4 is a diagram showing an example of the content of reference information 44. FIG. 4 is a diagram showing an example of the content of correspondence information 42. FIG. 9 is a diagram illustrating an example of updated reference information 44. 9 is a flowchart illustrating an example of a flow of a process performed by the imaging device. 5 is a flowchart illustrating an example of the flow of a specific process executed by the information processing device 100. It is a figure showing an example of signs that a rectangle was derived in a target picture. FIG. 7 is a diagram illustrating an example of a state in which a rectangular image IM is input to a specific model 126 and the specific model 126 outputs a price of regular gasoline. FIG. 4 is a diagram illustrating an example of the content of recognition information 128. FIG. 4 is a diagram illustrating an example of an image displayed on the terminal device 300. It is a figure showing an example of the functional composition of imaging device 20A of a 2nd embodiment. It is a flow chart which shows an example of the flow of processing performed by imaging device 20A. FIG. 9 is a conceptual diagram of a process in which an imaging device determines whether an image is a transmission target image or a non-transmission target image. It is a flow chart which shows an example of the flow of processing performed by imaging device 20A of a 3rd embodiment. It is a figure for explaining processing which specifies an image before and after refueling. It is a figure for explaining processing which specifies an image before and after having stopped at parking. FIG. 6 is a diagram illustrating an example of the contents of a plurality of specific models 126. It is a flow chart which shows an example of the flow of processing performed by information processor 100 of a 4th embodiment. FIG. 6 is a diagram illustrating an example of the contents of a plurality of specific models 126. It is a flow chart which shows an example of the flow of processing performed by information processor 100 of a 5th embodiment. It is a figure showing an example of functional composition of information processing system 1A of a 6th embodiment. FIG. 7 is a diagram showing an example of the content of learning data 412. FIG. 9 is a diagram showing another example of the content of learning data 412.

  Hereinafter, embodiments of an information processing apparatus, an information processing method, a program, and an application program of the present invention will be described with reference to the drawings.

[Overview (1)]
The information processing device according to the embodiment is realized by one or more processors. The information processing device (server) includes a communication unit that communicates with another device, an imaging control unit that causes an imaging unit that captures a landscape to capture an image based on reference information indicating a reference for capturing a landscape, and a position that obtains a position. From a terminal device including an acquisition unit, an image in which the landscape associated with positional information is captured is obtained, and from the obtained image, a meaning represented by the captured landscape is recognized, and the reference information is stored in the terminal device. And the terminal device transmits the image.

  The “terminal device (edge)” may be a terminal device mounted on a vehicle or a portable terminal device attachable to a vehicle. For example, the terminal device is a terminal device such as a drive recorder or a smartphone. In addition, the terminal device may obtain an image captured by an imaging unit mounted on the vehicle, or may obtain a specification result of a position specifying unit mounted on the vehicle. In the following description, the terminal device will be described as a drive recorder.

  The “reference information” includes, for example, information indicating a position at which a landscape is imaged, information based on a position where a predetermined event has occurred, information based on a position where a predetermined feature exists, and behavior of a vehicle (for example, speed). And information indicating the state of the vehicle (for example, a stopped state or a refueling state). The predetermined feature is, for example, a gas station or a parking lot. The “meaning of the landscape” is, for example, a feature, a character, a symbol, or the like recognized from the landscape.

[Overview (Part 2)]
The information processing device according to the embodiment is realized by one or more processors. The information processing apparatus includes a communication unit that communicates with another device, an imaging control unit that causes an imaging unit that captures a landscape to capture an image, a position acquisition unit that acquires a position, and transmits the image based on the captured image and model information. From a terminal device including a determination unit that determines an image, obtains an image in which the landscape associated with the position information is captured, and recognizes the meaning of the captured landscape from the obtained image, The model information is transmitted to the terminal device, and the terminal device transmits the image.

  The “image in which the landscape is captured” is an image in which a predetermined feature such as a gas station or a parking lot is captured. The “model information” outputs information indicating whether the image is a transmission target image when an image captured by the imaging unit is input. The model information is generated by machine learning of learning data in which an image to be transmitted is associated with information indicating that the image is a transmission target.

[Overview (Part 3)]
The information processing device (an example of the “information providing device”) of the embodiment is realized by one or more processors. The information processing device obtains an image in which the position information specified by the position specifying unit is associated with the other device having the position specifying unit and the imaging unit, and the image obtained by capturing the scenery by the imaging unit. Extracting a semantic region representing a specific meaning related to a feature included in the landscape, recognizing a specific meaning in the semantic region, and associating with a position where an image including the semantic region was captured, Provide the user with a specific recognized meaning.

  The “feature” includes, for example, one or both of a gas station and parking. The “meaning area” is, for example, a signboard of a gas station or a signboard of a parking lot. The “specific meaning” is, for example, information indicating the fuel price of a sign at a gas station or information indicating the availability of a sign at a parking lot. "Providing a user with a recognized specific meaning in association with a position" means, for example, associating the recognized specific meaning with a user in association with the position or an object at (or near) the position. To provide. Further, "providing a specific meaning to a user" means, for example, providing a specific meaning to a terminal device held by a user, or specifying a specific meaning in a database to provide to a terminal device held by a user. Is stored in association with the feature.

  Further, the information processing device recognizes a specific meaning in the semantic region by using a specific model applied from the plurality of specific models to each position where the image obtained by the obtaining unit is captured. Is also good. The specific model applied to each position is generated by machine learning of learning data in which a learning image captured at the position and a specific meaning in the meaning area in the learning image are associated with each other. It was done.

  Further, the information processing device may recognize a specific meaning in the semantic region from a plurality of specific models, using a specific model applied to the scenery of the image acquired by the acquisition unit. The specific model applied to the landscape of the image is obtained by machine learning of learning data in which the learning image in which the landscape is captured and the specific meaning in the meaning area in the learning image are associated with each other. It has been generated.

  Further, the information processing device may recognize a specific meaning in the semantic region from a plurality of specific models, using a specific model applied to a sign included in the image acquired by the acquiring unit. . The specific model to which the signboard is applied is generated by machine learning learning data in which a learning image obtained by capturing the signboard and a specific meaning indicated by the signboard are associated with each other.

  Also, the information processing apparatus extracts a meaning area representing the meaning of the specific thing in the image from the image including the specific thing acquired by the acquisition unit at a first time, and acquires the acquisition area at a second time. Extracting a semantic area representing the meaning of the specific thing in the image from the image including the specific thing acquired by the above, information on the semantic area of the image acquired by the acquiring unit at the first time, The specific meaning in the meaning area may be recognized based on a difference from the information of the meaning area acquired by the acquisition unit at a time. The specific meaning is, for example, the price of fuel of the vehicle or the availability of parking.

<First embodiment>
[Constitution]
FIG. 1 is a diagram illustrating an example of a functional configuration of the information processing system 1. The information processing system 1 includes, for example, a vehicle 10, an information processing device 100, a news server 200, and a terminal device 300. These devices communicate with each other via a network NW. The network NW includes, for example, a WAN (Wide Area Network), a LAN (Local Area Network), the Internet, a dedicated line, a wireless base station, a provider, and the like. Note that some or all of the configurations or functions included in the information processing device 100 may be included in the vehicle 10. In addition, some or all of the functions or configurations included in the vehicle 10 may be included in the information processing device 100.

[vehicle]
An imaging device 20 is mounted on the vehicle 10. The imaging device 20 includes, for example, a communication unit 22, an imaging unit 24, a position identification unit 26, an imaging control unit 28, a transmission control unit 30, and a storage unit 40. The imaging control unit 28 and the transmission control unit 30 are realized when a hardware processor such as a CPU (Central Processing Unit) executes the imaging application 46 stored in the storage unit 40. The imaging control unit 28 and the transmission control unit 30 cause the imaging unit 24 to capture an image and transmit the captured image to the information processing device 100 in cooperation with the information processing device 100. The storage unit 40 is realized by, for example, a ROM (Read Only Memory), a flash memory, an SD card, a RAM (Random Access Memory), and the like. The storage unit 40 stores correspondence information 42, reference information 44, and an imaging application 46. Details of the contents of the correspondence information 42 and the reference information 44 will be described later.

  The communication unit 22 performs wireless communication, connects to the network NW, and communicates with another device via the network NW. The imaging unit 24 images a landscape. For example, a landscape is acquired as a moving image by the imaging unit 24. The imaging unit 24 is, for example, a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera.

  The position specifying unit 26 specifies the position of its own device based on a signal received from a Global Navigation Satellite System (GNSS) satellite.

  The imaging control unit 28 causes the imaging unit 24 to image a landscape based on the reference information 44. The transmission control unit 30 transmits the image captured by the imaging unit 24 to the information processing device 100 using the communication unit 22. The image captured by the imaging unit 24 may be transmitted to the information processing device 100 using the communication unit of the vehicle 10.

[Information processing device]
The information processing apparatus 100 includes, for example, a communication unit 102, a first processing unit 104, a second processing unit 106, a third processing unit 108, a tracking processing unit 110, an information management unit 112, an updating unit 114 , A providing unit 116 and a storage unit 120. The first processing unit 104, the second processing unit 106, the third processing unit 108, the tracking processing unit 110, the information management unit 112, the updating unit 114, and the providing unit 116 are configured such that a hardware processor such as a CPU is stored in a storage device. It is realized by executing the set program. These functional units may be realized by hardware such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or the like. It may be realized by cooperation of software and hardware. In addition, the above-described program may be stored in a storage device in advance, or may be stored in a removable storage medium such as a DVD or a CD-ROM. May be installed in the storage device.

  The storage unit 120 is realized by, for example, a ROM, a flash memory, an SD card, a RAM, a hard disk drive (HDD), a register, and the like. Further, part or all of the storage unit 120 may be a NAS (Network Attached Storage), an external storage server device, or the like. The storage unit 120 stores, for example, correspondence information 122, reference information 124, a specific model 126, and recognition information 128. Details of each information will be described later.

  The communication unit 102 communicates with the imaging device 20 via the network NW. The communication unit 102 includes, for example, a communication interface such as a NIC (Network Interface Card).

  The first processing unit 104 divides the moving image captured by the imaging unit 24 into images, and generates a target image to be processed.

  The second processing unit 106 derives a rectangle estimated to include a predetermined feature or a rectangle estimated to have a predetermined feature from the image. The second processing unit 106 derives a rectangle based on a predetermined rectangle detection algorithm.

  The third processing unit 108 inputs the rectangular image derived by the second processing unit 106 to the predetermined specific model 126, and recognizes the meaning of the landscape included in the rectangle based on the output result of the specific model 126. The specific model 126 is a model for classifying an image such as a neural network generated by machine learning or the like.

  The tracking processing unit 110 tracks the same feature in a plurality of target images, and integrates processing results for the same feature. For example, the tracking processing unit 110 extracts the meaning of the feature based on the recognition result of the same feature in a predetermined number of images. For example, the tracking processing unit 110 recognizes the meaning of a feature in a predetermined number of continuous images (frames). Further, the tracking processing unit 110 determines the movement of the position of the feature between a plurality of frames. More specifically, in the processing of recognizing meaning using a plurality of images, the tracking processing unit 110 recognizes a feature twice as a sign of a gas station, and identifies the same feature once as a sign of a restaurant. If recognized, for example, the feature is identified as a gas station sign.

  The information management unit 112 manages information acquired by the own device and processing results of the own device. The updating unit 114 updates the reference information 124 and transmits the updated reference information 124 to the imaging device 20. The providing unit 116 provides the information included in the recognition information 128 to the terminal device 300 based on the request of the terminal device 300.

  The news server 200 provides information related to news to the information processing apparatus 100. The information on news is information including an event that the public pays attention to, such as an incident, an event, an accident, and the like, and a place and a location where the event has occurred.

  The terminal device 300 is a terminal device held by the user. The terminal device 300 is, for example, a smartphone, a tablet terminal, a navigation device, or the like.

[Process Performed by Information Processing System]
FIG. 2 is a sequence diagram illustrating a flow of processing performed by the imaging device 20 and the information processing device 100. In this processing, the information processing apparatus 100 corrects or instructs the place and time at which a moving image is imaged, and the imaging apparatus 20 instructs the imaging unit 24 to image a landscape.

  First, the information processing device 100 transmits the reference information 124 to the imaging device 20 (S10). Next, the imaging device 20 acquires the transmitted reference information 124 and stores the acquired reference information 124 (44) in the storage unit 40 (S12). Next, the image capturing apparatus 20 causes the image capturing section 24 to capture an image based on the position information specified by the position specifying section 26 and the reference information 44 (S14), and the captured image and the position where the image was captured. The corresponding information 42 corresponding to the information is transmitted to the information processing apparatus 100 (S16). Next, the information processing apparatus 100 stores the acquired correspondence information 42 in the storage unit 120 (S18).

(Standard information)
FIG. 3 is a diagram illustrating an example of the content of the reference information 44. The reference information 44 is information in which an imaging region for imaging a landscape is associated with a time zone of an imaging target. The reference information 44 is, for example, information defining a place and a time at which a moving object such as the imaging device 20 captures an image based on information extraction from a previous image. For example, if the images at the position have not been sufficiently collected, the reference information 44 is generated so that the position is imaged. The reference information 44 shown in FIG. 3 includes the imaging regions AR1 to AR3, and the imaging regions AR1 to AR3 are associated with a date and time. For example, the imaging regions AR1 to AR3 are regions indicated by circles formed with a predetermined radius around a predetermined position. For example, the imaging device 20 causes the imaging unit 24 to start imaging when entering the imaging regions AR1 to AR3. Note that the reference information 44 may be associated with only the imaging region.

(Corresponding information)
FIG. 4 is a diagram illustrating an example of the content of the correspondence information 42. The correspondence information 42 is, for example, information in which a captured moving image (image) and a date and time when the moving image is captured are associated with each other based on the reference information 44. For example, in the correspondence information 42, position information is associated with each moving image capturing time.

  Returning to the description of FIG. The information processing device 100 executes a specific process based on the correspondence information 122 (step S20). The specific process is a process of recognizing the meaning of a landscape in an image. Details of the specific processing will be described later with reference to FIGS.

  Next, the information processing apparatus 100 determines whether it is time to update the reference information 124 (44) (S22). When it is time to update the reference information 124, the information processing apparatus 100 updates the reference information 124 (S24), and transmits the updated reference information 124 to the imaging device 20 (S26). Then, the imaging device 20 acquires the updated reference information 124 (44), and acquires an image based on the acquired reference information 124 (S28).

(Updated reference information)
FIG. 5 is a diagram illustrating an example of the updated reference information 44. For example, when the information processing apparatus 100 acquires information about news from the news server 200, the information processing apparatus 100 updates the reference information 124 based on the acquired information about news. The information processing apparatus 100 updates the reference information 124 by adding the imaging area AR4 including the position of the news information to the imaging areas AR1 to AR3. For example, the imaging area AR4 is an area indicated by a circle formed with a predetermined radius around the position where the incident occurred. The update may be performed when a predetermined period has elapsed from when the reference information 44 was transmitted last time.

  Further, the updating unit 114 may generate the reference information 44 in which the imaging position and the time are corrected based on one or more items among the time, the latitude and longitude, or the speed at which the image was captured last time. In this case, for example, the updating unit 114 updates (corrects) the reference information 44 so that an image of the place or time zone where the image was sufficiently captured last time is captured. In addition, since a place where the vehicle traveled at a predetermined speed or higher could not sufficiently capture an image, the reference information 44 is generated so as to capture the place again. In this case, for example, the information processing device 100 acquires information (for example, the speed itself and position information acquired at predetermined intervals) for specifying the moving speed of the vehicle 10 or the imaging device 20.

  As described above, the information processing apparatus 100 corrects the location and position at which the imaging device 20 captures an image, and instructs the imaging device 20. Then, the imaging device 20 images the landscape based on the instruction. As described above, the imaging device 20 and the information processing device 100 perform the processing in cooperation with each other, so that it is possible to acquire necessary information while suppressing the communication amount.

[Process Performed by Imaging Device]
FIG. 6 is a flowchart illustrating an example of the flow of a process performed by the imaging device 20. First, based on the reference information 44 and the position specified by the position specifying unit 26, the imaging control unit 28 determines whether or not the current position is within the imaging region to be imaged (S100). If it is not the imaging region, the processing of one routine of this flowchart ends.

  If it is the imaging region, the imaging control unit 28 causes the imaging unit 24 to start imaging (S102). Next, the imaging control unit 28 causes the storage unit 40 to store the correspondence information 42 that associates the image captured by the imaging unit 24 with the position information (S104). Next, the transmission control unit 30 transmits the correspondence information 42 to the information processing device 100 (S106). The information processing apparatus 100 causes the storage unit 120 to store the transmitted correspondence information 42 as the correspondence information 122. Thus, the processing of one routine of this flowchart ends.

  For example, in order to utilize a moving image captured by a drive recorder or the like mounted on a vehicle, if the entire moving image is transmitted to the information processing apparatus 100, the communication amount may exceed an allowable amount. On the other hand, by performing the above-described processing, the imaging device 20 transmits the image captured based on the reference information 44 to the information processing device 100, and thereby transmits necessary information to the information processing device 100 while suppressing the communication amount. Can be sent.

[Specific processing executed by information processing device]
FIG. 7 is a flowchart illustrating an example of the flow of the specific processing executed by the information processing apparatus 100. First, the first processing unit 104 of the information processing apparatus 100 determines whether or not an unprocessed moving image exists in the correspondence information 122 (S200). If there is no unprocessed moving image, the processing of one routine of this flowchart ends.

  If there is an unprocessed moving image, the first processing unit 104 divides the moving image (S202) and generates a processing target image (S204). Next, the second processing unit 106 derives a rectangle in the target image (S206). FIG. 8 is a diagram illustrating an example of a state in which a rectangle is derived in the target image. In the target image, a rectangle estimated to include a predetermined feature (feature) is derived.

  Next, the third processing unit 108 inputs a rectangular image to the specific model 126 (S208). For example, the third processing unit 108 extracts all the rectangles derived from the target image, selects the extracted rectangular images one by one, and inputs the selected images to the specific model 126.

  Next, the third processing unit 108 specifies a feature included in the image based on the output result output to the specific model 126 (S210). FIG. 9 is a diagram illustrating an example of a state in which a rectangular image IM is input to the specific model 126 and the specific model 126 outputs the price of regular gasoline. For example, when a rectangular image includes a signboard describing the price of regular gasoline, and this image (or the characteristics of the image) is input to the specific model 126, the specific model 126 derives the price of regular gasoline. The feature of the image is, for example, the brightness of each pixel, a brightness gradient, or a feature amount derived based on a plurality of pixels in a predetermined range.

  Returning to the description of FIG. Next, the third processing unit 108 determines whether or not all the divided images have been processed (S212). When all the images have been processed, the processing of one routine of this flowchart ends. If all target images have not been processed, the process returns to S204. When a plurality of target images are processed in the process of this flowchart, the tracking processing unit 110 executes a process of integrating the processing results of the plurality of target images, and recognizes the meaning of the same feature in the target images. May be.

  Through the above-described processing, the information management unit 112 of the information processing apparatus 100 generates, for example, recognition information 128. FIG. 10 is a diagram illustrating an example of the content of the recognition information 128. The recognition information 128 is, for example, information defined for each moving image. The recognition information 128 includes, for example, image identification information, position information at which the image was captured, rectangular identification information, information indicating a feature, semantic information derived from the feature, acquisition date and time of the image, and image capture of the image. This is information in which identification information (ID) of the imaging device 20 is associated.

  Based on the recognition information 128, the information processing apparatus 100 can recognize the gasoline price of a gas station located at a predetermined position and the availability of predetermined parking. Then, the information processing device 100 can transmit the above recognition result to the terminal device 300. Thereby, the user of the terminal device 300 can acquire information on a desired feature.

  FIG. 11 is a diagram illustrating an example of an image displayed on the terminal device 300. For example, the providing unit 116 of the information processing device 100 provides the recognition information 128 to the terminal device 300 in response to a request of the terminal device 300. For example, when the providing unit 116 obtains, from the terminal device 300, a request for providing a price of gasoline around the position information Pc together with the position information Pc of the terminal device 300, the gasoline at a gas station around the position information Pc from the recognition information 128. The price is extracted, and information associating the extracted position of the gas station with the gasoline price is provided to the terminal device 300. Thereby, the gas prices of the gas stations around the position information Pc are displayed on the display unit of the terminal device 300 as shown in FIG.

  Further, the communication unit 102 may acquire a route (a designated route) on which the vehicle 10 is to travel. In this case, the providing unit 116 may provide the user with information indicating the fuel price recognized by the above-described processing or the parking availability based on the acquired route. For example, the providing unit 116 may provide the user with the price of fuel at a gas station existing along the planned route or along the road, or the availability of the space near the destination. Further, the providing unit 116 may provide the user with the position of the gas station where the price of the fuel is the lowest.

  According to the first embodiment described above, the information processing apparatus 100 captures an image of the landscape associated with the position information from the imaging apparatus 20 and, from the acquired image, the meaning of the captured landscape. By transmitting the reference information 124 to the imaging device 20 and causing the imaging device 20 to transmit an image, necessary information can be acquired while suppressing the amount of communication.

<Second embodiment>
Hereinafter, the second embodiment will be described. In the first embodiment, the description has been given assuming that the imaging device 20 captures an image based on the reference information 44 when the position of the imaging device 20 is within the imaging region. On the other hand, in the second embodiment, the imaging device 20 acquires the judgment model 45 from the information processing device 100 instead of (or in addition to) the reference information 44. Then, when the image capturing device 20 determines that the captured image is a transmission target based on the determination model 45, the image capturing device 20 transmits the image to the information processing device 100. Hereinafter, the second embodiment will be described.

  FIG. 12 is a diagram illustrating an example of a functional configuration of an imaging device 20A according to the second embodiment. The imaging device 20A includes a storage unit 40A instead of the storage unit 40. The determination model 45 is stored in the storage unit 40A. The determination model 45 is a model that, when an image is input, derives a score for the image. For example, the determination model 45 is a model that derives a score indicating that the image is an image including a predetermined feature (for example, a gas station or parking lot). The determination model 45 is a model for classifying an image such as a neural network in which learning data is generated by machine learning or the like. The learning data is, for example, information in which an image including a predetermined feature and information for identifying the feature are associated with each other.

  The imaging device 20A further includes a determining unit 29 in addition to the functional configuration of the imaging device 20. The determination unit 29 inputs the image captured by the imaging unit 24 to the determination model 45, and acquires the score output by the determination model 45. Then, the determination unit 29 determines whether to transmit the image to the information processing device 100 based on the acquired score. Note that, for example, the imaging control unit 28 of the second embodiment causes the imaging unit 24 to always capture an image.

[Process Performed by Imaging Device]
FIG. 13 is a flowchart illustrating an example of the flow of a process performed by the imaging device 20A. First, the imaging control unit 28 causes the imaging unit 24 to capture a moving image (S300). Next, the imaging control unit 28 causes the storage unit 40 to store the correspondence information 42 in which the image captured by the imaging unit 24 is associated with the position information (S302). Next, the determination unit 29 inputs the captured image (the image obtained by dividing the moving image) to the determination model 45 (S304), and transmits the image input to the determination model 45 based on the result output by the determination model 45. It is determined whether the image is a target image (S306). If the image input to the determination model 45 is not the image to be transmitted, the processing of one routine of this flowchart ends. If the image input to the determination model 45 is an image to be transmitted, the transmission control unit 30 transmits the image input to the determination model 45 to the information processing device 100 (S308). Thus, the processing of one routine of this flowchart ends.

  FIG. 14 is a conceptual diagram of a process in which the imaging device 20A determines whether an image is an image to be transmitted or an image not to be transmitted. In the illustrated example, the determination model 45 is a model that outputs information indicating the degree to which a feature that seems to be a gas station is included in the image. When the determination model 45 outputs a result indicating that the input image includes a gas station-like feature as shown in the upper diagram of FIG. 14, this image is set as a transmission target image. When the judgment model 45 outputs a result indicating that the input image does not include a feature that seems to be a gas station, as shown in the lower diagram of FIG. 14, this image is regarded as an image not to be transmitted.

  The determination model 45 is not limited to one, and a plurality of determination models 45 may be prepared. For example, the imaging device 20 may input an image to the plurality of determination models 45 and, when one of the determination models 45 outputs a result to be transmitted, may set the image as a transmission target.

  In addition, the determination unit 29 uses the plurality of determination models 45 to transmit, to the information processing apparatus 100, an image captured after the first feature is recognized and before the second feature is recognized. May be sent. For example, the determination unit 29 may transmit an image captured until the red building is recognized after the guardrail is recognized to the information processing apparatus 100.

  According to the second embodiment described above, the information processing device 100 transmits the determination model 45 to the imaging device 20A. Then, when determining that the image captured by the imaging unit 24 is the image to be transmitted based on the determination model 45, the imaging device 20A transmits the image to the information processing device 100. As described above, since the information processing apparatus 100 does not need to set an imaging region in a wide range, the information processing apparatus 100 can more easily acquire necessary information while suppressing a communication amount.

<Third embodiment>
Hereinafter, a third embodiment will be described. In the third embodiment, the information processing device 100 transmits a predetermined condition to the imaging device 20A instead of the reference information 44. When the timing at which an image is captured satisfies a predetermined condition, the imaging device 20A transmits the image captured at that timing to the information processing device 100. The predetermined condition is a condition set for information on the vehicle. Hereinafter, the third embodiment will be described focusing on the differences from the first embodiment.

  FIG. 15 is a flowchart illustrating an example of the flow of a process performed by the imaging device 20A according to the third embodiment. This processing is, for example, processing executed at predetermined intervals. First, the imaging control unit 28 causes the imaging unit 24 to capture a moving image (S400). Next, the imaging control unit 28 causes the storage unit 40 to store the correspondence information 42 in which the image captured by the imaging unit 24 is associated with the position information (S402). It is assumed that the imaging device 20 communicates with the vehicle and acquires the state of the vehicle at each time.

  Next, the determination unit 29 refers to the correspondence information 42 and information about the vehicle, specifies an image to be transmitted (S404), and transmits the specified image to the information processing device 100 (S406). Thus, the processing of this flowchart ends.

  Hereinafter, a method for specifying an image to be transmitted with reference to the correspondence information 42 and information about the vehicle will be described. For example, the determination unit 29 determines an image captured at a timing when the vehicle is controlled by a preset control procedure as an image to be transmitted. The vehicle is controlled by a preset control procedure, for example, a state in which the acceleration is controlled to be equal to or higher than a threshold value, or a state in which the speed is controlled to be equal to or higher than a first predetermined speed and then controlled to be equal to or lower than a second predetermined speed. And that the steering is operated at a predetermined degree or more.

  The determining unit 29 may determine an image captured at a timing at which the position information and the speed information of the vehicle satisfy a predetermined condition as an image to be transmitted. For example, the image before and after the change in speed may be transmitted to the information processing apparatus 100, such as when the vehicle has traveled at about 80 km on a highway, but has traveled at 50 km only in a predetermined section. In this case, an accident or an obstacle may be present.

  Further, for example, the determination unit 29 may determine an image to be transmitted based on the state of the vehicle as described below. For example, (1) images before and after refueling and (2) images before and after stopping at parking may be transmitted to the information processing apparatus 100. (1) Images before and after refueling are specified based on changes in fuel in the fuel tank. FIG. 16 is a diagram for explaining a process of specifying images before and after refueling. The vertical axis of FIG. 16 indicates the degree of fuel in the fuel tank, and the horizontal axis of FIG. 16 indicates time. For example, when the degree of fuel rises at a predetermined slope as shown at time P1 in the figure, the determination unit 29 determines that fuel has been replenished at that time (for example, it is determined that gasoline has been refueled at the gas station). Then, the images at times P2 and P3 before and after that are determined to be images to be transmitted.

  As described above, when the image of the gas station is to be transmitted, the imaging device 20A can more reliably transmit the image of the gas station to the information processing device 100 by the processing as described above.

  (2) The images before and after stopping at parking are specified based on the speed of the vehicle and the state of the parking brake. FIG. 17 is a diagram for describing a process of specifying images before and after the vehicle has been parked. The vertical axis in FIG. 17 indicates the speed of the vehicle, and the horizontal axis in FIG. 17 indicates time. For example, when the speed of the vehicle is zero for a predetermined time and the parking brake is in a specific state as shown in a time P11 in the drawing, the determination unit 29 determines that the vehicle has stopped in parking at that time. It is determined that at least the image at the time P13 after the specific state is the image to be transmitted.

  As described above, when the image of parking is to be transmitted, the imaging device 20A can more reliably transmit the image of parking to the information processing device 100 by the processing as described above.

  According to the third embodiment described above, the information processing device 100 transmits a predetermined condition to the imaging device 20A. Then, based on a predetermined condition, when determining that the image captured by the imaging unit 24 is the image to be transmitted, the imaging device 20A transmits the image to the information processing device 100. As described above, since the information processing apparatus 100 does not need to set an imaging region in a wide range, the information processing apparatus 100 can more easily acquire necessary information while suppressing a communication amount.

<Fourth embodiment>
Hereinafter, a fourth embodiment will be described. The fourth embodiment will be described based on the processing of the first embodiment. In the fourth embodiment, the information processing device 100 includes a plurality of specific models 126. Then, the information processing apparatus 100 selects the specific model 126 from the plurality of specific models 126 based on the position where the image is captured, inputs the image to the selected specific model 126, and analyzes the image. Hereinafter, differences from the first embodiment will be mainly described.

  The specific model 126 is stored in the storage unit 120 of the information processing apparatus 100 according to the fourth embodiment. FIG. 18 is a diagram illustrating an example of the contents of a plurality of specific models 126. For example, the plurality of specific models 126 are information in which positions (or regions), types of the specific models 126, and recognition targets are associated with each other. The specific model 126 associated with the position is a specific model capable of more accurately recognizing the meaning represented by the landscape included in the rectangle of the captured image at the associated position than the other specific models 126. As illustrated, the recognition target of the specific model 126 is not limited to one, but may be a plurality.

  FIG. 19 is a flowchart illustrating an example of the flow of a process performed by the information processing device 100 according to the fourth embodiment. The description of the same processing as in FIG. 7 is omitted. After the processing in S200, the first processing unit 104 divides the moving image (S202). Next, the first processing unit 104 specifies the target image and the position where the target image is specified (S203). Next, the first processing unit 104 specifies the specific model 126 based on the specified position (S205). Next, the second processing unit 106 derives a rectangle in the target image (S206). Next, the third processing unit 108 inputs a rectangular image to the specific model 126 specified in S205 (S208). Then, the process proceeds to the subsequent processing.

  As described above, the information processing apparatus 100 can more accurately recognize the meaning included in the image by specifying the specific model 126 to be used based on the position of the scenery included in the image.

  According to the fourth embodiment described above, the information processing apparatus 100 is applied to each position where an acquired image is captured, from among a plurality of specific models 126 applied for each position where an image is captured. By using the specific model 126 to recognize a specific meaning in a semantic area representing a specific meaning related to a feature included in the landscape and providing the recognized specific meaning to the user, Can provide more useful information.

  Note that, for example, when the target image is an image of a signboard of a gas station located at a predetermined position, the third processing unit 108 determines the same image of the same gas station as the previous image of the signboard of the same gas station. Price differences may be extracted from the difference from the current image of the signboard. For example, the third processing unit 108 derives the difference between the image taken in the rectangle including the gasoline price of the image taken last time and the image taken this time, and based on the derived difference and a preset template. Thus, the price of gasoline in the image captured this time may be derived. In this way, the third processing unit 108 can extract the price of gasoline based on the difference from the previously captured image when performing image recognition.

<Fifth embodiment>
Hereinafter, a fifth embodiment will be described. The fifth embodiment will be described based on the processing of the second embodiment. In the fifth embodiment, the first processing unit 104 selects the specific model 126 from the plurality of specific models 126 based on the type of the image transmitted from the imaging device 20, and stores the image in the selected specific model 126. Enter and analyze the image. Hereinafter, differences from the first embodiment will be mainly described.

  The storage unit 120 of the information processing apparatus 100 according to the fifth embodiment stores a plurality of specific models 126. FIG. 20 is a diagram illustrating an example of the contents of a plurality of specific models 126. For example, the plurality of specific models 126 are information in which an image type, a model type, and a recognition target are associated with each other. The specific model 126 associated with the type of the image is a specific model that can more accurately recognize the meaning of the landscape included in the rectangle of the image of the associated type than the other specific models 126. For example, the type of image is not limited to a type classified as a gas station or a parking lot as shown in the figure, and may be a type classified as a building or a signboard. As shown in the figure, the recognition target of the specific model 126 is not limited to one, but may be a plurality.

  FIG. 21 is a flowchart illustrating an example of the flow of a process performed by the information processing device 100 according to the fifth embodiment. The description of the same processing as in FIG. 7 is omitted. After the processing in S200, the first processing unit 104 divides the moving image (S202) and generates a target image to be processed (S204). Next, the first processing unit 104 specifies the specific model 126 based on the type of the target image (S205 #). Next, the second processing unit 106 derives a rectangle in the target moving image (S206). Next, the third processing unit 108 inputs a rectangular image to the specific model 126 specified in S205 # (S208). Then, the process proceeds to the subsequent processing.

  As described above, by specifying the specific model 126 to be used based on the type of the image, the information processing apparatus 100 can more accurately recognize the meaning included in the image.

  According to the fifth embodiment described above, the information processing apparatus 100 selects, from among a plurality of specific models 126 applied for each landscape (type) of an image, a specific model 126 applied for each landscape of an acquired image. Is more useful to the user by recognizing the specific meaning in the semantic area representing the specific meaning related to the feature included in the landscape and providing the recognized specific meaning to the user. Information can be provided.

<Sixth embodiment>
Hereinafter, the sixth embodiment will be described. The information processing system 1A of the sixth embodiment is different from each embodiment in further including a learning device 400. Hereinafter, differences from the first embodiment will be mainly described.

  FIG. 22 is a diagram illustrating an example of a functional configuration of an information processing system 1A according to the sixth embodiment. The information processing system 1A further includes, for example, a learning device 400 in addition to the functional configuration of the information processing system 1 of the first embodiment.

  The learning device 400 includes, for example, a learning unit 410 and learning data 412. The learning unit 410 generates the specific model 126 based on the learning data 412. Then, the learning device 400 transmits the generated specific model 126 to the information processing device 100.

  FIG. 23 is a diagram illustrating an example of the content of the learning data 412. For example, the learning data 412 is information in which an image indicating the price of fuel at a gas station is associated with correct answer data. The image indicating the price of the fuel at the gas station may be, for example, an image of a signboard displaying the price of the fuel of the same gas station (or an image of the same position or the same scenery). Alternatively, the signboard displaying the prices of the fuel at different gas stations may be an image of the sign. The correct answer data includes the identification information of the feature included in the image (for example, a sign indicating the price of regular gasoline), the information represented on the feature (for example, the price of regular gasoline), and the area corresponding to the feature. Are information associated with each other.

  The learning unit 410 machine-learns the learning data 412, and generates a specific model 126 that outputs regular gasoline price or information indicating that the price of regular gasoline is not included in the image when an image is input. That is, since the process of recognizing the same type of sign is performed, the learning process is performed exclusively for the sign, and the specific model 126 with high accuracy is generated.

  FIG. 24 is a diagram illustrating another example of the content of the learning data 412. For example, the learning data 412 is information in which an image indicating the parking availability is associated with the correct answer data. The image showing the parking availability may be, for example, an image of a signboard displaying the same parking availability (or an image of the same location or the same scenery) or a different parking location. The signboard displaying the vacancy status may be an imaged image. In the correct answer data, identification information of a feature included in the image (for example, a signboard indicating a parking vacancy), information indicating that the signboard is empty or full, and an area corresponding to the feature are associated with each other. Information. The learning unit 410 machine-learns the learning data 412 to generate a specific model 126 that outputs information indicating that a parking space or a signboard indicating parking free information is not included in the image when an image is input. I do.

  The predetermined feature may be a closed gas station, a closed convenience store, or an opened store. In this case, the correct answer data is information in which information such as a closed gas station, a closed convenience store, and a opened store is associated with information indicating these areas. When the learning unit 410 learns the learning data 412 and inputs an image, the image includes information that includes the closed gas station, the image includes information that includes the closed convenience store, and the image includes the opened store. A specific model 126 that outputs information indicating the fact is generated. This makes it possible to easily recognize the opening or closing of the store based on the image.

  Further, the predetermined feature may be a vehicle number, a person's face, or the like. In this case, the correct answer data is information in which an image including a vehicle number is associated with information indicating an area of a vehicle number, or an image including a person's face is associated with information indicating an area of a person's face. Information. Upon learning the learning data 412 and inputting an image, the learning unit 410 generates a specific model 126 that outputs information indicating whether or not the image includes a person's face and information indicating an area of the person's face. Thus, the face of the person can be easily recognized based on the image.

  In this way, by learning information such as the difference between the information of the individual signboards and the image of the image captured at the same position, it is possible to more accurately recognize the individual signboards and the semantic information of the predetermined position. A model 126 is generated. Since the specific model 126 is a model generated by learning the difference between image information, the information processing apparatus 100 uses the specific model 126 to recognize a specific meaning in the semantic region at the first time. 4 is an example of a process of recognizing a specific meaning in the meaning area based on a difference between the information on the meaning area of the image acquired in step S <b> 1 and the information on the meaning area acquired at the second time.

  According to the sixth embodiment described above, the learning device 400 learns the learning data 412 and generates the specific model 126. Using the specific model 126, the information processing apparatus 100 can accurately recognize a specific meaning in a meaning area representing a specific meaning related to a feature. As a result, more useful information for the user can be provided.

  As described above, the learning data 412 may be machine-learned to generate a rectangle derivation model. The rectangle derivation model is, for example, a model used by the second processing unit 106 for deriving a rectangle. Thereby, the information processing apparatus 100 can accurately derive a rectangle including a predetermined feature.

  Further, as described above, the learning data 412 may be machine-learned, and the determination model 45 may be generated. Accordingly, the information processing apparatus 100 can provide the imaging apparatus 20 with a model that can determine whether the imaging apparatus 20 is an image to be transmitted.

  According to each embodiment described above, the information processing apparatus 100 causes the communication unit 22 that communicates with another device and the imaging unit 24 that captures a landscape based on reference information indicating a reference for capturing a landscape to capture an image. A communication unit 102 for obtaining an image obtained by capturing an image of a landscape associated with position information from an imaging device 20 including an imaging control unit 28 and a position specifying unit 26 for obtaining a position; and an image obtained by the communication unit 102 To reduce the amount of communication by providing a third processing unit 108 that recognizes the meaning of the captured scenery, and an updating unit 114 that transmits the reference information to the imaging device 20 and causes the imaging device 20 to transmit the image. The required information can be acquired while performing.

  Further, according to each of the embodiments described above, the information processing apparatus 100 associates the position information specified by the position specifying unit from another device including the position specifying unit and the imaging unit, and A communication unit that acquires a captured image; and a semantic region representing a specific meaning related to a feature included in the landscape is extracted from the image acquired by the communication unit, and a specific meaning within the semantic region is extracted. And a providing unit 116 that provides the user with a specific meaning recognized by the third processing unit 108 in association with the position where the image including the meaning area is captured. Thereby, more useful information for the user can be provided.

  As described above, the embodiments for carrying out the present invention have been described using the embodiments. However, the present invention is not limited to these embodiments at all, and various modifications and substitutions may be made without departing from the gist of the present invention. Can be added.

10 vehicle, 20 imaging device, 22 communication unit, 24 imaging unit, 26 position specifying unit, 28 imaging control unit, 29 determining unit, 30 transmission control unit, 40, 40A storage unit, 42 {Correspondence information, 44} Reference information, 45} Judgment model, 46} Imaging application, 100} Information processing device, 102} Communication unit, 104} First processing unit, 106} Second processing unit, 108} Third processing unit , 110 tracking unit, 112 information management unit, 114 updating unit, 116 providing unit, 120 storage unit, 122 correspondence information, 124 reference information, 126 specific model, 128 recognition information, 200 News server, 300 ‥ terminal device, 400 ‥ learning device

Claims (9)

A communication unit that communicates with another device, an imaging control unit that causes an imaging unit that captures a landscape to capture an image, a position acquisition unit that acquires a position, and a determination unit that determines an image to be transmitted based on the image and the model information. An acquisition unit that acquires an image in which the landscape associated with the position information is captured,
A recognition unit that recognizes, from the image acquired by the acquisition unit, the meaning represented by the captured landscape;
A control unit that transmits the model information to the terminal device and causes the terminal device to transmit the image,
An information processing apparatus comprising: The image in which the landscape is captured is an image in which a predetermined feature is captured,
The information processing device according to claim 1. The model information is model information that outputs information indicating whether the image is a transmission target image when an image captured by the imaging unit is input,
The information processing device according to claim 2. The model information is generated by machine learning of learning data associated with an image to be transmitted and information indicating that the image is the image to be transmitted,
The information processing device according to claim 3. The predetermined feature is a gas station or parking,
The information processing apparatus according to claim 2. The terminal device is a terminal device mounted on a vehicle or an attachable portable terminal device.
The information processing device according to claim 1. Computer
A communication unit that communicates with another device, an imaging control unit that causes an imaging unit that captures a landscape to capture an image, a position acquisition unit that acquires a position, and a determination unit that determines an image to be transmitted based on the image and the model information. From the terminal device including, to obtain an image in which the scenery associated with the position information is captured,
Recognizing and recognizing the meaning of the captured landscape from the obtained image,
Transmitting the model information to the terminal device, causing the terminal device to transmit the image,
Information processing method. On the computer,
A communication unit that communicates with another device, an imaging control unit that causes an imaging unit that captures a landscape to capture an image, a position acquisition unit that acquires a position, and a determination unit that determines an image to be transmitted based on the image and the model information. From the terminal device including the location information is made to obtain an image of the scenery is associated,
From the obtained image, the recognition of the meaning of the captured scenery is recognized,
Causing the terminal device to transmit the model information, and causing the terminal device to transmit the image,
program. In the terminal computer,
By controlling the imaging unit that captures the landscape,
A position specifying part specifies the position of the landscape,
When an image transmitted by the information processing device and captured by the image capturing unit is input, an image captured by the image capturing unit is an image of the transmission target based on model information that outputs information indicating whether the image is a transmission target image. To determine if there is
Using a communication unit, causing the information processing apparatus to transmit the image of the transmission target in association with the position,
Application program.

Images