JP6997471B2 - Information processing system, information processing device, terminal device, server device, program, or method - Google Patents

Description

The technology disclosed in this application relates to an information processing system, an information processing device, a server device, a program, or a method.

In recent years, information processing using images has been performed, but the information in the images is not always properly used.

Japanese Unexamined Patent Publication No. 2009-75858 Japanese Unexamined Patent Publication No. 2004-318905

Therefore, various embodiments of the present invention provide an information processing system, an information processing apparatus, a terminal apparatus, a server apparatus, a program, or a method in order to solve the above-mentioned problems.

The first system according to one embodiment is
The first vehicle identification information that identifies the first vehicle in the first image captured by the first mobile terminal device, the first vehicle determination information that determines the driving state of the first vehicle based on the first image, and the first vehicle determination information. With the acquisition unit that acquires from the first mobile terminal device,
The second vehicle identification information that identifies the second vehicle in the second image captured by the second mobile terminal device, the second vehicle determination information that determines the driving state of the second vehicle based on the second image, and the second vehicle determination information. With the acquisition unit that acquires from the second mobile terminal device,
A determination unit that determines the identity between the first vehicle and the second vehicle by using the first vehicle identification information and the second vehicle identification information.
When it is determined that the first vehicle and the second vehicle are the same, the statistical information relating to the first vehicle is generated by using the first vehicle determination information and the second vehicle determination information. Statistical processing department and
A system equipped with.

The second system according to one embodiment is
A transmission unit that transmits statistical information relating to the first vehicle to the second mobile terminal device when it is determined that the first vehicle and the second vehicle are the same.
First system with.

The third system according to one embodiment is
A third acquisition unit for acquiring information related to an object in the first image from the first mobile terminal device.
One of the first and second systems.

The fourth system according to one embodiment is
The information related to the object is information related to the operation of the wiper, information related to the road, information related to the sidewalk, information related to the event on the roadway, information related to the advertisement, and / or for the vehicle in the first image. Including fuel information,
Third system.

The fifth system according to one embodiment is
An acquisition unit that acquires the third vehicle identification information that identifies the third vehicle in the image captured by the third mobile terminal device, the message transmitted by the third mobile terminal device, and the acquisition unit.
An acquisition unit that acquires the fourth vehicle identification information that identifies the fourth vehicle registered as its own vehicle in the fourth mobile terminal device, and
When the third vehicle and the fourth vehicle are determined to be the same vehicle, a transmission unit that transmits the message to the fourth mobile terminal device, and a transmission unit.
A system according to any one of 1 to 4.

The sixth system according to one embodiment is
An acquisition unit that acquires the third vehicle identification information that identifies the third vehicle registered as its own vehicle in the third mobile terminal device and the message transmitted by the third mobile terminal device.
The acquisition unit that acquires the fifth vehicle identification information that identifies the fifth vehicle related to the image captured by the fifth mobile terminal device, and the acquisition unit.
When the third vehicle and the fifth vehicle are determined to be the same vehicle, a transmission unit that transmits the message to the fifth mobile terminal device, and a transmission unit.
A system according to any one of 1 to 4.

The seventh system according to one embodiment is
After it is determined that the first vehicle and the second vehicle are the same, the transmission unit transmits statistical information relating to the first vehicle within a predetermined time.
The second system.

The eighth system according to one embodiment is
The first mobile terminal device and the second mobile terminal device are different mobile terminal devices.
Any one of the first to seventh systems.

The ninth system according to one embodiment is
The first acquisition unit acquires a moving image including the first image from the first mobile terminal device.
Any one of the first to eighth systems.

The tenth system according to one embodiment is
The moving image is a compressed moving image.
Ninth system.

The eleventh system according to one embodiment is
The first vehicle determination information is information generated by the machine-learned device in the first mobile terminal device.
Any one of the first to ten systems.

The twelfth system according to one embodiment is
The 1-vehicle determination information and the 2nd vehicle determination information include sudden steering, sudden acceleration, and / or sudden braking, respectively.
Any one of the first to eleven systems.

The thirteenth method according to one embodiment is
The computer
The first vehicle identification information that identifies the first vehicle in the first image captured by the first mobile terminal device, the first vehicle determination information that determines the driving state of the first vehicle based on the first image, and the first vehicle determination information. And the acquisition step of acquiring from the first mobile terminal device,
The second vehicle identification information that identifies the second vehicle in the second image captured by the second mobile terminal device, the second vehicle determination information that determines the driving state of the second vehicle based on the second image, and the second vehicle determination information. And the acquisition step of acquiring from the second mobile terminal device,
A determination step for determining the identity between the first vehicle and the second vehicle,
When the first vehicle and the second vehicle are the same, a statistical processing step for generating statistical information relating to the first vehicle by using the first vehicle determination information and the second vehicle determination information. ,
How to run.

The fourteenth program according to one embodiment is
Computer,
The first vehicle identification information that identifies the first vehicle in the first image captured by the first mobile terminal device, the first vehicle determination information that determines the driving state of the first vehicle based on the first image, and the first vehicle determination information. Is acquired from the first mobile terminal device,
The second vehicle identification information that identifies the second vehicle in the second image captured by the second mobile terminal device, the second vehicle determination information that determines the driving state of the second vehicle based on the second image, and the second vehicle determination information. Is acquired from the second mobile terminal device,
A determination means for determining the identity between the first vehicle and the second vehicle,
A statistical processing means for generating statistical information relating to the first vehicle by using the first vehicle determination information and the second vehicle determination information when the first vehicle and the second vehicle are the same.
A program that operates as.

The fifteenth program according to one embodiment is
A program for operating a computer as any one of the first to twelfth systems.

According to one embodiment of the present invention, image information can be utilized more appropriately.

FIG. 1 is a diagram illustrating a situation example in which the system according to one embodiment is applied. FIG. 2 is a block diagram showing the functions of the system according to the embodiment. FIG. 3 is a block diagram showing the functions of the system according to the embodiment. FIG. 4 is an example of a data structure used by the system according to the embodiment. FIG. 5 is an example of a data structure used by the system according to the embodiment. FIG. 6 is an example of a data structure used by the system according to the embodiment. FIG. 7 is an example of a data structure used by the system according to the embodiment. FIG. 8 is an example of a data structure used by the system according to the embodiment. FIG. 9 is an example of a data structure used by the system according to the embodiment. FIG. 10 is an example of a data structure used by the system according to the embodiment. FIG. 11 is an example of a data structure used by the system according to the embodiment. FIG. 12 is an example of a data structure used by the system according to the embodiment. FIG. 13 is an example of a data structure used by the system according to the embodiment. FIG. 14 is an example of a data structure used by the system according to the embodiment. FIG. 15 is an example of a data structure used by the system according to the embodiment. FIG. 16 is an example of a data structure used by the system according to the embodiment. FIG. 17 is an example of a data structure used by the system according to the embodiment. FIG. 18 is an example of a data structure used by the system according to the embodiment. FIG. 19 is an example of a data structure used by the system according to the embodiment. FIG. 20 is an example of a data structure used by the system according to the embodiment. FIG. 21 is an example of a data structure used by the system according to the embodiment. FIG. 22 is an example of a transition diagram used by the system according to the embodiment. FIG. 23 is a diagram showing a display example of the system according to the embodiment. FIG. 24 is a diagram showing a display example of the system according to one embodiment. FIG. 25 is a diagram showing a display example of the system according to one embodiment. FIG. 26 is a diagram showing a display example of the system according to one embodiment. FIG. 27 is a diagram showing a display example of the system according to one embodiment. FIG. 28 is a diagram showing a display example of the system according to one embodiment. FIG. 29 is a diagram showing a display example of the system according to one embodiment. FIG. 30 is a diagram showing a display example of the system according to one embodiment. FIG. 31 is a diagram showing a display example of the system according to one embodiment. FIG. 32 is a diagram showing a display example of the system according to one embodiment. FIG. 33 is a diagram showing a display example of the system according to the embodiment. FIG. 34 is a diagram showing a display example of the system according to the embodiment. FIG. 35 is an example of a data structure used by the system according to the embodiment. FIG. 36 shows an example of the flow of the system according to one embodiment. FIG. 37 is a diagram showing a display example of the system according to one embodiment. FIG. 38 is a block diagram showing an overall configuration according to the system according to one embodiment. FIG. 39 is a block diagram showing an overall configuration of the system according to the embodiment. FIG. 40 shows an example of the flow of the system according to one embodiment. FIG. 41 is a block diagram showing a configuration of an information processing apparatus according to an embodiment.

1. 1. Introduction An example system may include one or more terminal devices used by system users and one or more management systems used by system administrators. The user may be a passenger in the vehicle, may be a driver, or may be a passenger. The terminal device may be fixed to the vehicle or may be removable. FIG. 1 shows an example of a situation in which an example system is used. In this figure, arrow 001 indicates the traveling direction of the vehicle. The vehicle 002 indicates a vehicle using the user mobile terminal according to the example system. It is assumed that the vehicles 003A to I use the terminal device according to the example system as a drive recorder. It should be noted that some of these vehicles do not have to use the terminal device according to the example system. It is assumed that each vehicle is operated along the lane boundary line 004A and the lane boundary line 004B.

The user terminal device in the vehicle 002 may be able to identify each vehicle by the image pickup device. For example, when the angle of view of one image pickup device of the user terminal device in the vehicle 002 is wide, three vehicles of vehicles 003A, 003D, and 003F may be included in the image. Similarly, when the angle of view of the other image pickup device that captures the report facing the one image pickup device of the user terminal device in the vehicle 002 is wide, the four vehicles of the vehicles 003C, 003E, 003H, and 003I are images. May be included in. In some cases, such as the vehicle 003I, the image pickup device of the user terminal device in the vehicle 002 may be in the field of view. Further, since the vehicles 003B and 003G travel sideways of the vehicle 002, the front side is the image pickup direction of one image pickup device and the rear side is the image pickup direction of the one image pickup device in the vehicle 002. When the image pickup direction is set to the image pickup direction of the image pickup device, even if the vehicle does not enter the field of view while traveling right beside the vehicle 002, it enters the field of view of the image pickup device of the user terminal device in the vehicle 002 by moving back and forth while traveling. There may be. In this way, the vehicle in the field of view in the image pickup apparatus may be referred to as a "surrounding vehicle" in the documents of the present application, and when the terminal device according to the example system is used in the surrounding vehicle. When such a terminal device is viewed from the user terminal device in the user vehicle 002, it may be referred to as a "peripheral terminal device" in the documents of the present application. Further, among the surrounding vehicles, the vehicle in front of the user vehicle may be referred to as "front vehicle" in the document of the present application, and the vehicle behind the user vehicle may be referred to as "rear vehicle" in the document of the present application. Further, when the terminal device related to the example system in the front vehicle and the rear vehicle is used as an example system, the terminal devices used in each vehicle are referred to as "front terminal device" and "rear", respectively. It may also be called a "terminal device". Since the surrounding vehicle is included in the field of view in the above-mentioned imaging device, information for identifying the surrounding vehicle may be acquired by using the information related to the license plate described later.

The terminal device may include one or more accelerometers. A plurality of accelerometers may be provided in the terminal device so as to be able to measure accelerations in directions orthogonal to each other. A plurality of accelerometers may be provided in the terminal device so that the acceleration of the direction imaged by the image pickup device and the direction orthogonal to the image pickup direction can be measured.

The terminal device may include one or more image pickup devices. In the latter case, the terminal device may be equipped with an image pickup device so as to be able to image in opposite directions. Further, the terminal device may be installed in the vehicle so that one or more of the plurality of image pickup devices have the front of the vehicle as the image pickup direction. Further, the terminal device may be installed in the vehicle so that the other one or a plurality of image pickup devices among the plurality of image pickup devices have the rear side of the vehicle as the image pickup direction. The installation in the vehicle may be detachably installed in the vehicle or may be fixed.

The display direction of the display device in the terminal device may be the same as the image pickup direction of one image pickup device. Further, the display direction of the display device may be the rear of the vehicle.

The terminal device may have one or more position measuring devices. The position measuring device may be a position measuring function using GPS or a base station.

The terminal device may include a communication device. The communication device may be capable of communicating with the image pickup device installed in the vehicle in which the terminal device is installed. The communication device may be a wireless type or a wired type. In the case of wireless type, it may be WIFI, BLUETOOTH or the like. The image pickup device installed in the vehicle may be an image pickup device incorporated in the vehicle or an image pickup device installed in the vehicle.

The terminal device may be a portable terminal device that can be carried by a person. For example, the portable terminal device may be a smartphone, a PDA, or the like.

2. 2. Functions of the system of this example Next, the functions of the system of this example will be described with reference to FIGS. 2 and 3. FIG. 2 is a block diagram showing a specific example of the function related to the management system of this example, and FIG. 3 is a block diagram showing a specific example of the function related to the terminal device. As another example, a part of the functions related to the management system may be executed in the terminal device. In this case, the system can process based on the information acquired from a plurality of terminal devices, but when the terminal device executes the function in FIG. 2, one terminal device is used based on the information acquired by the terminal device. The function may be executed as the acquired information. Further, the information obtained by executing the information in the one terminal device may be transmitted to the management system, and further combined with the information acquired from the other terminal devices in the management system, and processed such as summing and averaging. In the documents of the present application, the term system is used as a superordinate concept of a management system and a terminal device, and the system may include a management system and may not include one or a plurality of terminal devices, or one or a plurality of terminal devices. It may include the management system and may not include the management system, or may include both the management system and one or more terminal devices.

The vehicle in which the terminal device is installed or placed is referred to as a self-vehicle, the information related to such a vehicle is referred to as self-vehicle information, and the vehicle included in the image captured by the image pickup device related to the terminal device is referred to as another vehicle. That is. When a user of the system uses a plurality of vehicles by owning or sharing a plurality of vehicles, the term "own vehicle" means one or more of these terminal devices may be installed or placed. You may point to the vehicle of.

2.1. Functions in the terminal device
2.1.1. Image acquisition unit The image acquisition unit has a function of acquiring an image. The image may be a moving image or a still image. In the document of the present application, an image is treated as a superordinate concept of a moving image and a still image.

When acquiring a moving image as an image, the image acquisition unit may store the moving image in a file format divided by a predetermined capacity. This is because the convenience of communication is improved when the capacity per file is smaller than that of one large file.

In addition, the image acquisition unit may acquire the captured location and / or the imaging time of the acquired moving image for each predetermined period, and store the acquired moving image in association with each predetermined period of the corresponding moving image. With such a configuration, there is an advantage that information on the imaged location and / or the imaged time can be acquired at predetermined intervals in the moving image.

The image acquisition unit may acquire an image from an image pickup device inside the terminal device, or may acquire an image from an image pickup device outside the terminal device. In this case, it may be connected to an image pickup device outside the terminal device by wire or wirelessly, and in the case of wireless communication, image information may be acquired by a connection method such as WIFI or BLUETOOTH.

In addition, the image acquisition unit may acquire sound. The image acquisition unit may also acquire audio when acquiring a moving image as an image. This has the advantage that, for example, sounds at the time of an accident, sudden braking, sharp curves, and the like can be acquired and recorded. Further, the image acquisition unit may acquire the sound itself separately from the moving image. As a superordinate concept of image and sound, it may be referred to as "image, etc." in the documents of the present application.

2.1.2. Own vehicle information generation unit The own vehicle information generation unit may have a function of acquiring information related to the own vehicle.

The information related to the own vehicle is the information related to the license plate related to the own vehicle, the message to be conveyed to the surrounding vehicle, the vehicle in front, and / or the vehicle behind the own vehicle (referred to as "transmission message" in the documents of the present application). There is also), may be included.

The information related to the license plate may be an image of the license plate of the vehicle, may be information on a number that identifies the vehicle generated from the image of the image of the imaged license plate, or may be information on a number that identifies the vehicle, depending on the user or the like. It may be the information of the number that identifies the entered vehicle. Further, in this case, the area name may be selected depending on the mode of selecting from those given in advance. When the information related to the license plate is generated from the image, the self-vehicle information generation unit may have a function of analyzing the image and extracting the information of the number specifying the vehicle.

The information related to the license plate may be registered one or more in the terminal device. Among the information related to one or a plurality of license plates registered in advance, the information related to one license plate is selected and may be used. Further, when the user can use a plurality of own vehicles, the user may select the own vehicle to be actually used and select the information related to the license plate related to the actually used own vehicle. .. This has the advantage that even when the user may use multiple own vehicles, the information related to the license plate can be used quickly without registering a new license plate each time the user uses it. ..

The transmitted message may be in various modes as long as the message of the own vehicle can be transmitted.

For example, the transmitted message may be a character string input by the user before or at the time of boarding. Such a character string may be any character input by the user.

Further, for example, the transmission message may be a predetermined character string. In this case, the user may be selected from one of the plurality of predetermined character strings. In this case, the transmission message may be, for example, a character string indicating that the vehicle is traveling slowly, a character string indicating that the driver is elderly, a character string indicating that the driver is a beginner, a character string indicating that a child is on board, or a character string indicating that the driver is in a hurry. good. These transmitted messages may be stored in advance or may be input in the terminal device.

Further, the transmission message may have a transmission message to the vehicle in front and a transmission message to the vehicle behind.

Further, a plurality of transmitted messages may be possessed, and one transmitted message may be selected from a plurality of transmitted messages to be used at an appropriate timing of the user. The selection of the transmitted message may be selected by voice or by gesture. The message conveyed by the gesture may be, for example, gratitude, apology, or giving up the other party. The gratitude is assumed to be, for example, gratitude for being able to enter the vehicle line when changing lanes, or gratitude for being given the lead when turning right or left. Apologies are also expected to be presented in place of gratitude in cases equivalent to gratitude. Then, it is assumed that the other party is handed over to encourage the other vehicle to take the lead when turning left or right or changing lanes. In addition, these gestures may include gestures or voices that identify the other vehicle showing the gesture, such as a vehicle in front, a vehicle behind, a vehicle on the right side, and a vehicle on the left side, before and after the message content.

The user of the terminal device according to the example system, the driver of the vehicle into which the terminal device is brought in, and the vehicle owner of the vehicle may be different or the same. Further, when the user of the terminal device according to the example system uses a plurality of vehicles, the user may have a function of specifying the vehicle to be boarded.

FIG. 4 is an example of a data structure in which information related to a license plate related to a user's vehicle and a message when the user's vehicle is boarded are stored in association with each other. Having such a data structure has an advantage that the user can select the user's vehicle to actually ride and can use the information related to the license plate according to the selected user's vehicle. ..

In addition, the information related to the own vehicle includes information indicating the driving status of the own vehicle (sometimes referred to as "own vehicle driving status information" in the documents of the present application) and / or information indicating the position of the own vehicle "own vehicle". "Position information" may be included. The own vehicle driving situation information may include sudden acceleration, sudden braking, sudden steering, speed, and / or acceleration of the own vehicle.

The sudden acceleration, sudden braking, and / or sudden steering of the own vehicle may be determined by using the function in the terminal device. For example, a sensor in the terminal device may determine the sudden acceleration, sudden braking, and / or sudden steering of the own vehicle. More specifically, an acceleration sensor as a sensor may determine sudden acceleration, sudden braking, and / or sudden steering when the acceleration is higher or lower than a predetermined acceleration. The accelerometer may be capable of measuring each of the three-dimensional accelerations. When the acceleration in the same direction as the front direction of the vehicle is higher than the predetermined acceleration, it is determined that the acceleration of the own vehicle is sudden, and when the acceleration in the direction opposite to the front direction of the vehicle is higher than the predetermined acceleration, the self When it is determined that the vehicle is suddenly braked and the acceleration in the lateral direction of the vehicle is higher than a predetermined acceleration, it may be determined that the vehicle is a steep steering wheel.

Further, the speed and acceleration of the own vehicle may be measured by measuring the acceleration by a sensor in the terminal device and using the acceleration.

In addition, the self-vehicle driving status information may include information relating to the distance of the self-vehicle from the vehicle in front (sometimes referred to as "distance information" in the documents of the present application). The own vehicle information generation unit may generate distance information using the information related to the image generated by the image information generation unit described later. Such distance information may include information for estimating the distance between the own vehicle and the vehicle in front, information corresponding to the distance, and / or information for determining that the vehicle has approached using the same. The information determined to be approaching is, for example, the size in the image when the vehicle in front is viewed from the rear as information corresponding to the distance between the own vehicle and the vehicle in front, and the size from the rear of the vehicle. When the size is larger than a predetermined size, it may be determined that they are approaching. The size from the rear of the vehicle may be the width or height of the vehicle. In addition, since the vehicle height can be changed depending on the vehicle type in the vehicle, the presence or absence of approach may be determined by using the vehicle height according to the vehicle type of the vehicle in the image. In this case, when the height of the vehicle in front obtained from the image is higher than the height of a predetermined vehicle height set according to the vehicle type, it may be determined that the vehicle has approached. On the other hand, the vehicle width does not differ greatly depending on the vehicle type, so there is an advantage that information processing based on the vehicle type is not required, and the vehicle width of the vehicle in front obtained from the image is longer than the predetermined vehicle width. It may be determined that the vehicle has approached.

Further, the self-vehicle information generation unit is provided with a plurality of ranks for each of the degree of sudden acceleration, sudden braking, sudden steering, and approach information, and one of these may be selected. For example, when the rank is 3, threshold values are set as low, medium, and high, respectively, and the rank may be determined by comparing them with the obtained information. The number of ranks is not limited to 3, and may be any number.

Further, the self-vehicle information generation unit may store the sudden acceleration, sudden braking, sudden steering, and / or approach information of the own vehicle in association with the situation in which these are determined. The determined situation may be, for example, the time and / or the place where the sudden acceleration, the sudden braking, the sudden steering wheel, and / or the approach information of the own vehicle occur.

The own vehicle information generation unit may generate statistical information regarding the driving situation of the own vehicle. The statistical information may be, for example, the number of times of sudden acceleration, sudden braking, sudden steering, and / or approach information in each predetermined item of travel. The predetermined item may be within a predetermined unit time, one run, one day run, a predetermined period, and the like. In addition, one run is, for example, from the start of the engine to the end of the engine, the time between running and running is longer than a predetermined period, or the time from the start to the stop of the software related to the system of one example. Or it may be from the start of recording of the moving image to the end of recording.

Further, as statistical information, the above-mentioned number of times may be generated in time series. Further, a graph may be generated for each, sudden acceleration, sudden braking, sudden steering, and / or approach information, with the horizontal axis as the timing and the vertical axis as the number of times of application, and may be used as a drive report.

FIG. 5 is an example of statistical information data regarding the driving situation of the own vehicle. This is an example in which each statistical information in one day's driving is recorded for each predetermined item. The IDs are 001 to 004, which is an example in which the total number of times is stored as each statistical information for running for 4 days.

As the own vehicle position information, the position information may be acquired from the GPS inside the terminal device, or the position information may be acquired from the GPS outside the terminal device. In this case, it may be connected to an image pickup device outside the terminal device by wire or wirelessly, and in the case of wireless communication, position information may be acquired by a connection method such as WIFI or BLUETOOTH.

The self-vehicle information generation unit generates a green signal when the image information generation unit described later generates information that the own vehicle does not move and that the traffic light targeting the own vehicle is a green light. However, in order to indicate that the own vehicle has not started, information prompting the own vehicle to start may be generated. Further, the self-vehicle information generation unit generates information that the own vehicle does not move and that the distance information with the vehicle in front is larger than a predetermined distance in the image information generation unit described later. If this is the case, information prompting the departure of the own vehicle may be generated in order to indicate that the own vehicle has not started even though the vehicle in front has started. In the latter case, if a traffic light targeting the own vehicle is detected in the image, a condition that it is a green light may be added.

Further, when the self-vehicle information generation unit generates information that the own vehicle is moving and that there is a stop sign or display in the image information generation unit described later, the self-vehicle information generation unit generates information that the vehicle is moving. It may be determined that the suspension violation has occurred, and information indicating the suspension violation may be generated.

Further, when the self-vehicle information generation unit generates information that the own vehicle is moving and the information indicating the continuation of the red light is generated in the image information generation unit described later, the red light is ignored. A determination may be made and information indicating that the red light is ignored may be generated.

In addition, the self-vehicle information generation unit uses the information indicating the stop sign violation and the information ignoring the red light to add these information indicating the stop violation and / or the red light to the above-mentioned statistical information. It may include the total number and / or the average number of information to be ignored. In addition, the information indicating these stop violations and / or the information ignoring the red light may include the location information in which they occur and / or the information related to the time.

2.1.3. Image information generation unit The image information generation unit has a function of generating information related to an image by using an image. The information related to the image may be information using various objects in the image (sometimes referred to as "objects" in the documents of the present application) and various situations, and the type and range thereof are not limited, but for example, The following information can be mentioned. Further, the image information generation unit may generate information after identifying a target for generating information in the image. Further, the image information generation unit may generate information related to the image by using the image and other information. As other information, information related to an image may be generated by using information related to at least a part of an image or the like acquisition unit and a self-vehicle information generation unit. The information related to the image or the like acquisition unit may be, for example, the information at the time of acquiring the image or the like, the image or the like, and the information related to the own vehicle information generation unit may be, for example, the own vehicle position information or the like. However, it is not limited to these.

The information related to the image may be information related to one or more other vehicles in the image. Information related to other vehicles is referred to as information that identifies another vehicle (sometimes referred to as "vehicle-specific information" in this document) and information that indicates the driving status of another vehicle (in this application, "other vehicle driving status information"). Sometimes) and so on.

The vehicle specific information may be information related to the vehicle in the image. The information relating to the vehicle in the image may include information relating to the license plate of the vehicle in the image, or may include information such as the vehicle type, color, and options of the vehicle in the image. The information related to the license plate may be the same as the information related to the license plate related to the user vehicle described above.

The other vehicle driving status information may be any information indicating the driving status of the other vehicle, and may be, for example, sudden acceleration of the vehicle, sudden braking of the vehicle, and / or sudden steering wheel of the vehicle.

The sudden acceleration of the vehicle in the image may be determined to be sudden acceleration when the size of the vehicle in the image changes slightly by a predetermined ratio or more within a predetermined period. Further, the determination of sudden acceleration may be made using the acceleration of the own vehicle. That is, when the acceleration of the own vehicle in the forward direction is within a predetermined range and the size of the vehicle in the image changes slightly by a predetermined ratio or more within a predetermined period, it is determined to be sudden acceleration. It's okay. In this case, there is an advantage that the determination accuracy is improved.

The sudden braking of the vehicle in the image may be determined to be sudden acceleration when the size of the vehicle in the image changes significantly by a predetermined ratio or more within a predetermined period. Further, the determination of sudden acceleration may be made using the acceleration of the own vehicle. That is, when the acceleration of the own vehicle in the rear direction is within a predetermined range and the size of the vehicle in the image changes significantly by a predetermined ratio or more within a predetermined period, it is determined to be a sudden braking. It's okay. In this case, there is an advantage that the determination accuracy is improved.

The steep steering wheel of the vehicle in the image means that the area of the vehicle on the side in the image increases at a speed of a predetermined ratio or more, and the shape of the side of the vehicle in the image is the shape when the vehicle is seen from the side. If any or a combination of the speeds approaching the above and the speeds equal to or higher than a predetermined ratio, it may be determined to be sudden acceleration. Further, in combination with these, it may be determined that the vehicle is suddenly accelerated when the size of the vehicle in the image changes small at a speed of a predetermined ratio or more within a predetermined period. The size of the vehicle may be the length of the width of the vehicle.

Further, the other vehicle driving status information may be stored in association with the date and time when the acquired target image was captured. This has the advantage of clarifying the date and time related to the driving status information of other vehicles.

Further, the other vehicle driving status information may be stored in association with the captured position information of the acquired target image. This has the advantage of clarifying the position related to the driving status information of other vehicles.

Further, when a plurality of vehicles are identified in one image, the other vehicle driving status information related to each vehicle may be stored in association with each vehicle identification information. In this case, there is an advantage that the driving status information is organized and accessible for each vehicle in the image.

FIG. 6 is an example in which information is generated based on a plurality of images. This figure shows the data structure, and the vehicle ID, other vehicle driving status information, date and time, and position are stored in association with the image ID. The vehicle ID may be associated with other vehicle specific information in other data structures. Here, the vehicle ID may be sequentially attached to the vehicle specified in the image, or the corresponding vehicle may be collated with the vehicle identification information to which the vehicle ID has been attached in the past. ID may be used. The storage unit may store a set of information of the vehicle identification information in the image captured in the past by the image pickup device related to the terminal device and the vehicle ID attached to the vehicle. In addition, the information of such a set may be stored for a predetermined period. Further, the information of such a set may be stored in the ring buffer. It is not essential to use the vehicle ID, and the vehicle identification information may be used directly instead of the vehicle ID. Further, in this figure, each driving status information is associated with a different image ID, but for example, when other vehicle driving status information is generated at the same time, a plurality of vehicles are generated for one image. And other vehicle driving status information corresponding to them may be associated. Such statistical information relating to the vehicle in the image may be referred to as "vehicle statistical information in the image" in the documents of the present application.

<Information on the movement of your vehicle>
Further, the image information generation unit may generate information on the movement of the own vehicle by using the time-series images. For example, the presence or absence of movement of the own vehicle may be determined by using the movement in the corresponding traffic light, sign, and / or landscape in the still image relating to the plurality of adjacent frames in the moving image. The information regarding the movement of the own vehicle may include information on the presence or absence of the own vehicle. Such information is that the vehicle is keeping a pause, the traffic light for the own vehicle turns blue, or the traffic light for the own vehicle is blue when the vehicle in front is moving. May be used if the vehicle departed.

<Information indicating the continuation of the red light>
Further, the image information generation unit includes a state in which the traffic light for the own vehicle is a red light in the first image, and the traffic light of the red light continues in the frame following the first image in time series. It may be determined whether or not the image is out of the image without a green light after being displayed. When such a determination is made, information indicating the continuation of the red light may be generated.

Here, the specification of the traffic light for the own vehicle may be made identifiable by learning by machine learning using an image including the traffic light.

<Information on suspension instructions>
Further, the image information generation unit may detect such information and generate information of a stop instruction when there is a stop sign or display in the road sign or road sign of the object in the image.

Further, the information related to the image may be information related to various objects detected in the image.

<Information related to wiper operation>
The information related to the target may include information related to the operation of the wiper. As the information related to the operation of the wiper, the information on the operation of the wiper related to the own vehicle may be used, or the operation of the wiper related to another vehicle may be used. The information related to the operation of the wiper may include information related to the operating status of the wiper. The information related to the operation status of the wiper may be the presence / absence of movement of the wiper and / or the speed of movement of the wiper. Further, the speed of movement of the wiper may be information that identifies one of a plurality of ranks indicating the speed of movement of the wiper. The plurality of ranks indicating the speed of movement of the wiper may be, for example, intermittent, slow, medium, fast, etc., but are not limited to these.

Further, the information relating to the operation of the wiper may include information relating to the operation status of the wiper and the position information obtained by capturing the image obtained from the information relating to the operation status of the wiper.

Further, the information relating to the operation of the wiper may include information relating to the operation status of the wiper and information relating to the operation status of the wiper when an image obtained by acquiring the information is captured.

FIG. 7 is an example of information relating to the operation of the wiper. In addition, this figure is an example of the data that separates the presence or absence of the movement of the wiper and the rank of the movement. May be good. Further, the data without the movement of the wiper may not be included in the data, and the data may be acquired only when the wiper moves.

<Information related to roads>
The information relating to the subject may include information relating to the road. The information related to the road may include the information related to the road condition. Information relating to road conditions may include information relating to road signs, information relating to road markings, information relating to traffic lights, and / or information relating to roadways.

The information relating to the road sign may include information on the presence / absence of the sign, the content of the sign, and / or the abnormality of the sign. Abnormalities in the sign are the presence or absence of something that obstructs the sign (eg, at least part of the sign is invisible by the tree) and / or the abnormality in the sign itself (eg, at least part of the sign is damaged). It may contain the information of (the state of being).

The information relating to the road marking may include the presence / absence of the road marking, the content of the road marking, and / or the abnormality of the road marking. The anomaly of the road marking may include information on the presence or absence of something that obstructs the marking and / or the anomaly of the marking itself.

The information relating to the traffic light may include information that the traffic light has been identified (information that there is a traffic light) and / or information that the traffic light is abnormal. Anomalies in a traffic light include anomalies in the appearance of the traffic light (for example, information that a part of the traffic light cannot be seen due to a tree or the like) and / or a failure of the traffic light (for example, a part of the traffic light is damaged). ), May include information.

The information related to the roadway may be an abnormality of the roadway and / or the width of the lane. In addition, in the document of this application, a lane may be a part of a band-shaped roadway (excluding a secondary road) provided for safe and smooth passage of a parallel car, and when the roadway is separated by a line, It may be a part through which a car or the like can pass. The roadway abnormality may be a lane boundary abnormality, a foreign object on the roadway, and / or a roadway destruction. The abnormality of the lane boundary line may be a part or all of the lane boundary line is missing. The foreign matter on the roadway may be, for example, a falling object on the roadway, a fallen tree, a fallen utility pole, or the like. Destruction of the roadway may be an abnormality in the shape of the roadway, such as a collapse of the roadway. In addition, the information related to the roadway may include information on parking on the street.

Further, the information related to the road may include information related to the road condition and the position information obtained by capturing the image obtained by acquiring the information related to the road condition.

Further, the information related to the road may include information related to the road condition and information related to the road condition in association with each other when the image obtained by acquiring the information is captured.

An example of information relating to a road is shown in FIG. Depending on the target, the information related to the road may be always acquired when it is detected in the image, or may be acquired only when the information detected in the image satisfies a predetermined condition. For example, the predetermined condition may be a predetermined target, or only a condition in which an abnormality is detected.

Further, as the information related to the road, the information related to the disaster may be generated. Information relating to a disaster may include information on foreign matter on the roadway and / or destruction of the roadway. In this case, the information related to the road may include information related to the disaster and the position information obtained by capturing the image obtained by acquiring the information related to the disaster, or may include the information related to the disaster and the information related to the disaster. It may include an image associated with the time when the acquired image is captured. FIG. 9 is an example in which the information related to the road includes the information related to the disaster.

When the information related to these roads is transmitted to the system related to automatic driving, it may be used to determine the propriety and priority of automatic driving.

<Information about sidewalks>
The information relating to the subject may include information relating to the sidewalk. The information on the sidewalk may include information on the condition of the person on the sidewalk. Information on the condition of a person on the sidewalk may include information on what the person wears. What is worn by such a person may include an umbrella (in the documents of the present application, the term "wearing" shall include the meaning of "wearing" when the subject is an umbrella). Also, what is worn by a person may include a cloak. Also, what is worn by a person may include a half sleeve and a long sleeve.

Further, the information related to the sidewalk may include information on the state of the person on the sidewalk and the position information obtained by capturing the image obtained by acquiring the information on the state of the person on the sidewalk.

Further, the information related to the sidewalk may include information on the state of a person on the sidewalk and information on the state of a person on the sidewalk in association with each other when an image obtained by acquiring the information is taken.

An example of information relating to the sidewalk is shown in FIG. Depending on the target, the information related to the sidewalk may be always acquired when it is detected in the image, or may be acquired only when the information detected in the image satisfies a predetermined condition. For example, as predetermined conditions, only those that are predetermined as information on the state of a person or those in which a predetermined number of people or more are detected in one image are predetermined among the people in one image. Only the detection of people less than or equal to the number of people may be possible.

When the information related to these sidewalks is transmitted to the system related to automatic driving, it may be used to determine the propriety and priority of automatic driving.

<Information on how crowded people are>
The information relating to the subject may include information relating to the degree of congestion of people. The information on the degree of congestion of people may include the number of people in a predetermined area. In addition, the information relating to the degree of congestion of people includes the number of such people and the position information in which the image used for determining the number of such people is captured or the information indicating such a predetermined area in association with each other. good. In addition, the information on the degree of congestion of people may be included in association with the number of such people and the time when the image in which the number of such people is determined is captured. The predetermined area may be predetermined or may be defined at the time of moving image shooting. Examples of the predetermined area include a predetermined area using map information and location information. Further, what is defined at the time of moving image shooting may be a region defined by using a predetermined distance from a coordinate value such as GPS, or may be a region defined by using the elapsed imaging time.

When the information on the degree of congestion of these people is transmitted to the system related to automatic driving, it may be used to determine the propriety and priority of automatic driving.

<Information about events on the road>
In addition, the information relating to the subject may include information relating to an event on the roadway. Information about roadway events may be information about construction and / or accidents. Information about the construction may include the presence or absence of construction and / or the scheduled completion deadline of the construction. The presence or absence of construction may be determined by detecting information such as stopping of the construction vehicle, installation of an instant traffic light, and vehicle guidance by a person involved in the construction in the image. Information about the accident may include the presence or absence of an accident and the magnitude of the accident. The presence or absence of an accident may be determined by the presence or absence of an accident vehicle and / or the presence or absence of police personnel. The detection of construction personnel and police personnel may be detected by the clothes and belongings that characterize them.

Further, the information regarding the event on the roadway may include information relating to the construction and / or the accident and the position information obtained by capturing the image obtained from the information regarding the construction and / or the accident.

Further, the information regarding the event on the roadway may include information relating to the construction and / or the accident and the information related to the construction and / or the accident when the acquired image is captured.

An example of information about an on-road event is shown in FIG.

Information about these on-road events, when transmitted to a system for autonomous driving, may be used to determine the propriety and priority of autonomous driving.

<Information related to advertising>
Further, the information related to the target may be information related to the advertisement. The information related to the advertisement may include the advertisement status information. The advertisement status information may include a specific brand of the advertisement, the advertiser of the advertisement, the field to which the advertisement belongs, the size of the advertisement, and / or the presence or absence of an abnormality in the advertisement. The presence or absence of an abnormality in the advertisement may include a state in which the advertisement is hidden by an object such as a tree, and / or damage to the advertisement itself. Advertising status information may be acquired based on the image.

Further, the information related to the advertisement may include an association between the advertisement status information and the position information obtained by capturing the image obtained from the advertisement status information.

Further, the information related to the advertisement may include an association between the advertisement status information and the time when the image obtained from the advertisement status information is captured.

An example of the information related to the advertisement is shown in FIG. As shown in this figure, the size of the advertisement may be configured to select one of a plurality of predetermined ranks for the size.

<Information on fuel for vehicles>
Further, the information relating to the target may be information relating to fuel for a vehicle. The information relating to the fuel for the vehicle may include the information relating to the status of the fuel for the vehicle. The information on the status of the fuel for the vehicle may include the type of fuel for the vehicle and the amount corresponding to the type of fuel for the vehicle. The type of fuel for the vehicle may include high-octane gasoline, regular gasoline, light oil, and / or kerosene, and the like. Information on the status of fuel for vehicles may be obtained from the presentation of places where fuel for vehicles is provided, such as service stations including gas stations, gas stations, etc., in the image.

Further, the information relating to the fuel for the vehicle may include information relating to the status of the fuel for the vehicle and the position information obtained by capturing the image obtained by acquiring the information relating to the status of the fuel for the vehicle. ..

Further, the information relating to the fuel for the vehicle may include information relating to the status of the fuel for the vehicle and information relating to the status of the fuel for the vehicle when an image obtained by acquiring the information is captured.

FIG. 13 is an example of information relating to fuel for a vehicle.

When these information relating to fuel for vehicles is transmitted to a system relating to autonomous driving, it may be used to determine whether or not autonomous driving is possible or prioritized.

The image information acquisition unit may have a machine-learned identification function. The machine-learned identification function may include a function capable of identifying a predetermined object. The image information acquisition unit may acquire information relating to the above-mentioned various objects from the image by using the machine-learned identification function. The machine-learned identification function may be stored in the information processing device in the terminal device. Since the object to be identified is limited in advance, there is an advantage that a high identification function can be realized even in a simple information processing device such as in a terminal device.

Various artificial intelligence techniques may be used for machine learning. Artificial intelligence technologies include machine learning technologies such as neural networks, genetic programming, functional logic programming, support vector machines, clustering, regression, classification, Basilian networks, reinforcement learning, expression learning, decision trees, and k-means clustering. May be used. In the following, an example using a neural network will be used, but the present invention is not necessarily limited to the neural network.

As the machine learning technique using the neural network, a deep learning (deep learning) technique may be used. This is a technology that makes it possible to generate an output corresponding to an input even for an unknown input by learning the relationship between the input and the output using a plurality of layers. There are teachered and unsupervised methods, whichever is applied. In the case of the method with a teacher, the learning image and the attribute information related to the learning image are associated with each other, and machine learning is performed using these as learning data. The machine learning function related to the image information acquisition unit includes, for example, information related to at least a part of an image information acquisition unit, an information acquisition unit, a self-vehicle information generation unit, and an image information generation unit, and the information is an image seen from a person. The relationship between the image determined to be displayed above and the image may be machine-learned. By such machine learning, the machine learning related to the image information acquisition unit identifies information related to at least a part of the image information acquisition unit, the self-vehicle information generation unit, and the image information generation unit from the image, and generates the corresponding information. It may have a function that can be performed.

The learning algorithm itself when using the deep learning technique may be a known one. Further, as the program used in the deep learning technique, an open source program may be used, or a program obtained by modifying them as appropriate may be used.

Further, the machine-learned function may have an update function. The update function may be a function that can update an identifiable object. For example, it may be possible to identify information related to a vehicle, information related to a road, information related to an advertisement, and the like, and update the information so that the target information can be generated. Such updates may be in a mode in which the program or its parameters are downloaded and installed.

In addition, the information related to the target may be stored in association with the information related to the time when each information was acquired and / or the information related to the place. The time information may be month, day, week, time, and the like. The information related to the location may be administrative divisions, GPS coordinates, and the like. The information related to the location may be acquired by the GPS related to the terminal device for acquiring the image.

Further, the machine-learned identification function may use the identification function by using the position information. For example, when there is a high possibility that a specific advertisement is in a specific position, it may have an identification function that emphasizes the identification of the specific advertisement in the vicinity of the specific position including the specific position.

2.1.4. Storage unit The storage unit has a function of storing information. For example, the storage unit may store the above-mentioned information related to the image acquisition unit, the self-vehicle information generation unit, and the image information generation unit. Further, the storage unit may have a function of associating and storing a plurality of information. In addition to the above, the information to be associated includes, for example, a part or all of an image or the like, time information, own vehicle driving status information, own vehicle position information, vehicle specific information, and driving status information. , You may remember.

Further, the storage unit may store the information by the ring buffer. Further, the storage unit may store each piece of information in a ring buffer. In this case, the upper limit of the ring buffer is defined for each information, and when the information exceeding the upper limit is stored, the storage may be deleted from the oldest information. Each information may be, for example, information such as an image or the like, own vehicle driving status information, own vehicle position information, vehicle specific information, other vehicle driving status information, and the like.

2.1.5. Output unit The output unit may have a function of making a sound or displaying. For example, information related to the own vehicle and / or information related to an image may be displayed.

The information relating to the own vehicle may include the own vehicle driving status information and / or the distance information. There is an advantage that information on the driving of the own vehicle can be obtained from the information on the own vehicle. Further, the information related to the image may include vehicle specific information and / or other vehicle driving status information. There is an advantage that information on other vehicles can be obtained from information on other vehicles. As the information related to the image may include various information as described in detail below, such information may be displayed.

Further, the output unit may display the information acquired by the information and communication unit. For example, information related to other vehicle statistical information may be displayed. Further, the output unit may display the processed information by using other vehicle statistical information. This has the advantage that the viewer can understand the information related to other vehicles. In particular, when the driving condition of the vehicle in front is different from the usual one, there is an advantage that the driver can drive with care.

Further, the output unit has a function of emitting a sound, and may notify the user of the terminal device. For example, the output unit may generate a sound prompting the driver of the own vehicle to start the vehicle when the information prompting the departure of the own vehicle is generated.

Further, the output unit may output information relating to at least a part of the image information acquisition unit, the self-vehicle information generation unit, the image information generation unit, the storage unit, and the information communication unit.

2.1.6. Information and communication unit The information and communication unit has a function of communicating information. The information communication unit may communicate information relating to at least a part of an image information acquisition unit, a self-vehicle information generation unit, an image information generation unit, a storage unit, and an output unit. The information and communication unit may transmit such information.

Execution of the function for communicating information may be performed at an arbitrary timing, or may be at a timing when a specific condition is satisfied. The latter specific condition may be, for example, the timing at which the WIFI function can be executed. In this case, there is an advantage that the communication cost for the user can be reduced.

The information communication unit may have a function of receiving information. The information and communication unit may acquire other vehicle statistical information described later from the management system.

2.2. Management system functions

2.2.1. Statistical processing unit The statistical processing unit has a statistical processing function. By performing statistical processing, the statistical processing unit performs other vehicle statistical information, traffic jam information, weather information, road information, road abnormality information, accident information, vehicle fuel provision information, advertising statistical information, human information, and 3D. You may generate at least some information about the map.

<Statistical information on other vehicles>
The statistical processing unit uses vehicle identification information and other vehicle driving status information obtained from one or more terminal devices to provide statistical information related to other vehicles (sometimes referred to as "other vehicle statistical information" in the documents of the present application). May be generated. The statistic may include the sum of the number of sudden steering, the number of sudden braking, and / or the number of sudden accelerations for one vehicle, or the number of sudden steering, sudden braking for one vehicle. And / or the number of times of sudden acceleration, etc. may be included. Further, the total value or the average value may be intended for a vehicle imaged during a predetermined period.

Such other vehicle statistical information, for example, regarding the number of times of sudden steering, for a specific vehicle obtained from the vehicle specific information, the vehicle specific information obtained from the one or a plurality of terminal devices and the other vehicle driving status information. In the vehicle identification information, the same vehicle as the specific one vehicle is specified, and the presence / absence or the number of times of sudden steering in the corresponding other vehicle driving status information is acquired for a predetermined range. By applying to the vehicle identification information and the other vehicle driving situation information obtained from all the one or more terminal devices of the above, the total value of the steep steering wheel may be generated. Similarly, the total number of cue brakes and the number of sudden accelerations may be generated, and the corresponding average value in a predetermined period, a predetermined area, or the like may be generated as well.

Further, the statistical processing unit may generate a ranking for the total value or the average value. The ranking may be targeted at vehicles imaged during a predetermined period, may be targeted at vehicles imaged in a predetermined area, or may be targeted at a vehicle traveling on a predetermined road or a predetermined road condition. It may be something to do. The predetermined road may be, for example, a type of road such as a road in a residential area, an arterial road, an expressway, or the like. The predetermined road condition may be, for example, depending on the speed of the vehicle, for example, during traffic jam, low speed running, high speed running, etc., while traveling at a speed within a specified range with the speed as an evaluation value. It may be intended for a vehicle. The ranking may be a predetermined number of rankings in descending order of the total value or the average value, or may be a predetermined number of rankings in ascending order.

FIG. 14 is an example of other vehicle statistical information.

<Congestion information>
The statistical processing unit may generate congestion information. The traffic jam information may be any data as long as it is information related to the traffic jam of the vehicle. Congestion information may include, for example, information indicating the presence of congestion associated with a particular area. Information indicating the existence of a traffic jam associated with a specific area may be generated, for example, when one or more vehicles having location information in the specific area satisfy a predetermined condition. The predetermined condition is, for example, when the number of vehicles determined to be congested is larger than the predetermined number, or the number of vehicles determined to be congested is a predetermined multiple or more of the number of vehicles determined not to be congested. It may be determined depending on the case.

The vehicle may be determined to be in a traffic jam if certain conditions are met for the vehicle speed and / or the above-mentioned distance information.

If the speed of the vehicle satisfies a predetermined condition, for example, if the speed of the vehicle is smaller than the speed of the vehicle, it may be determined to be a traffic jam. Further, when the average speed of the vehicle is smaller than the predetermined speed within a predetermined time or a predetermined distance, it may be determined to be a traffic jam. This is because in the case of traffic congestion, the average speed of the vehicle becomes smaller than the predetermined speed. As the speed of the vehicle, the speed and / or the acceleration in the self-vehicle driving status information obtained from the terminal device may be used.

Further, when the distance information satisfies a predetermined condition, for example, when the distance information is smaller than the predetermined distance, it may be determined as a traffic jam. Further, it may be determined that there is a traffic jam because the average of the distance information is not more than a predetermined time within a predetermined time or within a predetermined distance. This is because when the size of the vehicle in the image is large, it indicates that the distance between the vehicle in front and the vehicle of the company is short, and it is information that should be determined to be a traffic jam.

Further, as the speed and / or distance information of the vehicle used when using the above-mentioned traffic jam information, the information associated with the time in a predetermined range may be used. Such a case may be when measured by the acceleration sensor in the terminal device or when the image acquired by the terminal device is captured. This has the advantage that traffic congestion information can be collected in the information at the same time or within a predetermined range including the same time.

Further, the traffic jam information may be included in association with the time measured by the acceleration sensor in the terminal device, the time when the image acquired by the terminal device is captured, or the information in the time range including these.

FIG. 15 is an example of traffic jam information. In this figure, the traffic jam information is generated for the area, but the traffic jam length information may be generated and the traffic jam length information may be included in the traffic jam information. The information on the length of the traffic jam is between the terminal devices that have acquired the information for determining the traffic jam when it is determined that the traffic jam is within a predetermined range in which the area or the position information is a short distance. Using the length of the distance, it may be determined that at least the length of the distance is the traffic jam, and the length of the traffic jam may be used as the length of the traffic jam.

<Weather information>
The statistical processing unit may generate weather information. The weather information may include, for example, weather conditions information associated with a particular area. The weather conditions information associated with a particular area is, for example, rain information and / or temperature information obtained from an image captured by one or more terminal devices having location information within the particular area. However, it may be generated when a predetermined condition is satisfied.

As a predetermined condition of information related to rain, for example, when there is more information determined to be rain than a predetermined number, or the number of information determined to be rain relative to the number of information not determined to be rain. If the proportion of is greater than a predetermined proportion, the information relating to rain may include information that it is rain.

Here, the information determined to be rain may be generated by using the information related to the operation of the wiper. For example, if there is movement of the wiper, it may be generated as rain, and if there is no movement of the wiper, it may not be determined that it is rain. Further, when the speed of movement of the wiper indicates one of a plurality of ranks indicating the speed of movement of the wiper, since it is one of the plurality of ranks, information indicating the degree of rain is generated. It's okay.

In addition, as predetermined conditions for information related to rain, for example, when there is more information than a predetermined number of information determined to be one of a plurality of ranks indicating the speed of movement of the wiper, or the speed of movement of the wiper. If the ratio of the number of information determined to be one of multiple ranks indicating the speed of movement of the wiper to the number of information determined to be one of multiple ranks is larger than the predetermined ratio, it rains. The information relating to may include information indicating the corresponding degree of rain.

Further, as a predetermined condition of the information indicating the temperature, for example, when the number including the cloak is larger than the predetermined number in the information related to the sidewalk, or the number including the cloak is larger than the predetermined ratio. In that case, the information indicating the temperature may include the information that the weather is such that a cloak is required, and the number including the long sleeve is larger than the predetermined number in the information related to the sidewalk, or the number not including the long sleeve is not included in the number including the long sleeve. If it is larger than the predetermined ratio, the information indicating the temperature may include the information that the weather requires a long sleeve, and the number including the half sleeve in the information related to the sidewalk is larger than the predetermined number or the number including the half sleeve. If the number without half sleeves is greater than a predetermined percentage, the information indicating the temperature may include information that the weather is such that half sleeves are required.

Further, the weather information may include the weather condition information and the information indicating the specific area used to generate the weather condition information in association with each other.

Further, the weather information may include the weather condition information and the time when the image obtained by acquiring the information related to the wiper used to generate the weather condition information and / or the information related to the sidewalk is captured in association with each other. ..

FIG. 16 is an example of weather information.

<Road abnormality information>
The statistical processing unit may generate road abnormality information. The road abnormality information may include information relating to the road abnormality state. The information relating to the road abnormality state may include, for example, an abnormality of a sign, an abnormality of a road marking, an abnormality of a traffic light, and / or an abnormality of a roadway.

Abnormal signs, abnormal road signs, abnormal traffic lights, and / or abnormal roadways, are abnormal signs, abnormal road signs, abnormal traffic lights, and / or abnormal roadways in the information related to the road. , May be generated.

The statistical processing unit may generate road abnormality information from information related to roads acquired from one terminal device, or may generate road abnormality information from information related to roads acquired from a plurality of terminal devices. In the latter case, the statistical processing unit may use a predetermined number or a predetermined ratio of terminal devices on the road for a predetermined range of areas including the same position information in a predetermined range. Road abnormality information may be generated when such information includes an abnormality of a sign, an abnormality of a road marking, an abnormality of a traffic light, and / or an abnormality of a roadway.

The road abnormality information may include information related to the road abnormality state and a position information in which an image related to the abnormality is captured or a region in a predetermined range including such position information in association with each other.

Further, the road abnormality information may include the information related to the road abnormality state in association with the time when the image is captured or the time in a predetermined range including the time when the image is taken.

Further, the road abnormality information may include information related to the road abnormality state and an image related to the abnormality in association with each other.

FIG. 17 is an example of road abnormality information.

<Road information>
The statistical processing unit may generate road information. Road information may be generated using information related to roads. The statistical processing unit may generate road information from information related to roads acquired from one terminal device, or may generate road information from information related to roads acquired from a plurality of terminal devices. In the latter case, the statistical processing unit may use the same or a predetermined ratio of terminal devices for a predetermined range of areas including the same position information in a predetermined range including the same time. Road information may be generated when information on similar roads is acquired. Here, the similar information may be an error range as the information generated from the image. Road information, such as, may be included in association with time information, area or location information, as well as road anomaly information.

<Accident information>
The statistical processing unit may generate accident information. The statistical processing unit may generate accident information when the information about the event on the roadway includes the information related to the accident. Further, in such a case, the statistical processing unit may set the area where the accident has occurred by using the position information associated with the information related to the accident, and the accident information may include the information of the area. In addition, the accident information may include information related to such an accident in association with the area.

Accident information may be generated based on information about roadway events. Further, the information at the time in the accident information may be generated by using the information at the time related to the event on the roadway. In addition, the information at the time in the accident information is early and late among the information related to the multiple roadway events when the information related to the roadway event is acquired from a plurality of terminal devices. It may be a range.

Further, the statistical processing unit may generate information related to a roadway event whose continuous period is shorter than a predetermined period as accident information among the information related to the continuous roadway event. This is because accidents on the road are generally removed in a shorter period of time compared to construction information. Therefore, even when it is not possible to determine whether the work is a construction work or an accident only from the image, the accident information may be determined by paying attention to the continuity of the information related to the event on the roadway. From the viewpoint of improving the accuracy of the accident information, the accident information may not be determined by paying attention to such a period.

The statistical processing unit may generate accident information from information related to roadway events acquired from one terminal device, or may generate accident information from information related to roadway events acquired from a plurality of terminal devices. In the latter case, the statistical processing unit is on the roadway from a predetermined number or a predetermined ratio of terminal devices for a predetermined range of areas including the same position information in a predetermined range including the same time. Road abnormality information may be generated when the information related to the event includes an abnormality of a sign, an abnormality of a road marking, an abnormality of a traffic light, and / or an abnormality of a roadway.

FIG. 18 is an example of such accident information.

<Information on fuel provided for vehicles>
The statistical processing unit may generate information on the provision of fuel for vehicles. The vehicle fuel provision information may include the type of vehicle fuel and the amount of money corresponding to such type in association with each other. The type of vehicle fuel and the amount of money corresponding to such type may be generated using the information on the vehicle fuel.

Further, the vehicle fuel provision information may include the location information corresponding to the type and the amount of money in association with each other. Such location information may be generated using the location information in the information relating to the fuel for the vehicle corresponding to such type and amount.

Further, the vehicle fuel provision information may include the information corresponding to the type and the amount of money in association with each other. The information at that time may be generated using the information at that time in the information relating to the fuel for the vehicle corresponding to such type and amount.

The statistical processing unit may generate vehicle fuel provision information from information related to vehicle fuel acquired from one terminal device, or from information related to vehicle fuel acquired from a plurality of terminal devices. May be generated. In the latter case, the statistical processing unit is used for vehicles from a predetermined number or a predetermined ratio of terminal devices in a predetermined range including the same time and in a predetermined range including the same position information. When the information relating to the fuel of the vehicle includes the type of fuel for the vehicle and the corresponding amount of money, the information providing the fuel for the vehicle may be generated. The time in the predetermined range may be a predetermined period retroactive from the present time. This has the advantage of being able to keep up-to-date information.

FIG. 19 is an example of information on providing fuel for a vehicle.

<Advertising statistics>
The statistical processing unit may generate advertising statistical information. The advertisement statistical information may include the number related to the advertisement or may include the ratio related to the advertisement. The number of advertisements may be the number of advertisements satisfying a predetermined condition. Advertisements that meet certain conditions include advertisements of a specific brand, advertisements by a specific advertiser, advertisements related to a specific field, advertisements of a predetermined size, advertisements larger than a predetermined size, and advertisements smaller than a predetermined size. , And / or advertisements that are blocked by trees or buildings. Further, the ratio related to the advertisement may be the ratio of the advertisement satisfying the second predetermined condition to the advertisement satisfying the first predetermined condition. Further, the first predetermined condition and the second predetermined condition may be the above-mentioned predetermined conditions, and these may be different. Further, the above-mentioned specific brand may be able to identify such a brand by character recognition in an image. Further, the above-mentioned advertiser may be able to identify such an advertiser by character recognition in an image. Further, with respect to the advertisement obstructed by trees and buildings, the field of view of the advertisement may be obstructed for a predetermined ratio or more in the time zone in which the advertisement can be recognized in the image.

In addition, the advertisement statistical information may include the number related to the advertisement and the information indicating a specific area including the position information obtained by capturing the image used to generate the number related to the advertisement in association with each other. The information indicating such a region may be generated as a specific region including such a position by using the position information associated with the image.

In addition, the advertisement statistical information may include the number related to the advertisement and the information indicating a specific time range including the time when the image used to generate the number related to the advertisement is captured in association with each other. Information indicating such a specific time range may be generated as a specific time range including such time using the information when associated with such an image.

The statistical processing unit may generate advertisement statistical information from information related to advertisements acquired from one terminal device, or may generate advertisement statistical information from information related to advertisements acquired from a plurality of terminal devices. In the latter case, regarding the information related to the advertisement related to the same or predetermined range of position information, the advertisement statistical information is obtained only when the information related to the same advertisement is acquired from a predetermined number or a predetermined ratio or more of terminal devices. It may be configured to generate, or it may be configured to generate advertisement statistical information even when the information related to the advertisement is acquired from one terminal device.

The statistical processing unit may include a corresponding predetermined condition for the advertisement statistical information. This has the advantage that it is possible to retain information as to what kind of conditions the advertisement statistical information satisfies.

FIG. 20 is an example of advertising statistical information.

<Personal information>
The statistical processing unit may generate human information. Human information may be generated based on information related to the degree of congestion of people. The person information may include the number of people and the information indicating the area where the person is located in association with each other. Further, the human information may include the number of people and the information including the time when the area where the person is present is imaged in association with each other.

The statistical processing unit may generate human information from information related to the degree of congestion of people acquired from one terminal device, or may generate information from information related to the degree of congestion of people acquired from a plurality of terminal devices. good. In the latter case, regarding the information related to the degree of congestion of people associated with the time within the predetermined time and associated with the predetermined area, the information from the terminal device satisfying the predetermined condition and / or each terminal device The average number of people in the above may be used to generate the number of people mentioned above.

FIG. 21 is an example of human information.

<3D map>
Statistical information may generate a 3D map. The 3D map may be a 3D digital representation of the landscapes on both sides as seen from the roadway. The 3D map may be generated from the image. The 3D map may be generated using one or more images captured by an image pickup device that images the front and / or rear of the terminal device. The means for generating a 3D map from one or more images may be known means.

The statistical processing unit may generate a 3D map from an image acquired from one terminal device, or may generate a 3D map from an image acquired from a plurality of terminal devices. In the latter case, an image associated with a predetermined time range and associated with the same location information may be used to generate a 3D map. By using an image captured in a predetermined time range, there is an advantage that a 3D map can be generated based on information that does not change with time. Further, there is an advantage that a more accurate 3D map can be generated by generating a 3D map from a plurality of images captured at the same place having the same position information.

2.2.2. Information and Communication Unit The information and communication unit may communicate information with one or more terminal devices. The information communication unit may acquire information from one or more terminal devices, or may transmit information to one or more terminal devices. The information communication unit acquires information related to at least a part of an image acquisition unit, a self-vehicle information generation unit, an image information generation unit, a storage unit, and an output unit as information from one or a plurality of terminal devices. You can do it. In addition, the information communication unit may provide one or more information as information related to at least a part of an image acquisition unit, a self-vehicle information generation unit, an image information generation unit, a storage unit, an output unit, and a statistical processing unit. It may be transmitted to the terminal device of.
Further, the information communication unit may have a function of communicating information with various systems.

<Congestion information>
The Information and Communication Department is responsible for at least part of other vehicle statistics, congestion information, weather information, road information, road abnormality information, accident information, vehicle fuel provision information, advertising statistics information, human information, and 3D maps. Information may be transmitted to the system relating to the road user. In the documents of the present application, the road user provides a company that provides a map, a carrier that uses the road, a taxi company that uses the road, a company that provides car navigation, a company that maintains the road, and services related to automatic driving. Companies and / or companies that provide services to municipalities or municipalities may be included.

<Accident information>
The information and communication department may transmit the accident information to the system related to the insurance company.

<Advertising statistics>
The information and communication unit may transmit the advertisement statistical information to the system related to the advertising company.

<Personal information>
The information and communication unit may transmit the person information to the system related to the radio wave service of the mobile phone.

3. 3. Example
3.1. Example 1
The system example according to the first embodiment is an embodiment mainly used as a drive recorder by using some or all of the above-mentioned functions. A user of an example system may have, for example, a smartphone as a terminal device. The software related to the example system may be downloaded and installed in advance on such a smartphone.

Upon boarding a vehicle used by a user of an example system (hereinafter, also referred to as a "user vehicle"), the user attaches a terminal device to the user vehicle. Here, various mounting modes may be used. It may be removable. Further, the rectangular smartphone may be attached to the vehicle in a horizontally long direction that is longer in the horizontal direction than in the vertical direction, or may be attached in a vertically long direction that is longer in the vertical direction than in the horizontal direction. In the case of the landscape, there is an advantage that the lane in which the own vehicle travels and the vehicle traveling on the adjacent lane may also be widely included in the imaging range.

Next, the user activates the software related to the example system. When the example system is started, the example system may automatically start imaging. That is, the image pickup device in the terminal device may be activated to start the image pickup. By automatically starting the image pickup only by starting the image without any other operation, the user has an advantage that he / she can prevent forgetting to start the image pickup operation and save the trouble of starting the image pickup. Further, in another example of the system, the imaging does not have to be started at the same time as the start-up. In this case, the shooting may be started by selecting the shooting button by the user. The shooting button may be a mechanical and physical button in the smartphone, or may be a touch on the screen.

Further, the terminal device according to the example system may set the autofocus to infinity at the time of imaging and perform imaging. Depending on the mounting position of the terminal device in the vehicle, the bonnet of the own vehicle may enter the imaging field of view, and when the inventor conducted an experiment, the bonnet was sometimes focused. In this case, the focus is about 2 meters, and the focus is not on the vehicle in front, which may reduce the accuracy of acquiring information for identifying the vehicle in front. Also, especially in rainy weather, the focus of the terminal device may be on the water droplets on the windshield or the wiper in operation, and similarly, the accuracy of acquiring information that identifies the vehicle in front may decrease. .. As described above, there is an advantage that such a decrease in accuracy can be prevented by automatically setting the focal point at the time of imaging of the terminal device to infinity.

One example of the system stores the captured image and stores the information related to the driving situation at that time. The video to be stored may be stored in a file for each fixed file size. For example, a video file for each XMB (megabyte) of a predetermined storage amount may be created, and when the predetermined storage amount is exceeded, the next file may be created to store the video. Having such a file division function has an advantage that even when a long-time video is stored, one file does not become large and the convenience is improved when communicating with another information processing device. The video to be shot may include audio. This is because, as a drive recorder, voice in the driving situation is also important information.

The system of one example may display various displays or make sounds on the output unit according to the content of the captured image and the information related to the driving situation. For example, a warning may be issued when the distance between the vehicle driving ahead and the own vehicle is close. The warning may be displayed on the output unit in a manner that attracts the attention of the viewer, or may make a sound that attracts the attention of the person in the vehicle. Further, as information related to the driving situation, in the case of sudden braking, sudden acceleration, and sudden steering, these may be similarly warned.

An example system may generate a drive report when the vehicle has finished driving. For example, an example system may generate statistical information about driving conditions and include it in a drive report.

An example system may transmit the stored information to the management server. For example, the captured moving image and information related to the driving situation may be transmitted to the management server. The timing of transmission may be configured such that the terminal device starts transmission at a timing when transmission becomes possible with another information processing device by a predetermined communication mode. The predetermined communication mode may be, for example, WIFI. An example system may automatically erase the video stored in the terminal device after transmission to the management server. The terminal device may automatically delete the information confirmed that the video is stored in the management server after the terminal device acquires the information. In addition, automatic erasing may be performed without confirmation of the user. The video stored as a drive recorder will have a large storage capacity for the terminal device, so the next time it is used

An example system may have a function of reproducing the information stored in the terminal device. In addition, it may have a function of displaying information related to the operating status.

FIG. 22 is an example of a screen transition diagram of the terminal device according to the system of this example. When the startup screen 001 is started, the agreement agreement screen 002 is displayed. Such a screen is only when the application is started for the first time after installation. When you agree to the terms or start up for the second time or later, the TOP screen 003 is displayed. From the top screen, it is possible to shift to the recording screen 004, the captured image list screen 006, and the setting screen 008. After the recording screen, the screen can be moved to the drive report screen 005. The drive report screen can display statistical information about the driving status of the drive during recording. Further, it is possible to shift from the captured image list screen 006 to the captured image reproduction screen 007. This is a screen that can reproduce one image selected from the list of screens captured in the past. Further, various settings may be possible on the setting screen 008. The screen transition shown here is an example. For example, from the shot video list screen, the selected shot video may be reproduced or the drive report screen of the shot video may be displayed.

FIG. 23 is a screen waiting for a shooting instruction after the system of one example is started. On the display screen 001, the image pickup button 002 is displayed large on the display screen. In this example, instead of automatic imaging, imaging can be performed according to a user's instruction. The image pickup button may have an area of one tenth or more of the area of the display screen. When the image pickup button has an area larger than a predetermined area, there is an advantage that the user can more easily instruct the start of image pickup.

FIG. 24 is an example of an image pickup screen during driving. In the front screen 001, the vehicles in the front direction are surrounded by the red frame 002, indicating that each vehicle is recognized. In addition, information 003 regarding the operating status is also displayed. In the graph, the horizontal axis shows the elapsed time and the vertical axis shows the speed of the vehicle. Therefore, where the angle of the line in the graph is steep, such a display may be made because sudden acceleration, sudden braking, or sudden steering can be applied. Further, when the distance to the vehicle in front is short, particularly when the distance to the vehicle in front on the same lane is short, it may be displayed as a dangerous inter-vehicle distance. In addition, the sign 004 on the road may be recognized and the information thereof may be acquired.

FIG. 25 shows a list of captured images. A list of each video 001 is displayed according to the time. These may be from the start of one shooting to the end of shooting, or may be listed for each file divided by a predetermined storage amount as described above. FIG. 26 is another example showing a list of captured images.

FIG. 27 shows the captured image in 001 during display, and also shows a list 002 of some images.

FIG. 28 is an example showing a horizontally long image / display instead of the vertically long image / display described above. Similarly, the vehicle may be imaged by a smartphone mounted horizontally.

FIG. 29 is an example of displaying the operating status during imaging. The horizontal axis is the time, and the vertical axis is the speed of the own vehicle, and a mark indicating the time when sudden acceleration, sudden steering, or sudden braking occurs is displayed. Note that FIG. 30 is another example of displaying the operating status during imaging. The screen is a vertical screen. Similarly, in this figure, the horizontal axis is the time and the vertical axis is the speed of the own vehicle, and a mark indicating the time when sudden acceleration, sudden steering, or sudden braking occurs is displayed.

FIG. 31 is an example showing a state in which a smartphone mounted horizontally is recognizing a vehicle in front of the smartphone. The vehicle in front is surrounded by a frame 001 to indicate that it is recognized. The output unit may display information that identifies the vehicle thus identified. Further, not only the vehicle but also the output unit may display information specifying pedestrians, motorcycles (including bicycles) and the like. In this case, if the image information generation unit identifies a person and something like a two-wheeled vehicle, it is identified as a two-wheeled vehicle, and if it identifies a person without specifying something like a two-wheeled vehicle, it is identified as a pedestrian. It's okay. Further, the line of information for identifying such a vehicle may be a circle or a polygon instead of a rectangle. Further, the display for identifying the target vehicle indicating such recognition may be a line surrounding the target vehicle or a line not surrounding the target vehicle. The display identifying such a vehicle may be various vehicles. For example, it may be a passenger car, a commercial vehicle, a truck, a taxi, a motorcycle, and the like. Further, the display for identifying such a vehicle may be limited to the case where the above-mentioned distance information includes a distance within a predetermined range. For example, a display identifying such a vehicle may be made only if the vehicle is within a range of 15 meters. In addition, the display that identifies the vehicle may be displayed in various modes. As for the aspect, the color, the shape of the display, and the like may be changed. Further, the display for identifying the vehicle may change the mode of the marking according to the information included in the above-mentioned distance information. For example, when the distance information includes information of 2 meters or more and 5 meters or less, it may be yellow, and when the distance information includes information of 5 meters or more and 15 meters or less, it may be blue or the like. Further, the distance included in the distance information may be displayed in association with the display that identifies the vehicle. For example, it may be displayed as "3 m" in association with a display that identifies a vehicle in front. If the distance information includes a distance between 0 meters and 2 meters, it is not necessary to display the distance. Further, whether or not the information on the distance is displayed may be determined by the vehicle depending on the position of the output unit and whether or not the information is displayed. For example, the output unit may display only when the display specifying the vehicle is displayed at a predetermined position of the output unit. For example, the output unit may display only the vehicle in which the rectangle specifying the vehicle includes the center of the screen. Whether or not the rectangle that identifies the vehicle is in the center of the screen may be determined, for example, by comparing the coordinate position that identifies the rectangle with the coordinate position in the center of the screen. In this way, by displaying the distance information only for the central vehicle, for example (because the terminal device is generally installed at the center position with respect to the vehicle so that the front of the vehicle is centered), parking on the street. There is an advantage that the distance is not displayed for the vehicle at the edge of the screen, such as the vehicle inside, which is not bothersome to the viewer.

FIG. 32 is an example in which the aspect of such a rectangular line is changed according to the proximity of the vehicle in front on the same lane to the own vehicle. As an example of changing the aspect of the line, a color such as red or yellow that attracts the user's attention may be changed, or the thickness of the line or the decoration of the line may be changed. Further, in order to indicate that they are approaching, "approaching" may be displayed as shown in this figure.

FIG. 33 shows a state in which an image is being viewed after imaging. When one video is selected from the video list 003, video 001 is displayed. At this time, the sudden steering wheel, the sudden acceleration, and the sudden-brake are displayed in the graph 002 by each display. In the graph, the horizontal axis represents time and the vertical axis represents speed. Similarly, in FIG. 34, the case of a vertical screen is displayed.

3.2. Example 2: Information gathering by another company The system of one example may be provided with only the configuration essential to this embodiment, or may be provided with an embodiment required for another embodiment.

An example system acquires images from one or more terminal devices. Further, the system of one example may acquire one or more vehicle identification information and other vehicle determination information associated with the one or more vehicle identification information from one terminal device.

In one example system, other vehicle statistical information may be generated by collecting other vehicle determination information for each vehicle specific information. The other vehicle statistical information may be summarized in a predetermined period based on the images captured in the predetermined period. In addition, other vehicle statistical information may be summarized in a predetermined area based on an image captured in the predetermined area. Further, as information on the driving status of other vehicles, a predetermined coefficient may be associated with each other for sudden acceleration, sudden steering wheel, and sudden braking, and a total determination score weighted by multiplying the number of times thereof may be generated. FIG. 35 is an example of other vehicle statistical information. In this figure, vehicles are arranged in order of highest overall judgment score.

An example system acquires one or more specific vehicle information from one terminal device, searches the one or more specific vehicle information from other vehicle statistical information, and converts the one or more specific vehicle information into the one or more specific vehicle information. Such other vehicle statistical information may be transmitted to the above-mentioned one terminal device. Here, the other vehicle statistical information related to the specific vehicle information is, for example, the number of times of sudden acceleration, the number of times of sudden braking, the number of times of sudden steering, the numerical value related to each number of times, the total judgment score, and the ranking ranking. , And so on. The numerical value related to the number of times may be the probability of occurrence within a predetermined period or a coefficient thereof. Further, the other vehicle statistical information related to the specific vehicle information may be processed into information that simply indicates the degree of danger of danger. For example, it may include information of one of three options such as high, normal, and low.

In the one terminal device, for one or a plurality of other vehicles in the captured image, the other vehicle statistical information related to the specific vehicle information may be displayed in association with the one or a plurality of other vehicles. FIG. 37 is an example of such a display. In this example, by displaying "Caution vehicle" by the display associated with the vehicle 001, the vehicle is simply driven more dangerously than a predetermined standard (for example, sudden acceleration, sudden braking, and / or sudden steering wheel). This is an example showing that the number of times is higher than the predetermined standard, the total judgment score is higher than the predetermined standard, and / or the ranking ranking is higher than the predetermined standard). If the vehicle 002 does not have a driving history that is more dangerous than the predetermined standard, it is not necessary to display anything as shown in this figure, and it is more than the predetermined standard such as "normal". May also indicate that they do not have a dangerous driving history.

FIG. 38 shows an example in which an example system collects information acquired from a plurality of terminal devices, processes the information, and transmits the information to one or a plurality of terminal devices. An example management system 004 may acquire information from the terminal devices 001 to 003, perform predetermined processing, and transmit the information to the terminal devices 001 to 003. Here, the terminal device for acquiring information and the terminal device for transmitting information may be the same or different.

3.3. Example 3: The system of one example of vehicle information collection may be provided with only the configurations essential to this embodiment, or may be provided with embodiments required for other embodiments. An example system may acquire information relating to a target from one or a plurality of terminal devices, and the timing may be any. For example, in one system, information may be acquired when the terminal device is connected to a wireless communication device such as WIFI, or when the terminal device is connected to a communication standard such as 3G, 4G, or 5G. Then, the information may be acquired, or the information may be acquired in real time at the timing when the information to be transmitted by the terminal device is acquired. It should be noted that the real time may include those transmitted with a predetermined delay associated with information processing.

FIG. 39 shows an example in which an example system collects information acquired from a plurality of terminal devices, processes the information, and transmits the information to the corresponding companies. An example management system 004 may acquire information related to the target from the terminal devices 001 to 003 and transmit it to the systems 005 to 007 related to the target company. The target company may include road users, insurance companies, local governments or companies related to local governments, advertising companies, mobile phone related companies, and the like.

An example system is at least a portion of information (subordinate to each of the above terms) of congestion information, weather information, road information, road anomaly information, accident information, vehicle fuel supply information, human information, and 3D maps. (It may be conceptual information) etc. can be used to update the map when sent to the system related to the company that provides the map, and when sent to the system related to the carrier using the road, the efficiency of the carrier When it is sent to the system related to the taxi company that uses the road, it can be used for efficient use of the taxi, and when it is sent to the system related to the company that provides car navigation, the car navigation system Can be used to update information to, and can be used to generate road information when sent to a system related to a company that maintains the road. Automatic driving when sent to a system related to a company that provides services related to automatic driving. It can be used to provide services related to the above, can be used for the operation of the local government when sent to the system related to the local government, and can be used to create insurance information when sent to the system related to the insurance company. Further, there is an advantage that the system of one example can be used for the examination of the advertisement information when the advertisement statistics information is transmitted to the system related to the advertisement company. Further, there is an advantage that the system of one example can be used for the service related to the radio wave of the mobile phone when the person information is transmitted to the system related to the radio wave service of the mobile phone.

FIG. 40 is an example of a flow in an example system. An example system may acquire information transmitted from an information communication unit in such a terminal from one or more terminals. Next, the statistical processing unit related to the system of one example may perform statistical processing using the information acquired by the information and communication unit. The example system may then send information, including such statistically processed information, to the target company.

4. Hardware Configuration of System and Terminal Device Each of the system and terminal device according to the present invention may be composed of one or a plurality of information processing devices. As shown in FIG. 41, the information processing device 10 according to the present invention may include a bus 15, an arithmetic unit 11, a storage device 12, and a communication device 16. Further, the information processing device 10 in one embodiment may include an input device 13 and a display device 14. It is also directly or indirectly connected to the network 17.

The bus 15 may have a function of transmitting information between the arithmetic unit 11, the storage device 12, the input device 13, the display device 14, and the communication device 16.

An example of the arithmetic unit 11 is a processor. This may be a CPU or an MPU. Further, the arithmetic unit in one embodiment may include a graphics processing unit, a digital signal processor, and the like. In short, the arithmetic unit 12 may be any device that can execute the instructions of the program.

The storage device 12 is a device for recording information. This may be either an external memory or an internal memory, and may be either a main storage device or an auxiliary storage device. Further, a magnetic disk (hard disk), an optical disk, a magnetic tape, a semiconductor memory, or the like may be used. Further, it may have a storage device via a network or a storage device on the cloud via a network.

The register, L1 cache, L2 cache, etc. that store information at a position physically close to the arithmetic unit may be included in the arithmetic unit 11 in this block diagram, but in the design of the computer architecture, As a device for recording information, the storage device 12 may include these. In short, the arithmetic unit 11, the storage device 12, and the bus 11 may be configured to cooperate with each other to execute information processing.

The storage device 12 may include a part or all of a program capable of executing the process according to the present invention. In addition, data necessary for executing the process according to the present invention can be appropriately recorded. Further, the storage device 12 in one embodiment may include a database.

Further, the above describes the case where the arithmetic unit 12 is executed based on the program provided in the storage device 13, but it is one of the forms in which the bus 11, the arithmetic unit 12 and the storage device 13 are combined. As a result, part or all of the information processing according to the present invention may be realized by a programmable logic device capable of changing the hardware circuit itself or a dedicated circuit in which the information processing to be executed is determined.

Although the input device 13 inputs information, it may have other functions. Examples of the input device 14 include an input device such as a keyboard, a mouse, a touch panel, or a pen-type instruction device.

The display device 14 has a function of displaying information. For example, a liquid crystal display, a plasma display, an organic EL display, and the like can be mentioned, but in short, any device that can display information may be used. Further, the input device 13 may be partially provided like a touch panel.

The network 17 transmits information together with the communication device 16. That is, it has a function of transmitting information of 10 which is an information processing device to another information terminal (not shown) via a network 17. The communication device 16 may use any connection type, and may be IEEE1394, Ethernet (registered trademark), PCI, SCSI, USB, 2G, 3G, 4G, 5G, or the like. The connection to the network 17 may be either wired or wireless.

The information processing apparatus according to the present invention may be a general-purpose type or a dedicated type. Further, the information processing device may be a workstation, a desktop personal computer, a laptop personal computer, a laptop personal computer, a PDA, a mobile phone, a smartphone or the like.

Although described as one information processing device 10 in this figure, the system according to the present invention may be configured by a plurality of information processing devices. The plurality of information processing devices may be internally connected or may be externally connected.

Further, the system according to the present invention may be of various types of devices. For example, the system according to the present invention may be a stand-alone system, a server-client system, a peer-to-peer system, or a cloud system. The system according to the present invention may be a stand-alone information processing device, may be composed of a part or all information processing device of a server-client type, or may be a part or all information processing of a peer-to-peer type. It may be composed of devices, or may be composed of a part or all of information processing devices in cloud format.

Further, when the system according to the present invention is composed of a plurality of information processing devices, the owner and the manager of each information processing device may be different.

Further, the information processing apparatus 10 may be a physical existence or a virtual one. For example, the information processing apparatus 10 may be virtually realized by using cloud computing.

Although described above as the configuration implemented by the system of this example, these may be configurations implemented by one or more information processing devices in the system. Further, the information processing apparatus described above as a portable information processing apparatus may be an information processing apparatus that is appropriately installed and fixed.

It goes without saying that the examples of the invention described in the examples of the documents of the present application are not limited to those described in the documents of the present application, and can be applied to various examples within the scope of the technical idea. For example, in the embodiment of the document of the present application, the information presented on the screen of the information processing apparatus can be displayed on the screen of the other information processing apparatus and thus can be transmitted to the other information processing apparatus. The system may be configured. Further, the display of ◯ in various drawings may contain appropriate values according to each context, and may be the same or different.

Further, the processes and procedures described in the documents of the present application may be feasible not only by those explicitly described in the embodiments but also by software, hardware or a combination thereof. Further, the processes and procedures described in the documents of the present application may be able to be implemented by various computers by implementing the processes and procedures as a computer program. Further, these computer programs may be stored in a storage medium. Also, these programs may be stored on a non-transient or temporary storage medium.

Claims (23)

A terminal device that can be installed in a vehicle
The first image pickup unit that sets the autofocus to infinity and images the front of the vehicle,
A second image pickup unit that sets autofocus to infinity and images the rear of the vehicle,
An association unit that associates the imaging date and time with the image captured by the first imaging unit and / or the second imaging unit.
With respect to the captured image, a specific unit that identifies information related to an object including at least a pedestrian, a passenger car, a truck, and a two-wheeled vehicle in the image by using a machine-learned function.
A mobile terminal device that can be carried by a person.
The first image pickup unit and the second image pickup unit are terminal devices provided at positions facing each other in the mobile terminal device. The information relating to the subject includes information relating to the degree of congestion of people.
The terminal device according to claim 1. The information relating to the subject includes information relating to sidewalks including those worn by humans.
The terminal device according to any one of claims 1 and 2. The information relating to the subject includes information relating to the sidewalk including an umbrella, a cloak, a half sleeve, and / or a long sleeve.
The terminal device according to any one of claims 1 to 3. The information relating to the subject includes the presence or absence of construction based on either the presence or absence of a stop of the construction vehicle or the installation of an instant traffic light in the image.
The terminal device according to any one of claims 1 to 4. The information related to the subject is based on whether or not the construction vehicle is stopped and the vehicle guidance by the construction personnel, or the installation of an instant traffic light and the vehicle guidance by the construction personnel in the image. including,
The terminal device according to any one of claims 1 to 4. The construction personnel are determined by their clothes and / or their belongings.
The terminal device according to claim 6. The information relating to the subject includes information regarding the presence or absence of an accident based on the presence or absence of police personnel detected by the clothes and / or belongings of the police personnel in the image.
The terminal device according to any one of claims 1 to 7. The information related to the subject includes information related to roads, and includes information related to roads.
The information related to the road includes an abnormality of a traffic light.
The terminal device according to any one of claims 1 to 8. The information relating to the subject includes, including an abnormality in the roadway.
The terminal device according to any one of claims 1 to 9. The roadway anomalies include lane boundary anomalies, foreign objects on the roadway, and / or roadway destruction.
The terminal device according to claim 10. The foreign matter on the roadway includes a fallen object on the roadway, a fallen tree, and / or a fallen utility pole.
The terminal device according to claim 11. The information relating to the subject includes, including the width of the lane.
The terminal device according to any one of claims 1 to 12. The information relating to the subject includes information on parking on the street.
The terminal device according to any one of claims 1 to 13. The image pickup unit of the terminal device that can be installed in the vehicle
The first imaging step in which the autofocus is set to infinity to image the front of the vehicle, and
The second imaging step, in which the autofocus is set to infinity and the rear of the vehicle is imaged,
And run
The information processing unit of the terminal device
A step of associating the imaging date and time with the image captured by the first imaging step and / or the second imaging step.
The step of applying the machine-learned function to the captured image, and
A step of identifying information about an object in the image, including at least pedestrians, passenger cars, trucks, and motorcycles.
Is a way to do
A method of executing an image pickup unit to be imaged in the first image pickup step and an image pickup unit to be imaged in the second image pickup step on a portable terminal device provided at a position facing each other in the terminal device. .. The image pickup unit of the terminal device that can be installed in the vehicle,
The first imaging means for imaging the front of the vehicle by setting the autofocus to infinity ,
A second imaging means that sets the autofocus to infinity and images the rear of the vehicle.
It is a program that operates as
The information processing unit of the terminal device
A means for associating an imaging date and time with an image captured by the first imaging means and / or the second imaging means.
A means for applying a machine-learned function to the captured image,
A means for identifying information about an object in the image, including at least pedestrians, passenger cars, trucks, and motorcycles.
It is a program that operates as
A program for executing an image pickup unit to be imaged by the first image pickup means and an image pickup unit to be imaged by the second image pickup means on a portable terminal device provided at a position facing each other in the terminal device. .. A program for making a terminal device function as the terminal device according to any one of claims 1 to 14. The terminal device includes a memory.
The terminal device according to any one of claims 1 to 14. The terminal device includes an arithmetic unit.
The terminal device according to any one of claims 1 to 14 and 18. The terminal device includes a memory.
The method according to claim 15. The terminal device includes an arithmetic unit.
The method according to claim 15 or 20. Using the memory in the terminal device,
The program according to claim 16. The terminal device includes an arithmetic unit.
The program according to claim 16 or 22.

Images