JP6833472B2 - In-vehicle device and information processing system - Google Patents

Description

The present invention relates to an in-vehicle device and an information processing system.

Conventionally, there is known an in-vehicle device (in-vehicle camera device) in which an imaging unit (camera) is mounted on a vehicle, an image acquired by the imaging unit is analyzed, and processing is performed based on the analysis result (see, for example, Patent Document 1). ).

Japanese Unexamined Patent Publication No. 2016-139197

In a device such as the in-vehicle device described in Patent Document 1 described above, in which an imaging unit is mounted on a vehicle and an image acquired by the imaging unit can be analyzed, more useful processing can be executed using the analysis results. There is a need to do so.
The present invention has been made in view of the above circumstances, and an object of the present invention is to enable useful processing to be executed based on an image acquired by an image pickup unit mounted on a vehicle.

In order to achieve the above object, the present invention detects whether an image acquired by an imaging unit mounted on a vehicle includes a plurality of persons, and when a plurality of persons are detected, the positional relationship of the plurality of persons is determined. When it is analyzed and it can be recognized as a matrix , the facility corresponding to the head position of the recognized matrix is searched, and when a plurality of the matrix is recognized, the head position of each matrix is calculated and the head position is calculated. The corresponding facility is searched for each position, the information about the facility is associated with the information about the matrix, the search for the facility according to the predetermined search conditions is accepted, and the facility extracted as a result of the search is the information about the matrix. when There has been added, to a control unit which output the data related to map the matrix information about the overlap added to the information and the facility relating to the extracted facilities as a result of the search to the display panel It is characterized by.
Further, the present invention is characterized in that the control unit determines whether or not the image acquired by the imaging unit includes a person's face, and calculates the position of the head of the matrix based on the image. To do.

Further, the present invention is characterized in that, when the control unit can recognize that it is a matrix, it acquires the scale of the recognized matrix and outputs information indicating the scale of the matrix.

Further, in the present invention, the control unit determines whether or not a plurality of persons included in the image are lined up with a predetermined regularity based on the image acquired by the imaging unit, and the plurality of persons included in the image. When the characters are lined up with a predetermined regularity, it is recognized as a matrix.

Further, in the present invention, the control unit has a plurality of persons included in the image having a predetermined regularity when the variation in the separation distance between the adjacent persons in a predetermined direction is small for the plurality of persons included in the image. It is characterized in that it is determined that it is in a lined state.

Further, in order to achieve the above object, the present invention is an information processing system including an in-vehicle device and a server capable of communicating with the in-vehicle device, and the in-vehicle device is acquired by an imaging unit mounted on a vehicle. Detects whether the image contains multiple people, and if multiple people are detected, analyzes the positional relationship of the multiple people, and if it can be recognized as a matrix, searches for facilities corresponding to the recognized matrix. Then, when a plurality of the said matrixes are recognized, the head position of each matrix is calculated, the corresponding facility is searched for each of the head positions, and the information about the searched facility and the head position of the recognized matrix are related. When the information and the facility corresponding to the head position are transmitted to the server, the search for the facility based on the predetermined search conditions is accepted, and the information related to the matrix is added to the facility extracted as a result of the search. , The server outputs the data related to the map in which the information about the facility extracted as a result of the search and the information about the matrix added to the facility overlap , and the server outputs the information about the facility and the information about the matrix. The information about the facility and the matrix corresponding to the facility extracted as a result of receiving, storing the information about the facility and the information about the matrix in association with each other and searching based on the predetermined search condition by the in-vehicle device. The information is transmitted to the in-vehicle device .

According to the present invention, useful processing can be executed based on an image acquired by an image pickup unit mounted on a vehicle.

The figure which shows the structure of the information processing system which concerns on this embodiment. The block diagram which shows the functional configuration of an in-vehicle device and a control server. The flowchart which shows the operation of an in-vehicle device and a control server. A flowchart showing the operation of the in-vehicle device. The figure which shows the photographed image data. The figure which shows the photographed image data. A flowchart showing the operation of the in-vehicle device. The figure which shows the photographed image data. The figure which shows an example of the information displayed on a touch panel. The flowchart which shows the operation of an in-vehicle device and a control server. The figure which shows an example of the search result display screen.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing a configuration of an information processing system 1 according to the present embodiment.
As shown in FIG. 1, the information processing system 1 includes a control server 10 (server) connected to a network N configured including the Internet and a telephone network. Further, the information processing system 1 includes an in-vehicle device 12 mounted on the vehicle S. The control server 10 and the in-vehicle device 12 can communicate with each other via the network N. The configuration, functions, and processing based on the functions of these devices will be described later.
In FIG. 1, the control server 10 is represented by one block, but this does not mean that the control server 10 is composed of a single server device. The control server 10 may be configured to include a plurality of server devices, or may be a part of a predetermined system. That is, the control server 10 may have any form as long as it has the functions described below. Further, although details are omitted, secure communication is performed between the control server 10 and the in-vehicle device 12 by a predetermined encryption technique or other security-related techniques.

FIG. 2 is a block diagram showing a functional configuration of the in-vehicle device 12 and the control server 10.

The in-vehicle device 12 is a device mounted on the vehicle S. The in-vehicle device 12 has a function of detecting the current position of the vehicle S, a function of displaying the current position of the vehicle S on a map, and a route search for searching a route to the destination. It also has a function to display a map, display a route to the destination on the map, and perform route guidance to guide the route to the destination.

As shown in FIG. 2, the in-vehicle device 12 includes a control unit 20, a storage unit 21, an operation unit 22, a communication unit 23, a GPS unit 24, a relative orientation detection unit 25, a touch panel 26, and an imaging unit. 27 and.

The control unit 20 includes a CPU, ROM, RAM, other peripheral circuits, and the like, and controls each unit of the in-vehicle device 12. The control unit 20 controls each part of the in-vehicle device 12 in cooperation with hardware and software, for example, the CPU reads and executes a control program stored in the ROM.

The storage unit 21 includes a non-volatile memory and stores various data.
The storage unit 21 stores the map data 211.
The map data 211 relates to the display of a map such as road drawing data related to drawing the shape of a road, background drawing data related to drawing a background such as terrain, and character string drawing data related to drawing a character string such as an administrative division. Includes drawing data. Further, the map data 211 includes node information having information on nodes corresponding to connection points in a road network such as intersections, link information having information on links corresponding to roads formed between nodes, and the like. Includes information related to route search and route guidance. The map data 211 may include other information about the map.
The storage unit 21 stores the facility data 212.
The facility data 212 stores facility identification information, facility location information, facility category information, facility-related information, and facility name information in association with each facility on the map.
The facility identification information is identification information that identifies the facility. The facility location information is information that represents the location of the facility by a combination of longitude and latitude. The facility category information is information indicating the category of the facility. The facility name information is information indicating the name of the facility.
The facility data 212 may include other information about the facility.

The operation unit 22 includes an operation switch 221 provided in the housing of the vehicle-mounted device 12, detects an operation on the operation switch 221 and outputs a signal corresponding to the operation to the control unit 20. The control unit 20 executes a process corresponding to the operation based on the signal input from the operation unit 22.

The communication unit 23 accesses the network N according to a predetermined communication standard under the control of the control unit 20, and communicates with a device (including the control server 10) connected to the network N. The communication standard used for communication between the in-vehicle device 12 and the control server 10 may be anything. Communication standards are, for example, HTTP and WebSocket.
In the present embodiment, the in-vehicle device 12 has a function of accessing the network N. However, the in-vehicle device 12 may be configured to communicate with an external device that does not have the function and has a function of accessing the network N, and access the network N via the external device. For example, the in-vehicle device 12 performs short-range wireless communication with a mobile device (for example, a so-called smartphone or a tablet-type computer) owned by a user boarding the vehicle S, and accesses the network N via the mobile device. You may.

The GPS unit 24 receives GPS radio waves from GPS satellites via GPS antennas (not shown), and detects the current position and traveling direction of the vehicle S from the GPS signals superimposed on the GPS radio waves by calculation. The GPS unit 24 outputs the detection result to the control unit 20.

The relative orientation detection unit 25 includes a gyro sensor and an acceleration sensor. The gyro sensor is composed of, for example, a vibrating gyro, and detects a relative orientation of the vehicle S (for example, a turning amount in the yaw axis direction). The acceleration sensor detects the acceleration acting on the vehicle S (for example, the inclination of the vehicle S with respect to the traveling direction). The relative direction detection unit 25 outputs the detection result to the control unit 20.

The touch panel 26 includes a display panel 261 and a touch sensor 262.
The display panel 261 is a display device such as a liquid crystal display panel or an organic EL display panel. The control unit 20 drives the display panel 261 to display an image on the display panel 261.
When the touch panel 26 is touch-operated, the touch sensor 262 outputs a signal indicating the contact-operated position (hereinafter, referred to as “contact position”) to the control unit 20. The control unit 20 executes a process corresponding to the contact operation based on the signal indicating the contact position input from the touch sensor 262.

The imaging unit 27 is mounted on the vehicle S and has a camera that photographs the left side in the traveling direction of the vehicle S. The imaging unit 27 executes shooting at a predetermined cycle under the control of the control unit 20. The imaging unit 27 generates captured image data based on the shooting result, and outputs the generated captured image data to the control unit 20. The captured image data is a bitmap in which dots having information on colors (for example, information representing the color components of each color of RGB by gradation values of predetermined gradations) are arranged in a dot matrix corresponding to a predetermined resolution. It is data.
In this embodiment, the imaging unit 27 photographs the left side of the vehicle S. This is based on the premise that the vehicle S according to the present embodiment travels on the left side, and as will be described later, the captured image data based on the imaging result of the imaging unit 27 is used to determine whether or not a matrix is generated. Because. However, when the vehicle S passes on the right side, the imaging unit 27 may take a picture of the right side of the vehicle S. Further, the imaging unit 27 may have a camera for photographing the left side and a camera for photographing the right side in the traveling direction of the vehicle S, and may be configured to photograph the left side and the right side respectively. In this case, the control unit 20 executes the process related to the detection of the matrix, which will be described later, based on the captured image data based on the shooting of the left camera and the captured image data based on the shooting of the right camera. .. Further, the imaging unit 27 may have a wide-angle camera for photographing the front of the vehicle S, and the wide-angle camera may be used to photograph the front of the vehicle S.

As described above, the in-vehicle device 12 has a function of detecting the current position of the vehicle S, a function of displaying the current position of the vehicle S on the map, and a route to the destination. It has a function to search for a route to be searched, and a function to display a map, display a route to a destination on the map, and guide a route to the destination. When detecting the position of the own vehicle, the control unit 20 detects the current position of the vehicle S based on the input from the GPS unit 24 and the relative direction detection unit 25 and the map data 211. The current position of the vehicle S may be detected by any method, and at the time of detection, information other than the information exemplified in the present embodiment such as information indicating the vehicle speed may be used. Further, when displaying the vehicle position, the control unit 20 displays a map of a predetermined scale centered on the detected current position on the touch panel 26 based on the map data 211, and also displays a mark indicating the detected current position of the vehicle S on the touch panel 26. Is displayed on the map. Further, when searching for a route, the control unit 20 searches for a route from the detected current position to the destination set by the user based on the map data 211. Further, at the time of route guidance, the control unit 20 displays the route to the destination on the map, displays the current position of the detected vehicle S on the map, and guides the route to the destination.

The control server 10 is a server capable of communicating with the in-vehicle device 12.
As shown in FIG. 2, the control server 10 includes a server control unit 40, a server storage unit 41, and a server communication unit 42.

The server control unit 40 includes a CPU, ROM, RAM, other peripheral circuits, and the like, and controls each unit of the control server 10. The server control unit 40 controls each part of the control server 10 in cooperation with hardware and software, for example, the CPU reads and executes a control program stored in the ROM.
The server storage unit 41 has a non-volatile memory and stores various data.
The server communication unit 42 accesses the network N according to a predetermined communication standard under the control of the server control unit 40, and communicates with a device (including an in-vehicle device 12) connected to the network N.

Note that FIG. 2 is a schematic diagram showing the functional configurations of each device of the information processing system 1 classified according to the main processing contents in order to facilitate understanding of the present invention. The configurations of the in-vehicle device 12 and the control server 10 can be further classified into more components according to the processing content. It can also be categorized so that one component performs more processing. Further, the processing of each component may be executed by one hardware or may be executed by a plurality of hardware. Further, the processing of each component may be realized by one program or may be realized by a plurality of programs.

By the way, depending on the facility, there may be a line depending on the person who wants to use the facility. If a line is generated corresponding to a facility, it can be assumed that the facility is popular, and if the facility is used, it can be assumed that a waiting time will occur before using the facility. Based on this, the information about the matrix generated in response to the facility can be said to be useful information about the facility.
Based on the above, the in-vehicle device 12 and the control server 10 execute the following processes.

FIG. 3 is a flowchart showing the operation of the in-vehicle device 12 and the control server 10. The flowchart FA of FIG. 3 shows the operation of the in-vehicle device 12, and the flowchart FB shows the operation of the control server 10.

Here, the in-vehicle device 12 includes a first mode as an operation mode. The user can set the operation mode to the first mode by a predetermined means. The in-vehicle device 12 executes the process shown in the flowchart FA of FIG. 3 when the operation mode is the first mode.

As shown in the flowchart FA of FIG. 3, the control unit 20 of the vehicle-mounted device 12 acquires the captured image data input from the imaging unit 27 (step SA1).
As described above, the imaging unit 27 photographs the left side of the vehicle S at a predetermined cycle, and outputs captured image data based on the shooting result to the control unit 20. As a result, while the vehicle S is traveling, the imaging unit 27 photographs the left side of the traveling vehicle S at a predetermined cycle, and outputs the captured image data to the control unit 20 at a predetermined cycle. The control unit 20 executes the process of step SA1 at a cycle corresponding to the cycle in which the image pickup unit 27 outputs the captured image data, and is triggered by the execution of the process of step SA1 (acquisition of captured image data) in step SA2 or lower. Execute the processing of. The control unit 20 may execute the process of step SA1 at a cycle different from the cycle in which the imaging unit 27 outputs the captured image data. For example, the imaging unit 27 may output the captured image data in a cycle of 10 ms, while the control unit 20 may execute the process of step SA1 in a cycle of 1 s.

Next, the control unit 20 analyzes the captured image data acquired in step SA1 and determines whether or not a person is included in the image represented by the captured image data (hereinafter, referred to as “captured image”) (step). SA2).
In step SA2, the control unit 20 executes a face recognition process (a process of recognizing a person's face) by an existing technique using pattern matching based on the captured image data, and based on the result of the process, the control unit 20 executes the face recognition process. It is determined whether or not a person is included in the captured image. The process of step SA2 may be performed by any method.
When the captured image does not include a person (step SA2: NO), the control unit 20 ends the process because no matrix is generated in the captured actual space.
When the captured image includes a person (step SA2: YES), the control unit 20 determines whether or not the number of persons included in the captured image exceeds the first threshold value (step SA3). The first threshold value is set to a value such that when the number of people included in the captured image exceeds the first threshold value, a matrix may occur in the captured actual space. The first threshold is, for example, "7 people".

If the number of people included in the captured image does not exceed the first threshold value (step SA3: NO), it is very unlikely that a matrix has occurred or has occurred in the captured real space. , The control unit 20 ends the process.
When the number of people included in the captured image exceeds the first threshold value (step SA3: YES), the control unit 20 executes the matrix detection process (step SA4).

The flowchart FC of FIG. 4 is a flowchart showing the details of the matrix detection process.
The matrix detection process is a process of determining whether or not a matrix is generated in the photographed real space.
In the matrix detection process, the control unit 20 develops the captured image data in a predetermined virtual coordinate system (step SC1). As described above, the captured image data is bitmap data in which dots holding information about colors are arranged in a dot matrix. Therefore, by the process of step SC1, the positions of the dots can be represented by the coordinates in the predetermined virtual coordinate system for all the dots constituting the captured image data.
Next, the control unit 20 identifies the data (hereinafter, referred to as “face area data KD”) corresponding to the face area of the person included in the captured image data (step SC2).

FIG. 5 is a diagram schematically showing a state in which captured image data is developed in a predetermined virtual coordinate system in a mode suitable for explanation. In FIG. 5, the face center dot KP, which will be described later, is also displayed.
In the captured image data shown in FIG. 5, the area surrounded by the dotted line is the area of the human face. In this case, the data (dot group) included in the area surrounded by the dotted line corresponds to the face area data KD.

Next, the control unit 20 identifies a dot at the center of the data (hereinafter, referred to as “face center dot KP”; see FIG. 5) for each of the face region data KDs specified in step SC2, and identifies the face center. Acquire the coordinates of the dot KP (step SC3).
Next, the control unit 20 calculates a region AR having a radius of the first distance centered on the face center dot KP for each of the face center dot KPs, and a group of face center dot KPs in which a portion overlapping the region AR exists. Each of the face center dot KPs is grouped by a method of forming a group (step SC4).
FIG. 6 is a diagram showing a state in which the same captured image data as the captured image data of FIG. 5 is developed in a predetermined virtual coordinate system in order to explain the process of step SC4.
In FIG. 6, the region AR1 related to the face center dot KP1 and the region AR2 related to the face center dot KP2 have overlapping portions. Therefore, the control unit 20 determines that the face center dot KP1 and the face center dot KP2 belong to the same group. Further, the region AR2 related to the face center dot KP2 and the region AR3 related to the face center dot KP3 have overlapping portions. Therefore, the control unit 20 determines that the face center dot KP2 and the face center dot KP3 belong to the same group. Further, although there is no overlapping portion between the region AR1 related to the face center dot KP1 and the region AR3 related to the face center dot KP3, these face center dot KPs belong to the group to which the common face center dot KP2 belongs. Belongs. Therefore, the control unit 20 determines that the face center dots KP1 to KP3 belong to the same common group. As a result, in the example of FIG. 6, the control unit 20 determines that the face center dots KP1 to KP10 belong to the same common group GP1. Further, the control unit 20 determines that the face center dots KP21 to KP23 belong to the same common group GP2.
Here, when a matrix is generated, it is assumed that the plurality of people forming the matrix are lined up in a state where adjacent people are close to each other. Then, the value of the first distance described above is set to reflect the distance assumed as the separation distance of the adjacent persons in the matrix when the matrix is generated. Based on this, each group may be a set of face center dots KP3 of the persons forming the matrix when the matrix is generated.

Next, the control unit 20 determines whether or not the number of face center dot KPs belonging to the group exceeds the above-mentioned first threshold value among the grouped groups (step SC5).
For all groups, if the number of face center dot KPs belonging to the group does not exceed the first threshold value (step SC5: NO), no matrix is generated or a matrix is generated in the photographed real space. Since it is very unlikely that the procession has occurred, the control unit 20 determines that no matrix has been generated (step SC6), and ends the matrix detection process.

When there is at least one group in which the number of face center dot KPs belonging to the group exceeds the first threshold value (step SC5: YES), the control unit 20 controls the group in which the number of face center dot KPs belonging to the group exceeds the first threshold value. Each of the above is determined as a group to be subjected to the group analysis process described below (step SC7). Hereinafter, the group determined as the target group for group analysis processing is referred to as a “processing target group”. In the example of FIG. 6, the control unit 20 determines the group GP1 as the processing target group, while the group GP2 is not the processing target group.

Next, the control unit 20 determines one processing target group among the processing target groups that are not the target of the group analysis processing of step SC9 as the target of the group analysis processing of step SC9 (step SC8). ..
Hereinafter, the group determined as the target of the group analysis process in step SC9 in step SC8 is referred to as an “analysis target group”.
Next, the control unit 20 executes the group analysis process based on the analysis target group (step SC9).

The flowchart FD of FIG. 7 is a flowchart showing the details of the group analysis process.
The group analysis process corresponds to a process of analyzing the positional relationship of a plurality of people included in an image and recognizing a matrix.
As shown in the flowchart FD of FIG. 7, the control unit 20 acquires the distance between the face center dots KPs belonging to the analysis target group in the x-axis direction of the face center dots KPs adjacent to each other (step SD1). In the example of FIG. 6, the control unit 20 sets the distance between the face center dot KP1 and the face center dot KP2 in the x-axis direction (| (x2)-(x1) |), the face center dot KP2 and the face for the group GP1. Separation distance from the center dot KP3 in the x-axis direction (| (x3)-(x2) |), separation distance between the face center dot KP3 and the face center dot KP4 in the x-axis direction (| (x4)-(x3) |) ), The distance between the face center dot KP4 and the face center dot KP5 in the x-axis direction (| (x5)-(x4) |), and the distance between the face center dot KP5 and the face center dot KP6 in the x-axis direction (| (X6)-(x5) |), distance between face center dot KP6 and face center dot KP7 in the x-axis direction (| (x7)-(x6) |), face center dot KP8 and face center dot KP9 The separation distance in the x-axis direction (| (x9)-(x8) |) and the separation distance between the face center dot KP9 and the face center dot KP10 in the x-axis direction (| (x10)-(x9) |) are acquired. ..
In the following description, in the processing target group, the separation distance of the face center dots KP adjacent to each other in the x-axis direction is referred to as “x-axis direction separation distance”.

Next, the control unit 20 calculates the standard deviation based on each of the x-axis direction separation distances acquired in step SD1 (step SD2).
Here, when a plurality of people form a matrix, it is estimated that the distance between adjacent people is substantially constant. This is because the matrix has a regularity (characteristic) that a plurality of people are arranged side by side at equal intervals. Based on this, when each of the persons corresponding to each of the face center dot KPs belonging to the group constitutes a matrix, the variation in the x-axis direction separation distance is small (= each of the x-axis direction separation distances is substantially constant). There is a regularity (characteristic) that the variation in the separation distance in the y-axis direction, which will be described later, is small.

Next, the control unit 20 determines whether or not the standard deviation calculated in step SD2 is below a predetermined second threshold value (step SD3).
The second threshold value is a value such that the standard deviation based on the x-axis direction separation distance is lower than the second threshold value when each of the persons corresponding to each of the face center dot KPs belonging to the group constitutes a matrix. Is set to.
When the standard deviation does not fall below the second threshold value (step SD3: NO), the control unit 20 determines that the matrix corresponding to the analysis target group has not occurred (step SD4), and ends the group analysis process.

When the standard deviation is lower than the second threshold value (step SD3: YES), the control unit 20 acquires the distance between the face center dots KPs belonging to the analysis target group in the y-axis direction of the face center dots KPs adjacent to each other. (Step SD5). In the example of FIG. 6, the control unit 20 sets the distance between the face center dot KP1 and the face center dot KP2 in the y-axis direction (| (y2)-(y1) |), the face center dot KP2 and the face for the group GP1. Distance from the center dot KP3 in the y-axis direction (| (y3)-(y2) |), distance between the face center dot KP3 and the face center dot KP4 in the y-axis direction (| (y4)-(y3) |) ), The separation distance between the face center dot KP4 and the face center dot KP5 in the y-axis direction (| (y5)-(y4) |), and the separation distance between the face center dot KP5 and the face center dot KP6 in the y-axis direction (| (Y6)-(y5) |), the distance between the face center dot KP6 and the face center dot KP7 in the y-axis direction (| (y7)-(y6) |), the face center dot KP8 and the face center dot KP9. The separation distance in the y-axis direction (| (y9)-(y8) |) and the separation distance between the face center dot KP9 and the face center dot KP10 in the y-axis direction (| (y10)-(y9) |) are acquired. ..
In the following description, in the processing target group, the separation distance of the face center dots KP adjacent to each other in the y-axis direction is referred to as “y-axis direction separation distance”.

Next, the control unit 20 calculates the standard deviation based on each of the y-axis direction separation distances acquired in step SD5 (step SD6).
As described above, when each of the persons corresponding to each of the face center dot KPs belonging to the group constitutes a matrix, the variation in the x-axis direction separation distance is small and the variation in the y-axis direction separation distance is small. There is a regularity (characteristic) that it is small.
Next, the control unit 20 determines whether or not the standard deviation calculated in step SD6 is below a predetermined third threshold value (step SD7).
The third threshold value is a value such that the standard deviation based on the y-axis direction separation distance is lower than the third threshold value when each of the persons corresponding to each of the face center dot KPs belonging to the group constitutes a matrix. Is set to.
When the standard deviation does not fall below the third threshold value (step SD7: NO), the control unit 20 determines that the matrix corresponding to the analysis target group has not been generated (step SD4), and ends the group analysis process.
When the standard deviation is less than the third threshold value (step SD7: YES), the control unit 20 determines that a matrix corresponding to the analysis target group is generated (step SD8), and ends the group analysis process.

As shown in the flowchart FC of FIG. 4, after executing the group analysis process, the control unit 20 determines whether or not the group analysis process is completed for all the processing target groups (step SC10).
If there is a processing target group for which even one group analysis processing has not been completed (step SC10: NO), the control unit 20 shifts the processing procedure to step SC8.
When the group analysis process is completed for all the processing target groups (step SC10: YES), the control unit 20 determines whether or not there is at least one processing target group that has been determined to have a matrix (step SC10: YES). Step SC11).

When there is no processing target group determined to have a matrix (step SC11: NO), the control unit 20 determines that no matrix has been generated (step SC6), and ends the matrix detection process.
When there is at least one processing target group determined to have a matrix (step SC11: YES), the control unit 20 determines that a matrix has been generated (step SC12), and ends the matrix detection process. ..
Hereinafter, the processing target group determined to have generated a matrix is referred to as a "matrix group".

The matrix may be one column or may be composed of a plurality of "columns" having two or more columns. Even when the matrix is composed of a plurality of "columns", the X-axis direction separation distance and the y-axis direction separation distance vary with respect to the face center dot KP corresponding to the plurality of persons forming the matrix. Since the regularity (feature) of being small can be applied, the matrix can be accurately detected by the group analysis process described above.
However, when the matrix is composed of a plurality of "columns", the following processing may be executed in the group analysis process to detect the matrix more accurately.

FIG. 8 is a diagram schematically showing photographed image data generated by photographing a matrix composed of two columns in an embodiment suitable for explanation.
In FIG. 8, the face center dot KP is clearly shown in the captured image data. Further, in FIG. 8, each of the face center dot KPs belonging to the group GP3 is a dot corresponding to each of a plurality of persons forming a matrix related to the two columns.
In the following description, a matrix composed of n (n is a positive integer) "columns" is composed of a plurality of "rows" composed of n people arranged side by side in the direction in which the matrix extends. Shall be. For example, in the case where the matrix is composed of two "columns", the first column and the second column, one "row" is formed by the first person in the first column and the first person in the second column. , And one "row" is composed of the second person from the beginning of the first column and the second person from the beginning of the second column.

Here, when the matrix is composed of a plurality of columns, the matrix has the following regularity (characteristics). That is, in each "row", there is a regularity (characteristic) that the distances between adjacent persons are close to each other.
Based on this, in the group analysis process, the control unit 20 has a radius centered on the face center dot KP for each of the face center dot KPs belonging to the analysis target group as a second distance (however, the second distance <first distance). The area BR of.) Is calculated, and each of the face center dot KPs is grouped by a method of forming a group of face center dot KPs in which a portion overlapping the area BR exists as one group. The value of the second distance is set to reflect the distance assumed as the separation distance of adjacent people in the "rows" constituting the matrix when a matrix consisting of a plurality of "columns" is generated. Based on this, each group may be a set of face center dots KP3 of people constituting the "row" when a matrix composed of a plurality of "columns" is generated.
In the example of FIG. 8, the control unit 20 has the face center dot KP31a and the face center dot KP31b in the same group, the face center dot KP32a and the face center dot KP32b in the same group, and the face center dot KP33a and the face center dot KP33b. The face center dot KP34a and the face center dot KP34b are in the same group, the face center dot KP35a and the face center dot KP35b are in the same group, and the face center dot KP36a and the face center dot KP36b are in the same group. , Face center dot KP37a and face center dot KP37b are in the same group, face center dot KP38a and face center dot KP38b are in the same group, face center dot KP39a and face center dot KP38b are in the same group, and face center dot KP40a. And the face center dot KP40b are in the same group.

Next, the control unit 20 calculates the coordinates of the dots located at the center of the face center dot KP belonging to each group. For example, when there are two face center dot KPs belonging to the group, the control unit 20 calculates the coordinates of the dots located at the center of the straight line having the two face center dot KPs as end points. Further, for example, when the number of face center dot KPs belonging to the group is three or more, the control unit 20 calculates the coordinates of the dots located at the center of gravity of the polygon having each of the face center dot KPs as vertices. In the example of FIG. 8, the control unit 20 calculates the coordinates of the face center dot KP31c for the group GP31, calculates the coordinates of the face center dot KP32c for the group GP32, and calculates the coordinates of the face center dot KP33c for the group GP33. Calculate the coordinates of the face center dot KP34c for the group GP34, calculate the coordinates of the face center dot KP35c for the group GP35, calculate the coordinates of the face center dot KP36c for the group GP36, and calculate the coordinates of the face center dot KP37c for the group GP37. Then, the coordinates of the face center dot KP38c are calculated for the group GP38, the coordinates of the face center dot KP39c are calculated for the group GP39, and the coordinates of the face center dot KP40c are calculated for the group GP40.
Hereinafter, the dot located at the center of the face center dot KP belonging to each group is referred to as "row center dot GT".

Next, the control unit 20 calculates the standard deviation based on the x-axis direction separation distance and the standard deviation based on the y-axis direction separation distance for the row center dot GT belonging to the analysis target group, and based on these standard deviations, Whether or not a matrix is generated is determined by the same method as described above.
According to the method described above, when a matrix composed of a plurality of "columns" is generated, it can be detected more accurately.

As shown in the flowchart FA of FIG. 3, after executing the matrix detection process of step SA4, the control unit 20 determines whether or not it is determined that the matrix is generated in the matrix detection process (step SA5). ..
When it is determined in the matrix detection process that no matrix has been generated (step SA5: NO), the control unit 20 ends the process.

In the following description, for convenience of explanation, it is assumed that there is one matrix group. When a plurality of matrix groups are detected in the matrix detection process, the control unit 20 executes the following process for each matrix group.

When it is determined in the matrix detection process that a matrix has been generated (step SA5: YES), the control unit 20 acquires the matrix generation date and time information and the matrix scale information (step SA6).
The procession generation date and time information is information indicating the date and time when the imaging unit 27 took a picture. The date and time indicated by the matrix generation date and time information corresponds to the date and time when the matrix was taken.
The matrix scale information is information indicating the scale of the matrix. In the present embodiment, the concept of the scale of the matrix includes "large scale", "medium scale", and "small scale" depending on the number of persons forming the matrix. In step SA6, the face center dot belonging to the matrix group is compared with the threshold value corresponding to the “large scale” and the threshold value corresponding to the “medium scale”, and the matrix scale information is acquired based on the comparison result.

Next, the control unit 20 analyzes the captured image data and acquires information indicating the position of the head of the matrix corresponding to the matrix group (step SA7). The information indicating the position of the head of the matrix is information indicating the position of the head of the matrix by a combination of longitude and latitude.
For example, the control unit 20 acquires the position of the vehicle S when shooting is executed by the imaging unit 27 based on the input from the GPS unit 24, the input from the relative direction detection unit 25, and the map data 211. Next, the control unit 20 heads the matrix corresponding to the matrix group based on the acquired position of the vehicle S, the shooting direction of the imaging unit 27, and the position of the face center dot KP belonging to the matrix group in the captured image data. Get the position of.
The method of acquiring the position of the head of the matrix corresponding to the matrix group is not limited to the illustrated method, and any method may be used.
Hereinafter, the information indicating the head position of the matrix acquired by the control unit 20 in step SA7 is referred to as "matrix position information".

Next, the control unit 20 searches for a facility located at a position corresponding to the position indicated by the matrix position information based on the facility data 212 (step SA8). Here, as described above, the facility data 212 has facility identification information for identifying the facility and facility location information indicating the location of the facility for each facility on the map. In step SA8, based on the facility data 212, if there is a facility whose distance from the head position of the matrix is within a predetermined distance range, the matrix 20 is among the facilities within the range. Identify the facility located closest to the beginning of the. On the other hand, when the control unit 20 does not have a facility whose distance from the head position of the matrix is within a predetermined distance range, the control unit 20 does not have a facility located at a position corresponding to the position indicated by the matrix position information. Determine.
The facility located at the position corresponding to the position indicated by the matrix position information is set as the facility where the matrix is generated.

Next, in the process of step SA8, the control unit 20 determines whether or not the facility located at the position corresponding to the position indicated by the matrix position information can be specified (step SA9).
If the facility cannot be specified (step SA9: NO), the control unit 20 ends the process.
When the facility can be specified (step SA9: YES), the control unit 20 acquires the facility identification information of the specified facility based on the facility data 212 (step SA10).
Next, the control unit 20 controls the communication unit 23 to transmit the facility identification information acquired in step SA10, the matrix occurrence date / time information acquired in step SA6, and the matrix scale information to the control server 10.
Information necessary for communicating with the control server 10 and transmitting information to the control server 10, such as the address of the control server 10, the communication standard used for communication, and the format of the data to be transmitted to the control server 10, can be obtained. Registered in advance.

As shown in the flowchart FB of FIG. 3, the server control unit 40 of the control server 10 controls the server communication unit 42 to receive facility identification information, matrix generation date / time information, and matrix scale information (step SB1).
Next, the server control unit 40 updates the matrix relational database 411 based on the received information (step SB2).
The matrix-related database 411 is a database that has a record for each facility on the map and stores facility identification information, matrix occurrence date / time information, and matrix scale information in each record in association with each other. In step SB2, the server control unit 40 identifies a record having the facility identification information having the same value as the received facility identification information value among the records of the matrix relational database 411. Next, the server control unit 40 updates the matrix occurrence date / time information and the matrix scale information of the specified record with the received matrix occurrence date / time information and the matrix scale information, respectively.

Here, there are a plurality of vehicles S on which the in-vehicle device 12 according to the present embodiment is mounted. The plurality of vehicle-mounted devices 12 appropriately analyze the captured image data, and transmit the facility identification information, the matrix generation date / time information, and the matrix scale information to the control server 10. As a result, the value of the matrix occurrence date and time information and the value of the matrix scale information of each record of the matrix relational database 411 are the latest matrix occurrence date and time information and the matrix scale information received from any of the in-vehicle devices 12, respectively. It is a value based on.

As shown in the flowchart FA of FIG. 3, after transmitting the information to the control server 10, the control unit 20 executes the corresponding processing based on the acquired facility identification information, the matrix occurrence date / time information, and the matrix scale information. (Step SA12). The process executed by the control unit 20 in step SA12 is registered in advance by the user, for example.
In step SA12, for example, the control unit 20 executes the following processing.
For example, the control unit 20 displays information on the facility where the queue is generated on the touch panel 26 at a predetermined timing. The predetermined timing is, for example, the timing at which the vehicle S has stopped, and for example, the timing at which a predetermined instruction has been given by the user.
FIG. 9 is a diagram showing an example of information displayed on the touch panel 26 by the process of step SA12.
In the example of FIG. 9, in the left half area, information indicating the name of the facility where the matrix is generated, information indicating the category of the facility, and information indicating the scale of the matrix are displayed. By referring to this information, the user can easily and accurately recognize the name of the facility, the category of the facility, and the scale of the generated matrix for the facility where the matrix is generated. Here, the facility where the queue is generated is assumed to be a popular facility, and it is assumed that the user may be interested in it. According to the present embodiment, the user can acquire information about the facility that the user may be interested in without performing complicated work such as searching using the Internet. Highly convenient for users.
In the example of FIG. 9, a map is displayed in the right half area, and a mark M1 indicating the current position of the vehicle S and a mark M2 indicating the facility where the procession has occurred are displayed on the map. By referring to the map, the user can easily and accurately recognize the positional relationship between the vehicle S and the facility where the queue is generated.

The process of step SA12 has been described with an example, but in step SA12, the control unit 20 may execute a process different from the illustrated process. For example, when the in-vehicle device 12 has a function of outputting voice, it may be configured to output information about the facility where the matrix is generated and information about the matrix by voice.

Next, the facility search process that can be executed by the in-vehicle device 12 will be described.
The facility search process is a process of searching for a facility according to a user's instruction. As described below, the control unit 20 of the vehicle-mounted device 12 also provides the user with useful information about the matrix when searching for a facility.

The flowchart FE of FIG. 10 shows the operation of the in-vehicle device 12 when executing the facility search process, and the flowchart FF shows the operation of the control server 10.

As shown in the flowchart FE of FIG. 10, the control unit 20 of the in-vehicle device 12 monitors whether or not the user has instructed to start the search for the facility (step SE1). The user operates the operation switch 221 and touch-operates the touch panel 26 to give an instruction.
When it is detected that there is an instruction to start the facility search (step SE1: YES), the control unit 20 displays the category of the facility to be searched and the user interface for inputting the search conditions on the touch panel 26 (step). SE2).
The search condition is a condition that represents the range of facilities to be searched. For example, the search condition is "a facility located in the vicinity of the current position of the vehicle S (for example, within a predetermined range centered on the vehicle S)". Further, for example, the search condition is "a facility located in the vicinity of the destination (for example, within a predetermined range centered on the destination)". Further, for example, the search condition is "a facility located along the route from the current position of the vehicle S to the destination".

The control unit 20 has input of the category and the search condition in the user interface, and monitors whether or not the input is confirmed (step SE3).
When it is detected that the input is confirmed (step SE3: YES), the control unit 20 acquires the input category and the search condition (step SE4).
Next, the control unit 20 refers to the map data 211 and the facility data 212, and searches for and identifies the category and the facility that matches the search condition based on the category acquired in step SE4 and the search condition (). Step SE5).
In the following, for convenience of explanation, the processing of the control unit 20 will be described assuming that a plurality of facilities are specified as the categories acquired in step SE4 and the facilities that match the search conditions by the processing of step SE4.

Next, the control unit 20 controls the communication unit 23 and transmits the response request data to the control server 10 (step SE6). The response request data includes facility identification information of each of the plurality of facilities identified in step SE5, and is data for requesting transmission of matrix generation date / time information and matrix scale information for each facility.

As shown in the flowchart FF of FIG. 10, the server control unit 40 of the control server 10 controls the server communication unit 42 to receive the response request data (step SF1).
Next, the server control unit 40 refers to the matrix relational database 411 and acquires the matrix generation date and time information and the matrix scale information associated with the facility identification information for each of the facility identification information included in the response request data ( Step SF2).
Next, the server control unit 40 controls the server communication unit 42 to transmit the matrix generation date / time information and the matrix scale information acquired in step SF2 to the in-vehicle device 12 (step SF3).
In the matrix relational database 411, the default values of the matrix occurrence date and time information and the matrix scale information are null values. Therefore, when the matrix occurrence date / time information corresponding to one facility and the matrix scale information have never been updated, the values of the matrix occurrence date / time information and the matrix scale information corresponding to the one facility are null values. is there.
When the server control unit 40 deviates from the date and time indicated by the matrix occurrence date and time information of one facility by a certain period or more, the matrix occurrence date and time information and the matrix scale information of the one facility are set as default values. It may be a configuration to be updated.

As shown in the flowchart FF of FIG. 10, the control unit 20 of the in-vehicle device 12 controls the communication unit 23 to receive the matrix generation date / time information and the matrix scale information for each facility (for each facility identification information) (step). SE7).
Next, the control unit 20 displays the search result display screen G1 on the touch panel 26 using the received matrix generation date and time information and the matrix scale information (step SE8).

FIG. 11 is a diagram showing an example of the search result display screen G1.
FIG. 11 shows a search result display screen G1 when the search condition is “a facility located near the current position of the vehicle S” and the category is “ramen shop”.
On the search result display screen G1 illustrated in FIG. 11, a map of a predetermined scale centered on the position of the vehicle S is displayed, and a mark M2 representing the vehicle S is displayed on the map. Further, among the facilities displayed on the map on the search result display screen G1, the facilities specified in step SE5 are given a code consisting of alphabets and match the category and search conditions specified by the user. Is clearly stated. Hereinafter, among the facilities displayed on the search result display screen G1, the facility specified in step SE5 is referred to as a “specific facility”.

As shown in FIG. 11, in the map displayed on the search result display screen G1, balloons are displayed in association with each of the specific facilities, and the matrix occurrence date and time information and the matrix scale information are displayed in the balloons. Is displayed. If the matrix occurrence date / time information and the matrix scale information are null values, information indicating that there is no information about the matrix is displayed.
The user can estimate the status of the queue of a specific facility by referring to the information displayed in the balloon. In addition, the user can use the information about the matrix displayed in the balloon as useful information when selecting a specific facility to be actually used. For example, even if there is a waiting time, a user who wants to use a popular facility may preferentially select a facility whose queue size is "large". it can.

As shown in FIG. 11, on the search result display screen G1, block BKs are collectively displayed for each specific facility on the left side of the map. In the block BK, a code attached to the corresponding facility, information indicating the name of the facility, and information indicating the mileage when the vehicle S travels to the facility are displayed. The block BK is a touch-operable button. The user touch-operates the block BK corresponding to the facility to be used. When the control unit 20 detects that any of the block BKs has been touch-operated, the control unit 20 starts route guidance with the facility (specific facility) corresponding to the touch-operated block BK as the destination.

As described above, the control unit 20 of the vehicle-mounted device 12 according to the present embodiment detects whether the image acquired by the image pickup unit 27 mounted on the vehicle S includes a plurality of people, and detects the plurality of people. In that case, the positional relationship of the plurality of persons is analyzed, and if it can be recognized as a matrix, the facility corresponding to the recognized matrix is searched, and the information about the searched facility is displayed (output).
Here, as described above, if a line is generated corresponding to a facility, it can be assumed that the facility is popular, and when using the facility, there is a waiting time before using the facility. It can be assumed that it will occur. Therefore, the information about the facility where the queue is generated is useful information for knowing the facility. Based on this, according to the above configuration, it is possible to execute a useful process of outputting information about the facility corresponding to the matrix by analyzing the image acquired by the image pickup unit 27 mounted on the vehicle S.
As described above, the method of outputting information may be display or audio output.

Further, in the present embodiment, when the control unit 20 can recognize that it is a matrix, it acquires the scale of the recognized matrix and outputs information indicating the scale of the matrix.
Here, the scale of the queue generated in response to the facility affects the waiting time when the facility is used, and is therefore useful information when deciding whether or not to use the facility. Based on this, according to the above configuration, the control unit 20 can output useful information such as information indicating the scale of the matrix.

Further, in the present embodiment, the control unit 20 determines whether or not a plurality of persons included in the image are lined up with a predetermined regularity based on the image acquired by the imaging unit 27, and is included in the image. When a plurality of people are lined up with a predetermined regularity, it is recognized as a matrix.
According to this configuration, the control unit 20 can accurately recognize the matrix by analyzing the image, reflecting the characteristics of the matrix.

Further, in the present embodiment, the control unit 20 has a plurality of persons included in the image having predetermined regularity when the variation in the separation distance between the adjacent persons in a predetermined direction is small. It is determined that they are lined up with.
According to this configuration, the control unit 20 can accurately recognize the matrix by analyzing the image, reflecting the characteristics of the matrix.

Further, the information processing system 1 according to the present embodiment includes an in-vehicle device 12 and a control server 10 capable of communicating with the in-vehicle device 12.
The in-vehicle device 12 detects whether or not the image acquired by the imaging unit 27 mounted on the vehicle S includes a plurality of persons, and when the plurality of persons are detected, analyzes the positional relationship of the plurality of persons in a matrix. If it can be recognized, the facility corresponding to the recognized matrix is searched, and the information about the searched facility (facility identification information) and the information about the recognized matrix (matrix occurrence date / time information, matrix scale information) are sent to the control server 10. Send.
The control server 10 stores information about the facility and information about the matrix received from the in-vehicle device 12.
According to this configuration, the control server 10 can execute processing using the information about the facility received and stored from the in-vehicle device 12 and the information about the matrix.

Further, in the present embodiment, the control server 10 responds to an inquiry from an external device (in the present embodiment, the in-vehicle device 12) with information regarding a matrix for a specific facility.
This configuration allows an external device to obtain information about the matrix for a particular facility.

It should be noted that the above-described embodiment shows only one aspect of the present invention, and can be arbitrarily modified and applied within the scope of the present invention.
For example, the processing unit of the flowchart shown in the figure is divided according to the main processing contents in order to make the processing of each device easy to understand. The present invention is not limited by the method of dividing the processing unit and the name. The processing of each device can be divided into more processing units according to the processing content. Further, one processing unit can be divided so as to include more processing. Further, if the same processing can be performed, the processing order of the above flowchart is not limited to the illustrated example.

1 Information processing system 10 Control server (server)
12 In-vehicle device 20 Control unit 21 Storage unit 22 Operation unit 23 Communication unit 24 GPS unit 25 Relative orientation detection unit 26 Touch panel 27 Imaging unit 40 Server control unit 41 Server storage unit 42 Server communication unit 211 Map data 212 Facility data 221 Operation switch 262 Touch sensor 411 matrix-related database

Claims (6)

When it is detected whether or not the image acquired by the image pickup unit mounted on the vehicle contains a plurality of people, and when a plurality of people are detected, the positional relationship of the plurality of people is analyzed and it can be recognized as a matrix. Search for the facility corresponding to the position at the beginning of the recognized matrix,
When a plurality of the matrices are recognized, the head position of each matrix is calculated, the corresponding facility is searched for each head position, and the information about the facility is associated with the information about the matrix.
When a search for a facility based on a predetermined search condition is accepted and information about the matrix is added to the facility extracted as a result of the search, the information about the facility extracted as a result of the search and the facility are added to the facility. further comprising a control unit which forces out the information about it has the matrix data relating to the overlapping maps the display panel-vehicle apparatus according to claim. The first aspect of claim 1, wherein the control unit determines whether or not the image acquired by the imaging unit includes a person's face, and calculates the position of the head of the matrix based on the image. In-vehicle device. The control unit
The vehicle-mounted device according to claim 1 or 2 , wherein when it can be recognized as a matrix, the scale of the recognized matrix is acquired and information indicating the scale of the matrix is output. The control unit
Based on the image acquired by the imaging unit, it is determined whether or not a plurality of persons included in the image are lined up with a predetermined regularity, and a state in which the plurality of persons included in the image are lined up with a predetermined regularity. The vehicle-mounted device according to any one of claims 1 to 3, wherein the vehicle-mounted device is recognized as a matrix. The control unit
A feature of a plurality of persons included in an image is that when the variation in the separation distance between adjacent persons in a predetermined direction is small, it is determined that the plurality of persons included in the image are arranged side by side with a predetermined regularity. The in-vehicle device according to claim 4. An information processing system including an in-vehicle device and a server capable of communicating with the in-vehicle device.
The in-vehicle device is
When it is detected whether or not the image acquired by the imaging unit mounted on the vehicle contains a plurality of people, and when a plurality of people are detected, the positional relationship of the plurality of people is analyzed and it can be recognized as a matrix. The facility corresponding to the recognized matrix is searched, and when a plurality of the matrix is recognized, the head position of each matrix is calculated, the corresponding facility is searched for each head position, and the information about the searched facility is searched. And the information about the recognized head position of the matrix and the facility corresponding to the head position are transmitted to the server.
When a search for a facility based on a predetermined search condition is accepted and information about the matrix is added to the facility extracted as a result of the search, the information about the facility extracted as a result of the search and the facility are added to the facility. Outputs the data related to the map that overlaps with the information about the above matrix.
The server
Receives information about the facility and information about the matrix, and stores the information about the facility and the information about the matrix in association with each other .
An information processing system characterized in that information about the facility corresponding to the facility extracted as a result of the search based on the predetermined search condition and information about the matrix are transmitted to the vehicle-mounted device.

Images