JP7058708B1 - Boarding judgment system, setting program, mobile terminal, and management server - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a boarding determination system capable of simply and highly accurately determining the correctness of selection of a vehicle selected by a user. SOLUTION: The management server 3 has a mobile terminal 1 for acquiring necessary train information from an information providing server 2 for providing train information, and a management server 3 for acquiring a photographed image together with text information from the mobile terminal 1. Receives a photographed image taken under a predetermined shutter speed, an aperture value specified by the mobile terminal 1, and shooting conditions of ISO sensitivity specified corresponding to the shutter speed and the aperture value, and the photographed image thereof. Returns the judgment result of whether or not it is appropriate in relation to the character information. [Selection diagram] FIG. 4

Description

The present invention relates to a boarding determination system that can easily and surely determine whether or not the vehicle to be boarded is correct, and is particularly suitable for tourists.

It is not easy for tourists who are not familiar with the local geography to know exactly which train or bus to board, and especially for foreign tourists, the difficulty is remarkable. Therefore, staff members such as railway station staff are frequently asked questions by tourists, but if the labor shortage is regarded as a problem and people continue to respond as it is, the service level will be maintained. May be difficult.

Therefore, in consideration of this point, a system has been proposed in which an image of a vehicle or a guide display board taken by a camera of a mobile terminal is determined and appropriate information is returned to the mobile terminal (Patent Documents 1 to 1). 2).

For example, Patent Document 1 describes "a first means for acquiring a guide route (data) (searching for a guide route from a departure point to a destination that satisfies the route search condition received from a terminal device), and a photographing unit. The second means for acquiring the photographed image of the boarding guide display of the transportation system, the third means for extracting the display content in the photographed image, the guide route, and the route specified from the display content. , A fourth means for generating guidance information including whether or not the route is appropriate based on at least one of the station name and the direction of travel, and a fifth means for outputting the guidance information. Information processing system "is disclosed.

Japanese Patent No. 6017636 Japanese Patent No. 5775076

In the configuration described in Patent Document 1, it is an essential requirement to acquire a guide route, but it is not practically easy to realize the first means and the fourth means. To confirm this point concretely, first, the user needs to operate the terminal device to perform complicated processing of transmitting the route search condition including the departure point and the destination to the navigation server.

Moreover, the navigation server needs to search the guidance route from the departure point to the destination that satisfies the route search condition from the transportation network database and generate the guidance route data, and this processing also has a large processing burden on the navigation server. It becomes.

Considering this point, it is desired to complete a system in which the processing load on the user side is small and the processing load on the server side is also light. In particular, it is desired to complete a system that can optimally operate the built-in camera of the mobile terminal.

The present invention has been made in view of the above problems, and by optimally operating the built-in camera of the mobile terminal, it is possible to easily and highly accurately determine whether or not the vehicle selected by the user has been selected. The purpose is to provide a boarding determination system.

In order to achieve the above object, the boarding determination system according to claim 1 includes an information providing server that provides traffic information of transportation, and a mobile terminal configured to be able to acquire necessary traffic information from the information providing server. The mobile terminal is configured to include a photographed image related to traffic information and a management server configured to be able to acquire character information corresponding to the photographed image, and the management server has a predetermined shutter speed and the mobile terminal. In response to a shooting image taken under the shooting conditions of the aperture value specified in the above and the shooting conditions of the predetermined ISO sensitivity specified corresponding to the shutter speed and the aperture value, the shot image is the text information. It is configured to return the determination result of whether or not it is appropriate in relation to the above-mentioned mobile terminal.

Further, the boarding determination system according to claim 2 is composed of an information providing server that provides traffic information of transportation, a mobile terminal configured to be able to acquire necessary traffic information from the information providing server, and the mobile terminal. It is configured to have a captured image related to traffic information and a management server configured to be able to acquire text information corresponding to the captured image, and the management server is configured to capture a plurality of captured images taken continuously or to shoot a moving image. In response to the captured image of the moving image frame, the determination result of whether or not the captured image is appropriate in relation to the character information is returned to the mobile terminal.

In any of the above inventions, since the photographed image related to the traffic information is transmitted from the mobile terminal together with the corresponding character information, the management server only needs to determine the consistency between the photographed image and the character information, and controls the image. The burden is greatly reduced. That is, the management server does not need to search for a guide route from the departure point to the destination that satisfies the given condition (route search condition).

In addition, since the management server receives the character information corresponding to the photographed image from the mobile terminal together with the photographed image, the correctness or rejection of the vehicle selection selected by the user based only on the photographed image and the character information without going through the character recognition process. Information can be returned.

On the other hand, according to claim 1 , the present invention describes a predetermined shutter speed, an aperture value specified by a mobile terminal, and shooting conditions of a predetermined ISO sensitivity specified corresponding to the shutter speed and the aperture value. It is also a setting program for setting in the mobile terminal of the above, or a mobile terminal in which the setting program is installed.

Further, the present invention is a photographed image taken under a predetermined shutter speed, an aperture value specified by a mobile terminal, and a shooting condition of a predetermined ISO sensitivity specified corresponding to the shutter speed and the aperture value. Is received from the mobile terminal together with the corresponding character information, and the determination result of whether or not the captured image is appropriate in relation to the character information is returned to the mobile terminal. It is also a server.

By using the above invention, the mobile terminal can be operated under the optimum conditions, and the determination accuracy of the management server can be improved.

Transportation includes at least trains, buses, planes, ships, and taxis. The information providing server may be in the same domain as the management server, or may be in a different domain.

The image capture data may be a still image or a moving image. The imaging data is typically data obtained by photographing a side plate (destination indicator) of a transportation system such as a train. However, it may be a photograph of a train information display board placed on a station platform (Platform) or the like.

According to the present invention, by optimally operating the built-in camera of the mobile terminal, it is possible to easily and highly accurately determine whether or not the vehicle selected by the user has been selected.

This is the result of examining the optimum shutter speed when shooting the destination indicator on the side of the train. This is the result of examining the optimum ISO sensitivity for the train information display boards and destination indicators of the 1st to 3rd stations. This is the result of examining the optimum ISO sensitivity for the train information display boards and destination indicators of the 4th to 6th stations. It is a block diagram which shows the whole structure of the boarding determination system which concerns on Example. It is a drawing which illustrates the Web page distributed to a mobile terminal. It is a figure which shows the processing result of the image restoration processing and the reflection removal processing. It is a figure explaining the method of specifying the threshold value TH2 of a V value from the appearance histogram of the V value which shows the lightness (Value). It is a block diagram which shows the whole structure of the boarding determination system which concerns on another Example.

Hereinafter, the present invention will be described in detail based on examples. First, a shooting survey conducted to specify the shooting conditions for optimally operating the built-in camera of the mobile terminal and the optimum shooting conditions obtained from the survey results will be described.
<Specification of optimum shooting conditions>

The present inventor examined the shooting conditions under which the train guide display board installed on the station platform and the destination display on the side of the train can be accurately photographed by the camera. There are two types of destination indicators on the side of the train, one is a "subtitle type" in which characters are printed on a sheet, and the other is an "LED type" in which an LED indicator is used. Further, as the train guide display board, a "liquid crystal type" using a liquid crystal display is generally used.

In addition, whether or not the train information display board and destination display can be accurately photographed using the built-in camera of a mobile terminal such as a smartphone depends on the total amount of light passing through the camera lens and the brightness of the image itself. In relation to this, these are determined by the combination of the aperture value of the lens, the exposure time, and the film sensitivity (actually, the sensitivity of the image pickup element).
[Survey Study 1 (Shutter speed)]

Therefore, first, the optimum shutter speed was examined. Here, the shutter speed means the time (exposure time) in which the shutter is opened and the film or image pickup element is exposed to the light passing through the lens when taking a picture with a camera. The shorter the exposure time, the more the shutter. The faster the speed and the longer the exposure time, the slower the shutter speed.

In this shutter speed survey, for the built-in camera of the mobile terminal, (1) image stabilization function = OFF setting, (2) aperture value = automatic setting, (3) ISO sensitivity = automatic setting, and the shutter speed was set at each station. , 1/4000 seconds, 1/1000 seconds, 1/400 seconds, 1/125 seconds, 1/5 seconds, etc., while taking a large number of images and visually checking whether the shot images are appropriate or not. Judged.

The appropriateness of the captured image was judged from the viewpoint of whether or not the image recognition by the computer was possible, regardless of whether the captured image was bright or dark, and the highest priority condition was that the characters were not interrupted. This is because LED-type destination indicators often employ a so-called dynamic lighting method, and if the shutter speed is too fast, only interrupted characters can be photographed. In addition, not all pixels of the "liquid crystal type" train guide display board are lit at the same time, and there is a concern that characters may be interrupted.

First, as shown in FIG. 1 (a), according to the survey results of the "subtitle type" destination display, it was concluded that shooting was possible regardless of the shutter speed, although camera shake occurred when the shutter speed was 1/5 second. rice field.

Next, as shown in FIG. 1 (b), the "liquid crystal type" train guide display board was successfully photographed except for camera shake at 1/5 second and 1/20 second. Therefore, it can be concluded that even the "liquid crystal type" train guide display board can be photographed regardless of the shutter speed.

In the "LED type" destination indicator, the operation mode of dynamic lighting is not always constant. Therefore, characters may be interrupted even at a shutter speed of 1/160 second, but if the shutter speed is faster than 1/1000 second, it was almost impossible to shoot accurately (see FIG. 1 (c)). On the other hand, if the shutter speed is too slow, there is a risk of camera shake, so it is concluded that 1/40 to 1/125 seconds, when there was no failure, is optimal.

As described above, (1) in the case of the subtitle type, there is no problem if the shutter speed is faster than 1/5 second, and (2) in the case of the liquid crystal type, there is no problem if the shutter speed is faster than 1/40 second. It was clarified that (3) in the case of the LED type, there is no problem if the shutter speed is 1/40 to 1/125 seconds. Therefore, the present inventor has concluded that a shutter speed of 1/40 to 1/125 second is a suitable numerical range.
[Survey Study 2 (Brightness Survey)]

Then, in order to derive the setting value that can be taken with the correct brightness, the brightness was investigated and examined. The brightness is related to the shutter speed, the ISO sensitivity, and the aperture value.

First, ISO sensitivity is an ISO standard (International Organization for Standardization) for photographic film (an image sensor in a mobile terminal), and indicates how weak a certain image sensor can handle light. For example, an image sensor with an ISO sensitivity of 200 has a light receiving capacity twice that of ISO sensitivity 100, so that it can handle light with half the intensity of ISO sensitivity 100.

This ISO sensitivity is expressed in logarithmic notation I'or arithmetic notation I, but between the logarithmic notation I'and the arithmetic notation I, I = 10 ^{(I'-1) / 10} ... (Equation 1). Or, an approximate expression of I'= 10Log ₁₀ I + 1 ... (Equation 2) is established. In (Equation 1), the arithmetic notation I is rounded to the standard sensitivity of the closest value, and in (Equation 2), the logarithmic notation I'is rounded to an integer.

Therefore, for example, the ISO sensitivity 20 (arithmetic notation I = 20) is I'= 10Log ₁₀ 20 + 1≈14 in logarithmic notation. That is, the ISO sensitivity (arithmetic notation) 20 is 14 in logarithmic notation, and the relationship of 20≈10 ^1.3 is established.

Further, the aperture value F is specified as aperture value F = focal length / effective aperture (diameter) ... (Equation 3) based on the focal length of the lens and the effective aperture (diameter). Therefore, with the same lens, the smaller the aperture value, the brighter the captured image. For example, the aperture value of 2.0 is four times brighter than the aperture value of 4.0.

Based on the above, in Survey 2, the ISO sensitivity was set to 100, 200, under the conditions of (1) shutter speed = 1/125 second, (2) aperture value = 2.2, and (3) image stabilization = OFF setting. The brightness was changed to 400, 800, 1600, 3200, 4000, and 8000, and the suitability of the brightness of the images taken of the train guide display board and the destination display at multiple stations with different installation environments was judged.

In FIG. 2 (a), for the underground station 1 having an illuminance of 440 to 920 lux, a large number of photographs taken of the train guide display board and the destination indicator with different ISO sensitivities are evaluated, and the% notation of the appropriate ratio is shown. It is shown.

It was found that when the ISO sensitivity exceeds 1600, it is too bright and more than half are evaluated as failures, while when the ISO sensitivity is lowered to 100, it is too dark and fails by about 30%. Therefore, at this underground station 1, ISO sensitivity of 200 to 400 is considered appropriate.

FIG. 2B is a survey result of the ground station 2 having an illuminance of 590 to 1130 lux. When the ISO sensitivity exceeds 800, more than half are evaluated as failures, and it was found that the lower the ISO sensitivity, the higher the shooting success rate. The failure of ISO sensitivities 200 and 400 is due to the shooting of the deadhead train and can be ignored. Therefore, at the ground station 2, ISO sensitivity of 100 to 400 is considered appropriate.

FIG. 2C shows the survey results for the ground station 3 having an illuminance of 140 to 1900 lux. At this ground station 3, ISO sensitivity of 200 to 400 is evaluated to be appropriate.

FIG. 3A is a survey result of the ground station 4 having an illuminance of 600 to 1060 lux. According to the results of this survey, it is evaluated that shooting at an ISO sensitivity of 100 to 200 is appropriate at the ground station 4.

FIG. 3B is a survey result of the ground station 5 having an illuminance of 280 to 5500 lux. At the ground station 5, it is confirmed that the lower the ISO sensitivity, the higher the shooting success rate. If the ISO sensitivity exceeds 800, more than half of the failures will occur, so it is evaluated that shooting with an ISO sensitivity of 100 to 200 is appropriate.

FIG. 3C shows the survey results for the ground station 6 having an illuminance of 50,000 to 200,000 lux. The ground station 6 is exposed to direct sunlight, and it is difficult to open its eyes, and even if the ISO sensitivity is lowered to 80, it will fail with a high probability. If you set the aperture value to 4 or more, you can shoot even in direct sunlight, but considering the suitable shutter speed (1/40 to 1/125 seconds), the shutter speed cannot be faster than 1/125 seconds. Under the conditions, it is not possible to set it so dark on a mobile terminal.

As described above, under the conditions of aperture value = 2.2 and shutter speed = 1/125 second, (1) ISO sensitivity of 200 to 400 is appropriate for underground station 1, and (2) ISO for ground station 2. Sensitivity 100-400 is appropriate, (3) ISO sensitivity 200-400 is appropriate for ground station 3, (4) ISO sensitivity 100-200 is appropriate for ground station 4, (5) ) It became clear that ISO sensitivity 100 to 200 is appropriate at the ground station 5, and (6) it is difficult to shoot with a mobile terminal at the ground station 6. Therefore, the present inventor has concluded that an ISO sensitivity of 200 is desirable, provided that direct sunlight is avoided at the time of photography.

However, the above conclusion is only for the case where the aperture value is 2.2, and the aperture value differs depending on the mobile terminal. Moreover, this aperture value is generally not freely changeable. Therefore, based on this limitation, it is necessary to specify a configuration that operates under appropriate conditions on any mobile terminal.

Therefore, as a result of further studies, the present inventor can find one correct setting value because the three parameters (1) shutter speed, (2) ISO sensitivity, and (3) aperture value affect each other. For example, I came up with the idea that any value can be changed and supplemented with other values. That is, (1) the shutter speed is the time (exposure time) when the shutter is opened and the image pickup element is exposed to the light passing through the lens. Therefore, the slower the shutter speed S, the brighter the captured image, for example. It can be said that the shutter speed S = 1/500 second is twice as bright as S = 1/1000 second.

(2) The ISO sensitivity is a parameter of how weak the light can be handled by the image sensor. The larger the ISO sensitivity, the brighter the captured image. For example, the ISO sensitivity 1000 is twice as bright as the ISO sensitivity 500. It can be said that.

(3) From the aperture value F = focal length / effective aperture (diameter) ... (Equation 3), the smaller the value (effective aperture) of the aperture value, the brighter the captured image, for example, the aperture value. It can be said that 2.0 is four times brighter than the aperture value of 4.0.

According to these (1) shutter speed S, (2) ISO sensitivity I, and (3) aperture value F, the shutter speed S, ISO sensitivity I, and so on so that the following (Equation 4) holds. It is considered that a constant brightness can be maintained by adjusting each value of the aperture value F.
S × I / F ² = constant ... (Equation 4)
Then, from the optimum values (S = 1/125, I = 200, F = 2.2) of the previous survey results, the constant value of (Equation 4) is approximately 0.33 (≈1/125) under the optimum conditions. × 200 / 2.2 ² ) is specified.

That is, if the values of (1) shutter speed S, (2) ISO sensitivity I, and (3) aperture value F are set so that the calculated value of (Equation 4) is 0.33, which mobile phone is used. Even with the terminal, it will be possible to take pictures of the destination display and train information display board with the correct brightness.

Generally, in a mobile terminal, the aperture value F of the camera lens is a fixed value. In addition, as a result of the investigation, since shooting darker to some extent is required, the shutter speed is fixed at 1/125 second, which is the fastest in the optimum range (1/40 to 1/125 second), and the aperture value F of the mobile terminal is set. Therefore, it was concluded that it is desirable to calculate the ISO sensitivity I based on (Equation 4) and adjust it to the ISO sensitivity.

That is, in the embodiment described below, the ISO sensitivity I is set to I = 0.33 × F ² × 125 ... (Equation 5) based on the aperture value F of the camera lens of the mobile terminal to be used. I decided to take a picture of the destination display and the train information display board.

However, if (Equation 5) is used as it is, I = 165 when F = 2.0, I = 199.65 when F = 2.2, and so on. Not an exact match. Therefore, a determination table CNV is prepared in advance, and for example, 160 is adopted when F = 2.0, and ISO sensitivity 200 is adopted when F = 2.2.

<Ride judgment system>
FIG. 4 is a block diagram showing a boarding determination system for operating the built-in camera of the mobile terminal under the above-mentioned optimum shooting conditions. Here, as an example, a case where a traveler (system user) who is unfamiliar with geography determines whether or not the selection of the train selected for boarding to go to the destination station is correct will be described.

Generally, a train information display board installed on a station platform displays the departure time from the station platform, the type of train scheduled to depart, and the destination station (terminal station or transfer station). In addition, the type of train and the destination station (terminal station or transfer station) are generally displayed on the destination indicator on the side of the train.

Here, the type is information that classifies all trains into normal / semi-express / express / rapid express / limited express, etc., and specifies the superiority / inferiority relationship of arrival time at the destination station, the difference in the stop station on the way, and the like. ing.

As shown in FIG. 4, the boarding determination system of the embodiment includes a mobile terminal 1 with a camera, a train information providing server 2 that provides train information necessary for arriving at a destination station, and predetermined text information and image information. Is received from the mobile terminal 1, and based on the information received from the image recognition server 4, the management server 3 that returns the appropriateness of the train selection by the user to the mobile terminal 1 and the image information transferred from the management server 3 are registered. It includes an image recognition server 4 that compares completed image information and returns necessary determination information.

The train information providing server 2, the management server 3, and the image recognition server 4 are HTTP servers connected to the Internet line NET, respectively. Further, the mobile terminal 1 has a built-in camera and is equipped with a browser for acquiring a WEB page on the Internet line NET via the mobile communication line MOB.

Further, a boarding determination application (application name is JUDG) that executes the process described below in cooperation with the built-in browser is pre-installed on the mobile terminal 1 together with the determination table CNV (Table 1).

This boarding determination application JUDG is configured to be activated when the user taps a predetermined button described on the Web page. However, the configuration is not limited to this, and the boarding determination application itself may exhibit a browser function, display a Web page in the application, and wait for the user's operation.

Regardless of which configuration is adopted, the shutter speed S of the built-in camera is set to 1/125 second by the initial setting process executed when the boarding determination application JUDG is started, and the aperture value F of the mobile terminal 1 is set. The corresponding ISO sensitivity I is set to the optimum value based on the determination table CNV shown in Table 1.

In addition, the train information providing server 2 distributes a WEB page (see FIG. 5) containing train information such as departure time for any departure station and destination station based on instructions from the mobile terminal 1 (WEB client). It is configured to do.

As shown in Fig. 5, this Web page shows the train type (express in the figure) that moves from the departure station (○○) to the arrival station (XX), as well as the departure time and arrival time, along with the fare and other information. Has been done. In addition, a "destination determination button" is arranged on this Web page, and when the user taps this button, the installed boarding determination application JUDG is activated.

Specifically, for example, the URL scheme is used for this Web page, and the application name (JUDG) is specified in the scheme name part of the anchor tag instead of http that specifies the communication protocol. App JUDG can be executed. That is, an anchor tag such as <a href = "JUDG: //"> destination determination </a> is described in this Web page.

Then, when the boarding determination application JUDG is activated in response to the user's tap operation, the built-in camera is activated and the shutter speed S is set to 1/125 second, which corresponds to the aperture value F of the mobile terminal 1. The optimum ISO sensitivity I is set.

Next, a guidance screen will appear stating, "For the train you are about to board, please take a picture of the destination indicator on the side of the train or the train information display board." Therefore, the user has to release the shutter after pointing the built-in camera at the destination display on the side of the train or the train information display board on the station platform.

In addition, on this Web page, the destination station, train type (normal / semi-express / express / rapid express / limited express), departure station, and departure time are described as hidden data in the input tag of the hidden attribute. There is. Furthermore, this Web page describes a JavaScript program that functions based on shutter operations, etc., and the destination station, train type, departure station, and departure time are, for example, POST data together with the shooting data taken by the built-in camera. , Is sent to the management server 3.

The specific method is appropriate, but in HTML5 (Hyper Text Markup Language version 5), the function of the built-in camera of the mobile terminal 1 can be used directly from JavaScript, so the image at the moment when the shutter is released is appropriate for the Web page. It may be pasted on a tag (for example, a canvas tag) and POSTed to the management server 3 together with other information.

The configuration in which the built-in browser of the mobile terminal 1 accesses the train information providing server 2 has been described above. In the case of adopting such a configuration, most of the processing can be executed by JavaScript, so the boarding judgment application JUDG should only execute the processing to optimally set the shooting conditions of the built-in camera and guide the subsequent operations. It will be enough.

However, the train information providing server 2 needs to prepare a "destination determination button" which is a Web page in which the above-mentioned JavaScript program is described and is composed of an anchor tag in which the application name JUDG is described. Therefore, in order to eliminate this complexity, it is preferable to provide the boarding determination application itself with a browser function so that the boarding determination application JUDG directly accesses the train information providing server 2.

When adopting the latter configuration like this, the train information providing server 2 does not need to prepare a special Web page, and arranges a "destination determination button" in which a normal anchor tag whose scheme name is http is embedded. Then, for example, a Web page in which the destination station, the train type, the departure station, and the departure time are described by the hidden attribute will be delivered. In this case, the boarding determination application JUDG executes all of the above-mentioned series of processes in response to the tap operation of the "destination determination button" by the user as well as the initial setting process.

Subsequently, the management server 3 will be described. As a general rule, the management server 3 plans to communicate with the mobile terminal 1 equipped with the boarding determination application JUDG, and the information received from the mobile terminal 1 (destination station / train type / departure station / departure time / shooting data). ) Is temporarily stored in an appropriate work area WORK, and the shooting data is transferred to the image recognition server 4.

Corresponding to the configuration of the management server 3, the image recognition server 4 is configured to start the image recognition program RECOG in response to the reception of shooting data. This image recognition program RECOG is configured to have a pouring algorithm that pours out a plurality of features from the transferred shooting data.

Further, the image recognition server 4 receives in advance model shooting data (model shooting data) for all destination indicators and all guidance display boards that may be transferred by this system. Then, the image recognition program RECOG has completed the dispensing process of the feature amount from all the model shooting data, and the dispensed feature amount is stored in the feature parameter table TBL2 together with the unique image ID. ..

That is, the train type and the destination station are always displayed on the destination display arranged on the side of the train, and the image recognition program RECOG of the image recognition server 4 preliminarily displays all combinations of the train type and the destination station. From the acquired model shooting data, the required feature quantity has been poured out.

This point is the same for the guide display board installed on the station platform, and the image recognition server 4 (image recognition program RECOG) has all the train types and destination stations that may be displayed on the guide display board. Regarding the combination of, the required feature amount has been poured out from the model shooting data acquired in advance.

The image taken by the camera usually reflects not only the train type and destination station, but also the outline image of the destination display and other text information. However, the image recognition program RECOG of the image recognition server 4 sequentially orders a plurality of feature quantities extracted from the transferred imaging data to be determined and feature quantities registered in the feature parameter TBL2 for each image ID. In contrast to the above, the image ID in which the number of matching feature amounts is the maximum and the correct answer rate (score value) for selecting this image ID are specified and returned to the management server 3. Therefore, information other than the target information (train type / destination station) is effectively ignored. The correct answer rate is calculated based on the number of matching feature quantities and the like.

On the other hand, the management server 3 is provided with a reference table TBL1 for each departure station corresponding to the feature parameter table TBL2 of the image recognition server 4. In this reference table TBL1, the destination station and the character information of the train type are specified and stored in the same image ID as the feature parameter table TBL1 for all the destination indicators and all the guide display boards. Also. In this reference table TBL1, one or a plurality of departure times of the train are stored for each image ID.
<Operation of each part of the boarding judgment system>

Since the configuration of the boarding determination system has been described above with reference to FIG. 4, a series of operations for determining whether or not the selection of the train selected by the system user is correct will be described.

In the following explanation, (1) the train information providing server 2 arranges a "destination determination button" in which a normal anchor tag whose scheme name is http is embedded, and the destination station / train type / departure station / departure time. The case where the Web page described with the hidden attribute is distributed, and (2) the boarding determination application JUDG exerts the browser function as its own internal processing to display and operate the Web page will be described. Further, in the following description, it is assumed that the system user selects the train to be boarded at the station platform.
[Steps 1 to 3 of FIG. 4 (operation of mobile terminal 1)]

When the user activates the boarding judgment application JUDG, the activated boarding judgment application JUDG exerts the browser function after setting the built-in camera to the optimum shooting conditions. That is, since any HTTP server can be accessed after the initial setting process, the system user can access the train information providing server 2 and access the Web regarding the departure station where he / she is and the destination station he / she wants to go to. Acquire the page and display the train information screen as shown in FIG. 5 in the application.

Then, after confirming the departure time, etc., when the system user taps the "destination determination button", the built-in camera is activated and "the destination indicator on the side of the train or the train information display board on the station platform is displayed. Please take a picture. "Is displayed on the guidance screen. In this case, the display screen of the built-in camera displays a horizontally long frame having an aspect ratio corresponding to the destination display and the train guide display board.

Therefore, the user points the built-in camera at the destination display or the train guide display board, and presses the shooting shutter at the timing when the vehicle type and the destination station are captured in the horizontally long frame.

Then, in response to this shutter operation, the boarding judgment application JUDG records the shooting data taken by the built-in camera together with the destination station / train type / departure station / departure time described in the hidden attribute on the Web page, as well as the management server. POS is sent to 3.
[Fourth step in FIG. 4 (operation of management server 3)]

As described above, the management server 3 that has received the POST data from the mobile terminal 1 temporarily stores the destination station / train type / departure station / departure time in the work area WORK. Further, the management server 3 transfers the shooting data to the image recognition server 4 and waits for a reply from the image recognition server 4. The reference table TBL1 corresponding to the departure station is specified based on the departure station received from the mobile terminal 1.
[Fifth step in FIG. 4 (operation of the image recognition server 4)]

The image recognition server 4 dispenses a feature amount for the shooting data received from the management server 3, and sequentially compares the dispensed feature amount with the feature amount described in the feature parameter table TBL2, and the matching feature amount is obtained. The maximum number of image IDs and the correct answer rate (score value) for selecting this image ID are specified and returned to the management server 3.

Here, the correct answer rate indicates the degree of similarity between the shooting data transferred to the image recognition server 4 and the model shooting data specified by the image ID. Therefore, if the captured image is unclear and the feature amount cannot be poured out, or if the model captured data similar to the transferred captured data is not detected, the correct answer rate becomes 0.

The image ID and the correct answer rate returned to the management server 3 do not necessarily have to be one set, and if the correct answer rate is at a predetermined level or higher, the plurality of sets of image IDs and the correct answer rate are returned to the management server 3. Will be done.
[Sixth step in FIG. 4 (operation of management server 3)]

When the management server 3 in the standby state receives the image ID and the correct answer rate from the image recognition server 4, the management server 3 refers to the reference table TBL1 corresponding to the departure station, and based on the image ID, the destination station and the train. Specify the type.

Then, it is determined whether or not the destination station poured out from the reference table TBL1 and the train type match the destination station and the train vehicle type temporarily stored in the work area WORK. In this judgment result, G judgment (a): an image ID with a high correct answer rate is received and it matches the destination and type in temporary storage, and G judgment (b): an image ID with a high correct answer rate is received. Nevertheless, when it does not match the destination or type in temporary storage, when the judgment result of NG judgment (c): correct answer rate = 0 is received, or when only the image ID with low correct answer rate can be received. It is classified into.

Here, in the case of the G determination (a) or the G determination (b), the management server 3 returns the recognition result to the mobile terminal 1 after confirming the validity of the departure time. The content of the reply will be, for example, a web page displaying "Yes, that train is bound for the limited express XX station ...." or "This is not the train. Please look for another train." In this Web page, the shooting data taken by the user and the model shooting data specified based on the image ID received from the management server 3 are displayed in comparison on the mobile terminal 1.

However, in the case of the NG determination (c), after executing the image restoration processing and the reflection removal processing described below in this order, the process returns to the fourth step again, and the image recognition server 4 performs correction shooting. Send the data.

Then, it is determined whether or not the image ID returned from the image recognition server 4 matches the information temporarily stored in the management server 3 (fifth step), and whether it is a G determination (a) or a G determination (b). ), The corresponding answer is returned to the mobile terminal 1 (sixth step).

On the other hand, regarding the corrected shooting data, in the case of NG judgment (c) again, the answer "The image is unclear and cannot be judged" is returned to the mobile terminal 1 (sixth step).
<Image restoration processing (A) and reflection removal processing (B)>

Subsequently, the operation after the management server 3 receives the first NG determination (c) from the image recognition server 4 will be described. The management server 3 that has received the NG determination (c) executes the image restoration process (A) and the reflection removal process (B) described below in this order to generate corrected shooting data.

First, in the image restoration process (A), when the original character information is latent due to reflected light from a fluorescent lamp or the like, the character image information of the defective portion is complemented based on the surrounding image information. means.
[Image restoration (A) 1st processing]

In the image restoration process (A), the management server 3 grayscales the captured image data in order to delete the color information from the captured image data temporarily stored in the work area WORK.

The specific method of grayscale is appropriately selected, but in this example, the Decolor function of the Open CV (Open Source Computer Vision Library) library for computer vision is used in the operating environment of the programming language Python. The following explanations are all explanations in the operating environment of the programming language Python.

When using the Decolor function, in Python, the format is Grayscale.color_boost = cv.deColor (src). Here, when the captured image is composed of N pixels, the first argument src of the cv.deColor function means an array Photo (R, G, B) with 3 * N elements, and this array Is stored with RGB data having a length of 1 byte each.

On the other hand, the return value is an array GS (i) having the number of elements N, and the grayscale information (Gray Scale value) of each pixel is stored in a byte length. The larger the value, the brighter the value, the smaller the value, the darker the value, 0 means pure black, and 255 means pure white.
[Image restoration (A) Second process]

Next, with respect to the Gray Scale value (hereinafter abbreviated as GS value) obtained in the first image restoration process, pixels brighter than a predetermined threshold value TH1 are extracted. Specifically, the OpenCV inRange function is used, but in Python, the format is dst = cv.inRange (src, lowerb, upperb).

Here, the first argument src of the inRange function is the result array GS (i) of the number of elements N calculated in the first image restoration process, and the GS value is stored in 1 byte length for each pixel. There is. Further, the second argument lowerb and the third argument upperb are scalar values, respectively, and here, lowerb = 0 and upperb = TH1.

On the other hand, the return value dst indicates a result array having the number of elements N, and 255 or 0 is stored in the result array DST (i). Specifically, the bright pixel (GS value> TH1) is 0, and the other pixel with 0 ≦ GS value ≦ TH1 is 255. As a result, the array DST (i) indicates the region where the valid information is missing due to the reflected light by 0. Usually, this defective region is often the light emission of a fluorescent lamp, and in such a case, the defective region is formed as a straight band having a predetermined width.
[Image restoration (A) Third process]

Subsequently, the image information of the defective portion is complemented from the image information around it. Specifically, the inpaint function of Open CV is used to complement the pixel information of the missing area (8 bits each for RGB) from the surrounding pixel information. In Python, for example, the format is st = cv.inpaint (src, Mask, Radius, flags).

Here, the first argument src is an array Photo (R, G, B) having 3 * N elements for storing captured image data of the number of captured pixels N, and RGB data is stored with a length of 1 byte each. ing. In this embodiment, the second argument Mask is a mask array DST (i) calculated in the second image restoration process, and is an array having N elements. As described above, in the element of the mask sequence DST (i), 0 indicates a portion (defective region) to be repaired.

On the other hand, the third argument Radius is the radius of the circular region in the vicinity including the portion to be repaired, and is appropriately determined based on the width of the defective region. In addition, the fourth argument flags is a flag value that specifies the completion algorithm, and it is possible to specify whether the method is based on Navier-Stokes or Alexandru Telea. In this example, the method by Alexandru Telea was specified.

The result array dst is an array Repair (R, G, B) of the same size and type as the input array src, and the RGB data after image restoration is saved. As a result of the above processing, the character information of the reflected light portion such as the fluorescent lamp is restored to some extent. FIG. 6A shows a color image in which the defective portion is complemented, and the emission of each of the inclined line of the reflected light displayed in the upper left part and the white area displayed in the lower right part is significantly reduced. Has been done.
[Reflection removal (B) first treatment]

In the subsequent reflection removal processing, H, S, and V of the HSV color space are first calculated from each of the 8-bit RGB data for each pixel of the image data (hereinafter referred to as the repaired image data) that has completed the image restoration processing (A). do. Here, the HSV color space is a method of expressing the color of a certain pixel by three combinations of "hue", "saturation", and "value". The method for calculating the H, S, and V values is not particularly limited, but here, the cvtColor function of Open CV is used, and in Python, for example, the format is dst = cv.cvtColor (src, code).

When the repair image data is composed of N pixels, the input array SRC specified by the first argument src of the cv.cvtColor function in the above Python format is the result array Repair (R) of the image repair process (A). , G, B).

As a result, RGB data for each pixel is stored in the array Repair (R, G, B) in the order of B → G → R. The second argument code indicates that the process to be executed is conversion to the HSV color space. The return value dst is a result array with 3 * N elements, and the H value, S value, and V value are stored pixel by pixel in the result array DST (i) by the processing of the cvtColor function. Become.

Therefore, in this embodiment, next, based on the result array DST (i) of the first reflection removal process, a V value array Value (i) having the number of elements N storing the V values of N pixels is generated. ..
[Reflection removal (B) second treatment]

Next, for the V value array Value (i) for the N lightness values generated in the first reflection removal process, a histogram of the appearance frequency for all pixels is created, and the V value indicating the lightness (Value) is appropriate. The threshold TH2 is specified.

FIG. 7 shows a captured image and a histogram of the appearance frequency of the V value. Although the entire captured image is displayed here, in actual operation, the outside of the horizontally long frame is all blackened.

In any case, in the case of the image to be subjected to the reflection removal processing, as shown in the figure, the appearance frequency of V value = 0 is overwhelming, and the appearance frequency of other V values is extremely low, but 0 is set. When the V value exceeds (> 0), the appearance frequency peaks at any V value. Therefore, in this embodiment, the V value above the V value indicating the peak value, for example, the V value at the 70% position of the peak value is selected as the threshold value TH2.
[Reflection removal (B) 3rd process]

Next, in the reflection removal third process, pixels having a brightness higher than the threshold value TH2 specified in the reflection removal second process are extracted. For example, we use the OpenCV inRange function, but in Python it has the format dst = cv.inRange (src, lowerb, upperb).

Here, the first argument src is a V value array Value (i) for the V value obtained in the second reflection removal process. Further, the second argument lowerb and the third argument upperb are scalar values related to HSV, respectively, and here, lowerb = TH2 and upperb = 255.

On the other hand, the return value dst indicates the result array Mask (i) having the number of elements N, and 255 or 0 is stored in the result array Mask (i). Specifically, a pixel with high brightness (TH2 ≦ V value ≦ 255) is white (255), and a normal pixel with V value <TH2 is black (0). This process is nothing but a process of specifying whether or not to adopt each pixel for N pixels in the result array Mask (i).
[Reflection removal (B) 4th process]

Subsequently, RGB image data is extracted from the result array Repair (R, G, B) of the image repair process (A) for the pixels determined to be high-brightness pixels in the reflection removal third process. Here, for example, the OpenCV bitwise_and function is used, but in Python, the format is dst = cv.bitwise_and (src1, src2, dst, mask).

Here, the first argument src1 and the second argument src2 are the arrays used in the first reflection removal process, that is, the restoration image array Repair (R, G, B) in which the restoration image RGB data by the image restoration (A) is saved. It becomes. The third argument dst means an output array and indicates an array Take (R, G, B) having 3 * N elements.

On the other hand, the fourth argument mask means a mask array. Specifically, the result array MASK (i) of the number of elements N calculated in the reflection removal third process is used. In this bitwise_and function, the logical operation of dst (i) = src1 (i) Λsrc2 (i) is executed only when MASK (i) ≠ 0. Note that Λ means an AND operation.

As described above, in this embodiment, each element of the mask array MASK (i) is 255 for pixels with high brightness (TH2 ≦ V value ≦ 255) and 0 for pixels that are not. Therefore, the following processing is executed only for the high-brightness pixels in which the element of the mask array MASK (i) is 255, and the repair image array Repair (R, G, B) that stores the repair image RGB data is It will be copied to the output array Take (R, G, B). The specific processing content is Reject (R, G, B) = Repair (R, G, B) Λ Repair (R, G, B).

On the other hand, the above AND operation is not executed for the pixel of normal brightness in which the element of the mask array MASK (i) is (0). That is, the elements of the result array Take (R, G, B) remain in the initial state (0, 0, 0), so after all, high brightness is important for the result array Take (R, G, B). Only the part will be saved.

On the train information display board for railways, the "departure time", "train model", and "destination station name" are displayed brighter than other parts based on their importance. Then, by cutting out only the high-brightness portion by the reflection removing process, the reflection is alleviated and only the necessary portion is extracted. That is, an image in which the image representing what is not known due to the reflection of the reflected light is firmly converted into an image displaying the destination station and the type. FIG. 6B shows the processing result of the reflection removal processing, and it is confirmed that the character information of the important portion is made clearer.

In addition, in FIG. 6B, in order to contrast the effect of the reflection removal processing, the train guide display board is intentionally photographed over three lines (three destinations), but in actual operation, the camera is used. By displaying a horizontally long frame frame, only a single destination station can be acquired.

As described above, by executing the image restoration process (A) and the reflection removal process (B), the character information is restored in the result array Take (R, G, B) of the reflection removal process third process, and the character information is restored. , The corrected shooting data in which the reflected light is relaxed will be saved. Therefore, the management server 3 transmits the corrected shooting data to the image recognition server 4 and waits for the determination result (fourth step in FIG. 4).

Then, the management server 3 determines whether or not the image ID returned from the image recognition server 4 matches the information temporarily stored in the management server 3, and determines whether it is a G determination (a) or a G determination (b). ), The corresponding answer is returned to the mobile terminal 1.

On the other hand, regarding the corrected shooting data, in the case of the NG judgment (c), the answer "The image is unclear and cannot be judged" is returned to the mobile terminal 1.

The boarding determination system shown in FIG. 4 has been described in detail above, but the description so far is based on the premise that the boarding determination application JUDG is installed on the mobile terminal 1. However, in the mobile terminal 1 in which the boarding determination application JUDG is not installed, it is not possible to optimally set the built-in camera by the JavaScript program in HTML5.

Therefore, in preparation for such a case, it is preferable to operate the built-in camera in the continuous shooting mode or the moving image mode for the mobile terminal in which the boarding determination application JUDG is not installed.

The specific method is appropriate, but for example, in response to the tap operation of the "destination determination button", the "boarding determination application is not used" as a cautionary note on the Web page distributed by the train information providing server 2. Please take a picture of the destination display / train information display board in continuous shooting mode or movie mode. "

Next, a case where the system user operates the camera corresponding to this cautionary note will be described. When the management server 3 continuously receives the captured image data or receives the moving image frame data, all of them are temporarily stored in the work area WORK, and an appropriate number of continuous shot images or a moving image frame is used. A plurality of images are superimposed and transferred to the image recognition server 4 as captured image data (process in the fourth step).

The process of the fourth step may be continuously executed in the process of the third step in which the moving image or the continuous shot image is transmitted from the mobile terminal 1, or the image recognition server 4 may perform an NG determination (c). It may be executed after receiving, that is, following the fifth step.

It is not unlikely that the image recognition server 4 will receive a G-judgment (a) or a G-judgment (b) from one of the continuous shot images or one frame image cut out from the moving image. , The latter procedure should be used. That is, only after receiving the NG determination (c) from the image recognition server 4, a plurality of image data should be superposed and retransmitted to the image recognition server 4 as captured image data.

Although the embodiments of the present invention have been described in detail above, the specific description contents are not limited to the present invention and can be appropriately changed. In particular, there is no need to assume the operating environment of the programming language Python. However, using Open CV functions is effective in simplifying the program.

Further, in the above embodiment, the configuration for operating the mobile terminal on which the boarding determination application JUDG is not installed is described in the continuous shooting mode or the video mode, but the mobile terminal on which the boarding determination application JUDG is installed is also described. A similar operation mode may be provided.

Further, in the above description, although the railway vehicle has been described, almost the same configuration can be adopted for the determination process at the time of boarding a bus, an airplane, a ship, a shared taxi, or the like. Further, although the shutter speed S and other camera setting values have been specifically described, only an example is described, and the present invention is not particularly limited.

By the way, in the above embodiment, the case where the system user accesses the train information providing server 2 at the station platform has been described, but the present invention is not limited thereto. For example, the first process of accessing the train information providing server 2 and transmitting necessary information (destination station / train type / departure station / departure time) to the management server 3 and the destination indicator on the side of the train at the station platform. Alternatively, a significant time difference may be provided in the second process of photographing the train information display board of the station platform and transmitting the photographed image data to the management server 3.

FIG. 8 is a drawing illustrating such an embodiment. In FIG. 8, the first process of accessing the train information providing server 2 and transmitting necessary information (destination station / train type / departure station / departure time) to the management server 3 is as the first step to the third step. It is shown. This process is executed at a hotel or the like, for example, the day before the scheduled boarding.

On the other hand, the management server 3 stores the necessary information (destination station / train type / departure station / departure time) transmitted in the first process in an appropriate work area WORK together with a unique user ID. In addition, in order to properly manage the work area WORK, the reception date and time are also saved.

Then, the management server 3 returns the above-mentioned user ID to the mobile terminal 1 together with a Web page displaying "required information has been received" (fourth step in FIG. 8). Then, this user ID is stored in the mobile terminal 1 as, for example, a cookie DATA.

Next, after a predetermined time has elapsed, when the system user activates the boarding judgment application JUDG of the mobile terminal at the station platform, the activated boarding judgment application JUDG sets the built-in camera to the optimum shooting conditions and then performs the browser function. Activate the built-in camera and display a guide screen saying "Please take a picture of the destination indicator on the side of the train or the train guide display board on the station platform." Along with the horizontally long frame.

Therefore, the user points the built-in camera at the destination display or the train guide display board, and presses the shooting shutter at the timing when the vehicle type and the destination station are captured in the horizontally long frame.

Then, in response to this shutter operation, the boarding determination application JUDG transmits the shooting data taken by the built-in camera to the management server 3 together with the user ID (fifth step in FIG. 8).

Subsequent processing is the same as in the case of the configuration of FIG. 4, and the management server 3 transfers the shooting data to the image recognition server 4 and waits for a reply from the image recognition server 4 (sixth step of FIG. 8). ). The reference table TBL1 is specified based on the departure station stored in the work area WORK.

On the other hand, the image recognition server 4 dispenses a feature amount for the shooting data received from the management server 3, and sequentially compares the dispensed feature amount with the feature amount described in the feature parameter table TBL2 to match the feature amount. The image ID having the maximum number of pieces and the correct answer rate (score value) for selecting this image ID are specified and returned to the management server 3 (7th step in FIG. 8).

Then, the management server 3 in the standby state returns the determination result to the mobile terminal 1 based on the image ID and the correct answer rate received from the image recognition server 4 (8th step in FIG. 8), or captures the shooting data. It is corrected and retransmitted to the image recognition server (sixth step in FIG. 8). The data stored in the work area WORK is deleted when it is no longer necessary or when the storage period from the reception date has expired.

When this configuration is adopted, the system user can select a train in a state where there is plenty of time, such as the night before, and there is an advantage that it is sufficient to take a picture at the station platform on that day. In the mobile terminal 1 in which the boarding determination application JUDG is not installed, the built-in camera functions in the continuous shooting mode or the moving image mode as in the case of FIG.

Although the two examples have been described above, appropriate changes can be made as long as they are included in the scope of the present invention. In particular, in each of the above embodiments, the departure time is a problem, but from the viewpoint of simplification, the management server 3 is configured to return the recognition result to the mobile terminal 1 without making the departure time a problem. It is also suitable to take.

1 Mobile terminal 2 Information provision server 3 Management server

Claims (10)

An information providing server that provides traffic information of transportation, a mobile terminal configured to be able to acquire necessary traffic information from the information providing server, a photographed image related to traffic information from the mobile terminal, and corresponding characters. The management server is configured to be able to retrieve information, and is configured to have
The management server has a shooting image taken under shooting conditions of a predetermined shutter speed, an aperture value specified by the mobile terminal, and a predetermined ISO sensitivity specified corresponding to the shutter speed and the aperture value. In response to the above, a boarding determination system configured to return a determination result as to whether or not the photographed image is appropriate in relation to the character information to the mobile terminal. An information providing server that provides traffic information of transportation, a mobile terminal configured to be able to acquire necessary traffic information from the information providing server, a photographed image related to traffic information from the mobile terminal, and corresponding characters. The management server is configured to be able to retrieve information, and is configured to have
The management server receives a plurality of shot images shot continuously or shot images of a moving image frame shot as a moving image, and determines whether or not the shot image is appropriate in relation to the character information. , A boarding determination system configured to reply to the mobile terminal. It is configured to include a feature parameter table that stores one or more feature quantities corresponding to ID information that can identify the character information.
The boarding determination system according to claim 1 or 2, wherein the feature amount is dispensed in advance for each captured image that may be received from the mobile terminal and stored in the feature parameter table. The determination of whether or not the captured image acquired by the management server is appropriate in relation to the character information is determined.
The boarding determination system according to claim 3, wherein the feature amount poured out from the captured image and the feature amount stored in the feature parameter table are compared and executed. The boarding determination system according to claim 3 or 4, wherein the feature amount is included in a photographed image and is dispensed with respect to a destination and / or type of transportation. The boarding determination system according to claim 5, wherein the ID information is attached to each combination corresponding to a plurality of combinations relating to the destination of the transportation and the type of the transportation. A setting program for setting a predetermined shutter speed, an aperture value specified by the mobile terminal, and shooting conditions of a predetermined ISO sensitivity specified corresponding to the shutter speed and the aperture value in the mobile terminal. The boarding determination system according to claim 1, wherein is installed. The mobile terminal according to claim 1 is set with a predetermined shutter speed, an aperture value specified by the mobile terminal, and shooting conditions of a predetermined ISO sensitivity specified corresponding to the shutter speed and the aperture value. Setting program for . A mobile terminal on which the setting program according to claim 8 is installed. A shot image taken under a predetermined shutter speed, an aperture value specified by a mobile terminal, and a shooting condition of a predetermined ISO sensitivity specified corresponding to the shutter speed and the aperture value corresponds to this. A management server configured to receive text information from the mobile terminal and return a determination result of whether or not the captured image is appropriate in relation to the text information to the mobile terminal.

Images