JP6664293B2 - Device direction estimation method, image recognition device, and control program therefor - Google Patents

Description

  The present invention relates to a device direction estimating method used for recognizing an image captured by a device such as a mobile terminal, an image recognition device, and a control program therefor.

  In recent years, when an image captured by a portable terminal having an imaging function is transmitted to an image recognition server on the Web, the captured image is recognized based on a dictionary image of a database in the server, and information corresponding to the recognition result is obtained. Has been considered as an image recognition service (hospitality service) that can be obtained by being returned to the terminal (for example, see Non-Patent Document 1).

NTT Technical Journal 2016 Vol.28 No.5 pp57-59 "Starting technical verification for realizing hospitality service with simple portable terminal equipped with transmissive display"

  When an object (recognition target) is photographed by a portable terminal of each user who uses the image recognition service, the direction of the terminal, for example, whether the photographer holds the camera vertically or horizontally, is various and constant depending on the user. Therefore, the direction of the image sent from each user terminal to the image recognition server is not fixed.

  For this reason, in the image recognition server (image recognition system), the match / mismatch of the direction between the image captured by the user (the image to be recognized) and the dictionary image changes, and the recognition success rate deteriorates.

  The portable terminal has a built-in direction detection function using an acceleration sensor or the like, and information about the direction of the terminal is added to the captured image to be transmitted to the image recognition server, so that the direction of the image matches the direction of the dictionary image. It is possible to recognize.

  However, in order to realize various services at low cost, it is necessary to reduce the functions of the terminal, and a terminal having the built-in direction detection function cannot be used for such a terminal.

  The present invention has been made in view of such a problem, and even if a terminal that captures an object to be recognized is a terminal that does not have a built-in direction detection function, the recognition success rate of an image captured by the terminal is improved. It is an object of the present invention to provide a device direction estimating method, an image recognizing device, and a control program for the device direction estimating method.

The device direction estimating method according to the present invention obtains an image captured by a terminal, requests the image recognizing unit that recognizes the image using the obtained image as a recognition target image, and requests the image recognition. When the recognition result of the recognition target image acquired from the image recognition unit in response to the recognition is a failure, rotate the direction of the recognition target image and request the image recognition unit to perform image recognition again, the If the recognition result of the recognition target image obtained from the image recognition unit in response to the re-request for image recognition is successful recognition, estimate the direction of the terminal based on the direction of the recognition target image , If the recognition result of the recognition target image obtained from the image recognition unit in response to the request for image recognition is successful, or the image obtained from the image recognition unit in response to the re-request for image recognition recognition When the recognition result of the elephant image is a recognition success, a terminal screen including information representing the content of the recognized image is generated based on the estimated direction of the corresponding terminal, and the generated terminal screen is generated. The information is transmitted to the terminal and displayed on the terminal .

An image recognition device according to the present invention includes: an image receiving unit that receives an image captured by an external terminal; and an image recognition unit that recognizes an image, using the image received by the image receiving unit as a recognition target image. Requesting image recognition, and when the recognition result of the recognition target image obtained from the image recognition unit in response to the image recognition request by the image recognition request unit is a recognition failure, the recognition target image Image recognition re-requesting means for re-requesting image recognition from the image recognizing unit by rotating the direction of the image recognition unit; If the recognition result of the target image is a recognition success, the terminal direction storage means for storing the direction of the terminal based on the direction of the recognition target image, the image recognition by the image recognition requesting unit When the recognition result of the recognition target image obtained from the image recognition unit in response to the request is successful, or obtained from the image recognition unit in response to a re-request for image recognition by the image recognition re-requesting unit. If the recognition result of the recognized recognition target image is a recognition success, a terminal screen including information representing the content of the recognized image is generated based on the direction of the corresponding terminal stored in the terminal direction storage unit. Terminal screen generating means for transmitting the terminal screen generated by the terminal screen generating means to the external terminal, and displaying the terminal screen on the terminal .

  ADVANTAGE OF THE INVENTION According to this invention, even if the terminal which image | photographs a recognition target object is a terminal which does not have a direction detection function, it becomes possible to improve the recognition success rate of the image image | photographed by the said terminal.

FIG. 1 is a functional block diagram showing a configuration of an image recognition system 10 according to an embodiment of a device direction estimation method and an image recognition device of the present invention. 5 is a flowchart showing an image recognition process of the image recognition system 10; FIG. 4 is a sequence diagram showing an operation of the first embodiment in a case where image data first captured by the portable terminal 20 is image-recognized by the image recognition system 10. FIG. 4 is a sequence diagram showing an operation of the first embodiment when image data captured by the portable terminal 20 for the second time or later is recognized by the image recognition system 10. FIG. 9 is a sequence diagram showing an operation of the second embodiment when image data captured first by the portable terminal 20 is recognized by the image recognition system 10. FIG. 9 is a sequence diagram showing an operation of the second embodiment in a case where image data captured second and subsequent times by the portable terminal 20 is image-recognized by the image recognition system 10. FIG. 11 is a functional block diagram showing a configuration of an image recognition system 10 (10A, 10B) according to another embodiment of the device direction estimation method and the image recognition device of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a functional block diagram showing a configuration of an image recognition system 10 according to an embodiment of a device direction estimation method and an image recognition device of the present invention.

  The image recognition system 10 includes a terminal communication unit 11, a terminal control unit 12, an image rotation unit 13, an image inquiry unit 14, a recognition processing unit 15, and a dictionary image storage unit 16.

  The image recognition system 10 is configured by a server device connected to a communication network such as the Internet, and a CPU built in the server device stores a server control program (including an image recognition program) stored in a storage device such as a hard disk in advance. ) To control the operation of each section of the electronic circuit, whereby the sections of the terminal communication section 11, the terminal control section 12, the image rotation section 13, the image inquiry section 14, the recognition processing section 15, and the dictionary image storage section 16 are performed. Function.

  The terminal communication unit 11 (1) receives image data captured by the portable terminal 20 connected via the communication network together with the terminal ID (identification information) of the portable terminal 20 and sends the data to the terminal control unit 12. And (2) a function of transmitting terminal screen data generated by the terminal control unit 12 to the portable terminal 20 together with a screen display instruction in response to the recognition result of the image data sent to the terminal control unit 12.

  The terminal control unit 12 has (1) a function of acquiring the terminal ID sent from the terminal communication unit 11, and (2) a terminal direction (estimated) set value corresponding to the acquired terminal ID is set value storage. A function of determining whether the terminal direction (estimated) setting value corresponding to the terminal ID is stored in the setting value storage unit 12D, based on the setting value; (4) a function of instructing the image rotation unit 13 to rotate the image data sent from the terminal communication unit 11 and requesting the image inquiry unit 14 to recognize the image data rotated by the image rotation unit 13; When the terminal direction (estimated) set value corresponding to the terminal ID is not stored, a function of directly requesting the image inquiry unit 14 to recognize the image data transmitted from the terminal communication unit 11; (5) The image inquiry unit 14 (6) a function of acquiring the recognition result of the image data; (6) when the recognition result of the acquired image data is a recognition failure, instructing the image rotating unit 13 to repeat the rotation processing of the image data (for example, 90 A function of requesting the image inquiring unit 14 to re-recognize the rotated image data while performing the image processing at a maximum of three times (°). (7) The recognition result of the image data acquired from the image inquiring unit 14 is a successful recognition. If the terminal direction (estimated) set value corresponding to the terminal ID is not stored, the terminal direction (estimated) set value corresponding to the terminal ID is determined from the direction of the image data when the recognition is successful. (8) When the recognition result of the image data is a successful recognition, the terminal screen data including information corresponding to the recognition result is newly generated and stored in the setting value storage unit 12D. Or (9) a function of generating in a direction corresponding to the terminal direction (estimated) set value), and (9) responding to the recognition failure if the recognition result after repeating the re-recognition of the image data a maximum number of times is a recognition failure. It has a function of generating terminal screen data including information, (10) a function of transmitting terminal screen data generated according to the recognition result (recognition success or recognition failure) to the terminal communication unit 11, and the like.

  In the present embodiment, the terminal direction (estimated) set value stored in the set value storage unit 12D in association with each terminal ID is “0” as a reference direction, “1” as a direction rotated right 90 °, “ “2” indicates a direction rotated by 180 ° to the right (or 180 ° to the left), and “3” indicates a direction rotated by 270 ° to the right (or 90 ° to the left).

  The image rotation unit 13 has a function of rotating the image data based on the terminal direction (estimated) set value in response to an instruction from the terminal control unit 12.

  The image query unit 14 has a function of (1) sending the image data to a recognition processing unit 15 in response to a request for recognition of image data from the terminal control unit 12, and (2) a function of transmitting the image data from the recognition processing unit 15. And a function of acquiring the recognition result of the above and sending the result to the terminal control unit 12.

  The recognition processing unit 15 determines (1) the similarity between the image data of the recognition target sent from the image inquiry unit 14 and a large number of dictionary image data stored in the dictionary image storage unit 16 It has a function of performing image recognition and responding the recognition result to the image inquiry unit 14.

  The dictionary image storage unit 16 stores, for example, a large number of image data obtained by photographing various things existing in the world as a dictionary image together with information indicating (representing) the contents of the image data.

  In the image recognition system 10 configured as described above, the CPU controls the operation of each part of the circuit according to the instructions described in the server control program (including the image recognition program), and the software and the hardware cooperate. By performing such operations, various functions as described in the following operation description are realized.

  The portable terminal 20 includes a management system communication unit 21, a camera (imaging unit) 22, and a display (display unit) 23, and the portable terminal 20 has a built-in CPU such as a flash ROM or other storage device. By executing a terminal control program stored in advance in the PC and controlling the operation of each section of the electronic circuit, the sections of the management system communication section 21, the camera 22, and the display 23 are made to function.

  The management system communication unit 21 (1) a function of transmitting image data captured by the camera 22 according to a user together with the terminal ID of the terminal 20 to the image recognition system 10 of a server device on the communication network; 2) It has a function of receiving terminal screen data including information corresponding to the recognition result of the transmitted image data from the image recognition system 10 and displaying it on the display 23.

  Next, the operation of the image recognition system 10 having the above configuration will be described.

  FIG. 2 is a flowchart showing the image recognition processing of the image recognition system 10.

  In the flowchart shown in FIG. 2, the process excluding the portion "based on the set value" surrounded by the dotted line in step S12 is the first embodiment, and the process including the portion is the second embodiment.

  The operation of the present embodiment will be described on the assumption that the portable terminal 20 used by a user performs image recognition of image data captured first time and image recognition of image data captured second and subsequent times.

(1st Embodiment)
FIG. 3 is a sequence diagram showing an operation of the first embodiment in a case where the image recognition system 10 performs image recognition of image data captured first by the portable terminal 20.

  When the terminal communication unit 11 of the image recognition system 10 receives the image data of the recognition target that is first shot by the user operation of the portable terminal 20 (Step S1 (Yes) <T1 to T4>), the terminal control unit At 12, the terminal ID of the portable terminal 20 received together with the image data is obtained. Further, the number of times of re-recognition management N = 0 is set (steps S2, S3 <T5>).

  Here, it is determined whether or not the terminal direction (estimated) set value corresponding to the terminal ID is stored in the set value storage unit 12D (step S4), and if it is determined that it is not stored (step S4 (No)). )), The image data recognition request is output to the image inquiry unit 14 (step S6 <T6>).

  Then, the image data of the recognition target is sent from the image inquiry unit 14 to the recognition processing unit 15, and the recognition processing unit 15 outputs the image data of the recognition target and a large number of image data stored in the dictionary image storage unit 16. Image recognition is performed by determining the degree of similarity with the dictionary image data, and the recognition result (recognition success or recognition failure) is returned to the image inquiry unit 14 (step S7 <T7, T8>).

  When the recognition result (for example, recognition failure) is returned from the image recognition unit 15 to the image inquiry unit 14, the recognition result (recognition failure) is acquired in the terminal control unit 12. (Steps S8, S9 (No) <T9, T10>), it is determined that the re-recognition management count N is not (N = 3) at present (N = 0) (Step S14 (No)), and The target image data is rotated by, for example, 90 ° to the right by the image rotation unit 13 (step S15 <T11>). Then, the re-recognition management number N is incremented by (+1) and set to (N = 1) (step S16).

  Then, a request for recognizing the image data rotated right by 90 ° is output from the terminal control unit 12 to the image inquiry unit 14 again (step S <b> 6 <T <b> 12>). Re-recognition of the rotated image data is performed (step S7 <T13, T14>).

  Here, when the recognition result (for example, recognition success) is returned from the image recognition unit 15 to the image inquiry unit 14, the recognition result (recognition success) is acquired in the terminal control unit 12. Then (steps S8, S9 (Yes) <T15, T16>), it is determined that the terminal direction (estimated) set value corresponding to the terminal ID is not stored in the set value storage unit 12D (step S10 (No)). )), A terminal direction setting value corresponding to the terminal ID is generated and stored in the setting value storage unit 12D (step S11 <T17>).

  Here, in the terminal control unit 12, a recognition result (recognition success) is acquired from the image recognition unit 15 in response to a recognition request for image data rotated right by 90 °, so that the portable terminal 20 is in the reference direction. Is determined to be used in a direction rotated 90 ° to the right, the terminal direction setting value “1” is generated and stored.

  Then, the terminal control unit 12 generates terminal screen data including information (information indicating (representing) the content of the recognized image data) corresponding to the recognition result (recognition success) (step S12 <T18>). ), The generated terminal screen data and its display instruction signal are transmitted to the portable terminal 20 via the terminal communication unit 11 (step S13 <T19, T20>).

  As a result, in the portable terminal 20, terminal screen data including information indicating (representing) the content of the image data captured by the user operation is displayed on the display 23, and the user can quickly display information on the captured image. It can be easily obtained <T21, T22>.

  If the recognition result of the terminal control unit 12 in response to the recognition request for re-recognition of the image data with respect to the image inquiry unit 14 fails again (step S9 (No)), the image data is further processed. The image is rotated 90 ° to the right and image recognition is repeatedly performed (steps S14 to S16 → S6 to S8 <T11 to T16>).

  Here, the re-recognition of the image data is repeated at the maximum until the re-recognition management number N is determined to be (N = 3) in the step S14. Then, if the recognition result (recognition failure) is obtained even after the image recognition process performed by rotating the image data by 90 ° right is repeated three times (step S14 (Yes)), the recognition result is obtained. Terminal screen data including information such as a message corresponding to (recognition failure) is generated (step S17), and the generated terminal screen data and its display instruction signal are transmitted to the portable terminal 20 via the terminal communication unit 11. It is transmitted (step S13).

  In a case where a recognition result (successful recognition) is obtained in the first (N = 0) image recognition processing of the image data captured by the portable terminal 20 for the first time by the image recognition system 10 (steps S1 to S4). (No) → S6 to S9 (Yes) <T1 to T8, T14 to T16) Since it is determined that the portable terminal 20 is used in the reference direction, the terminal direction setting value associated with the terminal ID “0” is generated and stored in the set value storage unit 12D (Step S10 (No), S11 <T17>).

  FIG. 4 is a sequence diagram showing the operation of the first embodiment in the case where the image recognition system 10 performs image recognition of image data shot second and subsequent times by the portable terminal 20.

  In the image recognition processing by the image recognition system 10 of the image data captured second and subsequent times by the portable terminal 20, the terminal direction (estimated) set value corresponding to the terminal ID of the terminal 20 is stored in the set value storage unit 12D. If it is determined that the image data is stored (steps S1 to S4 (Yes) <T2 to T5>), the image data is subjected to rotation processing based on the terminal direction setting value, and then subjected to image recognition processing (steps S5 to S5). S7 <T5a to T8>).

  Specifically, when the terminal direction setting value associated with the terminal ID of the portable terminal 20 is, for example, “1”, the image data is rotated right 90 ° by the image rotation unit 13 to perform image recognition. It is processed.

  At this time, based on the terminal direction (estimated) set value corresponding to the use direction of the portable terminal 20, the image data captured by the terminal 20 is subjected to an image recognition process after being subjected to a rotation process in advance. A recognition result (recognition success) can be obtained at the recognition success rate (steps S8, S9 (Yes) <T15, T16>).

  Then, similarly to the above, terminal screen data including information (information indicating (representing) the content of the recognized image data) corresponding to the recognition result (recognition success) is generated (step S10 (Yes) → S12 <). T18>), the generated terminal screen data and its display instruction signal are transmitted to the portable terminal 20 (step S13 <T19, T20>).

  Therefore, according to the image recognition system 10 of the first embodiment having the above configuration, image data captured by the portable terminal 20 is received, and image recognition is performed based on a large number of dictionary image data stored in the dictionary image storage unit 16. If a recognition result (recognition failure) is obtained at the time of processing, the image data is repeatedly subjected to image recognition processing while rotating the image data by, for example, 90 ° to the right by the image rotation unit 13. Then, when a recognition result (successful recognition) is obtained, a terminal direction (estimated) set value is generated in accordance with the rotation angle (amount) of the image data when the recognition success is obtained, and the terminal ID is set as the terminal ID. The setting value is stored in the setting value storage unit 12D in association with the setting value. Thereafter, when image data newly photographed by the portable terminal 20 is received, the terminal direction (estimated) set value stored in the set value storage unit 12D in association with the terminal ID of the terminal 20 , The image data is rotated in advance, and then the image recognition process is performed.

  As a result, even if the portable terminal 20 that captures an object to be recognized is a terminal 20 that does not have a direction detection function, the recognition success rate of image data captured by the terminal 20 is improved regardless of the direction in which the terminal 20 is used. It becomes possible to do.

  In the image recognition system 10 of the first embodiment, when the terminal direction (estimated) set value associated with the terminal ID of the portable terminal 20 is stored in the set value storage unit 12D, the terminal 20 The image data to be recognized received from the terminal is rotated in advance in accordance with the terminal direction (estimation) set value, and then the image recognition processing is performed. However, not only this, but the following second embodiment will be described. In this way, the terminal screen data generated in accordance with the recognition result (successful recognition) of the image recognition processing is generated based on the terminal direction (estimated) set value, and the terminal screen data is generated in accordance with the use direction of the portable terminal 20. May be transmitted.

(2nd Embodiment)
FIG. 5 is a sequence diagram showing an operation of the second embodiment when the image recognition system 10 recognizes an image of image data captured first by the portable terminal 20.

  FIG. 6 is a sequence diagram showing an operation of the second embodiment in the case where the image recognition system 10 performs image recognition of image data shot second and subsequent times by the portable terminal 20.

  In the sequence diagrams of the second embodiment shown in FIGS. 5 and 6, the same operation parts as those in the sequence diagrams of the first embodiment shown in FIGS. 3 and 4 are denoted by the same reference numerals T1 to T22. Description is omitted.

  That is, in the image recognition system 10 of the second embodiment, the terminal control unit 12 obtains a recognition result (successful recognition) from the image recognition unit 15 (Step S9 (Yes) <T8, T14 to T16>), A terminal direction (estimated) setting value corresponding to the terminal ID is newly generated and stored in the setting value storage unit 12D (step S10 (No) → S11), or if it has already been generated and stored (step S10). S10 (Yes)), terminal screen data including information corresponding to the recognition result (recognition success) is generated based on the terminal direction (estimated) set value (step S12 <T18a>).

  Specifically, when the terminal direction setting value associated with the terminal ID of the portable terminal 20 is, for example, “1”, the terminal screen data including information corresponding to the recognition result (recognition success) is used as a reference. The terminal screen data is generated in a direction rotated by 90 degrees to the left from the direction described above, and when it is, for example, "0", the terminal screen data is generated in a direction corresponding to the reference direction.

  Then, the generated terminal screen data and its display instruction signal are transmitted to the portable terminal 20 via the terminal communication unit 11 (Step S13 <T19, T20>).

  Thereby, in the portable terminal 20, the terminal screen data including information indicating (representing) the content of the image data captured by the user operation is displayed in the direction in which the user is looking, regardless of the use direction of the terminal 20. The information is displayed, and the user can quickly, easily, and easily obtain information on the captured image <T21, T22>.

  Therefore, according to the image recognition system 10 of the second embodiment having the above configuration, similarly to the first embodiment, even if the portable terminal 20 that shoots the recognition target is the terminal 20 that does not have a built-in direction detection function, Regardless of the direction in which the terminal 20 is used, the recognition success rate of image data captured by the terminal 20 can be improved, and the terminal screen data including the information of the image data that has been recognized can be displayed in the same manner. It can be generated in accordance with the direction of use of the terminal 20 and displayed for easy viewing.

(Other embodiments)
The image recognition system 10 of the first and second embodiments includes a terminal communication unit 11, a terminal control unit 12, an image rotation unit 13, an image inquiry unit 14, an image recognition unit 15, and a dictionary image storage unit 16. It is configured to be included in one image recognition system 10.

  FIG. 7 is a functional block diagram showing a configuration of an image recognition system 10 (10A, 10B) according to another embodiment of the device direction estimation method and the image recognition device of the present invention.

  In another embodiment, the image recognition system 10 includes a terminal management system (server device A) 10A including the terminal communication unit 11, the terminal control unit 12, the image rotation unit 13, and the image inquiry unit 14, and the image recognition unit. 15, an image recognition system (server device B) 10B including a dictionary image storage unit 16 and configured separately.

  According to this, the terminal management system (server device A) 10A includes an external image recognition system (server device B) dedicated to image recognition together with a recognition request for image data to be recognized taken by the portable terminal 20. 10B, and receives the recognition result (recognition success or recognition failure) from an external image recognition system (server device B) 10B dedicated to the image recognition.

  In the image recognition processing of each of the embodiments, the image recognition processing is performed after rotating the image data captured by the portable terminal 20 based on the terminal direction (estimated) set value stored in the set value storage unit 12D. If performed, the direction of the image data and the dictionary image data match, and a recognition result (recognition success) is obtained (steps S1 to S9 (Yes) <T1 to T5a to T8, T15, T16>). ).

  However, if the user's use direction of the portable terminal 20 changes after the terminal direction (estimated) set value is stored in the set value storage unit 12D, the photographed image data is stored in the terminal direction (estimated). Even if the image recognition processing is performed after the rotation based on the set value, the recognition result (recognition failure) is obtained (Steps S1 to S9 (No)). Is rotated and the image recognition process is performed again to obtain a recognition result (recognition success) (steps S14 to S16 → S6 to S9 (Yes)).

  Therefore, in this case, even if the terminal direction (estimated) set value corresponding to the terminal ID of the portable terminal 20 is already stored in the set value storage unit 12D (step S10 (Yes)), the terminal ID (estimated) corresponding to the terminal ID is not changed. The terminal direction (estimation) setting value may be overwritten and updated based on the rotation angle (amount) of the image data when the recognition result (recognition success) is obtained.

  The method of each processing by the image recognition system 10 described in each of the above embodiments, that is, the image recognition processing shown in the flowchart of FIG. 2, the image recognition processing of the first embodiment shown in the sequence diagrams of FIGS. Each of the techniques such as the image recognition processing of the second embodiment shown in the flowchart of FIG. (Trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, etc., and stored in a medium of an external recording device for distribution. The computer (control device: CPU) of the electronic device reads the program recorded on the medium of the external recording device into the storage device, and the operation is controlled by the read program. It realizes an image recognition function of image data captured by the portable terminal, and can execute the same processing by the above-described method.

  Further, the data of the program for realizing each of the above methods can be transmitted over a network in the form of a program code. To realize the image recognition function of the image data photographed by the portable terminal described above.

  The present invention is not limited to the above embodiments, and can be variously modified in an implementation stage without departing from the scope of the invention. Furthermore, the embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent features. For example, even if some components are deleted from all the components shown in each embodiment, or some components are combined in different forms, the problems described in the section of the problem to be solved by the invention. Can be solved and the effect described in the section of the effect of the invention can be obtained, a configuration in which this component is deleted or combined can be extracted as the invention.

Reference Signs List 10 image recognition system 11 terminal communication unit 12 terminal control unit 12D setting value storage unit (terminal direction (estimated) setting value)
DESCRIPTION OF SYMBOLS 13 ... Image rotation part 14 ... Image inquiry part 15 ... Recognition processing part 16 ... Dictionary image storage part 10A ... Terminal management system (other embodiment)
10B ... Image recognition system (other embodiment)
Reference numeral 20: portable terminal 21: management system communication unit 22: camera (imaging unit)
23… Display (display unit)

Claims (9)

Get the image taken by the terminal,
Requesting image recognition to an image recognition unit that recognizes the image, with the obtained image as a recognition target image,
When the recognition result of the recognition target image acquired from the image recognition unit in response to the request for image recognition is a recognition failure, the direction of the recognition target image is rotated to perform image recognition by the image recognition unit. Request again,
If the recognition result of the recognition target image acquired from the image recognition unit in response to a re-request for the image recognition is a recognition success, the direction of the terminal is estimated based on the direction of the recognition target image,
If the recognition result of the recognition target image obtained from the image recognition unit in response to the request for image recognition is a successful recognition, or obtained from the image recognition unit in response to a re-request for image recognition If the recognition result of the recognition target image is successful recognition, a terminal screen including information representing the content of the recognized image is generated based on the direction of the terminal that is estimated,
Transmitting the generated terminal screen to the terminal and causing the terminal to display the terminal screen,
Device orientation estimation method. The request for image recognition, when acquiring an image captured by the terminal, if the direction of the terminal is estimated, after rotating the direction of the acquired image based on the direction of the terminal, Requesting the image recognition unit to perform image recognition on the rotated image as a recognition target image,
The method of claim 1, wherein: The re-request for the image recognition is, when the recognition result of the recognition target image acquired from the image recognition unit in response to the request for the image recognition is a recognition failure, the direction of the recognition target image is set in advance. Re-requesting image recognition from the image recognition unit for each rotation while rotating by the rotation amount,
The device direction estimating method according to claim 1 or 2, wherein: If the recognition result of the recognition target image obtained from the image recognition unit in response to the request for image recognition is a successful recognition, or obtained from the image recognition unit in response to a re-request for image recognition If the recognition result of the recognition target image is successful recognition, if the direction of the terminal has already been estimated, newly estimate the direction of the terminal based on the direction of the successfully recognized recognition target image,
Device direction estimation method according to any one of claims 1 to 3, characterized in that. Image receiving means for receiving an image taken by an external terminal;
Image recognition requesting means for requesting image recognition to an image recognition unit that recognizes an image, using the image received by the image receiving means as a recognition target image,
When the recognition result of the recognition target image obtained from the image recognition unit in response to the image recognition request by the image recognition request unit is a recognition failure, the direction of the recognition target image is rotated to perform the image recognition. Means for re-requesting image recognition from the unit,
If the recognition result of the recognition target image acquired from the image recognition unit in response to the re-request for image recognition by the image recognition re-requesting unit is a successful recognition, the terminal based on the direction of the recognition target image. Terminal direction storage means for storing the direction of
When the recognition result of the recognition target image acquired from the image recognition unit in response to the image recognition request by the image recognition request unit is a successful recognition, or when the image recognition requesting unit re-requests the image recognition. In a case where the recognition result of the recognition target image obtained from the image recognition unit in response is a successful recognition, a terminal screen including information representing the content of the recognized image is stored by the terminal direction storage unit. Terminal screen generating means for generating based on the direction of the corresponding terminal,
Terminal screen transmitting means for transmitting the terminal screen generated by the terminal screen generating means to the external terminal and causing the terminal screen to display the terminal screen;
An image recognition device comprising: The image recognition requesting unit is configured such that, when an image captured by the external terminal is received by the image receiving unit, if the direction of the terminal is stored by the terminal direction storage unit, the received image is received. After rotating the direction of the image based on the direction of the terminal, request the image recognition unit to perform image recognition on the rotated image as a recognition target image,
The image recognition device according to claim 5 , wherein: The image recognition re-requesting unit, when the recognition result of the recognition target image obtained from the image recognition unit in response to the image recognition request by the image recognition requesting unit is a recognition failure, the recognition target image Re-requesting image recognition from the image recognition unit for each rotation while rotating the direction by a preset rotation amount,
The image recognition device according to claim 5 or 6 , wherein When the recognition result of the recognition target image acquired from the image recognition unit in response to the image recognition request by the image recognition request unit is a successful recognition, or when the image recognition requesting unit re-requests the image recognition. When the recognition result of the recognition target image acquired from the image recognition unit in response is successful recognition, when the direction of the corresponding terminal is stored by the terminal direction storage unit, the direction of the terminal is changed. Comprising a terminal direction update storage means for updating and storing based on the direction of the recognition target image that has been successfully recognized,
The image recognition device according to any one of claims 5 to 7 , wherein: Computer
Image receiving means for receiving an image taken by an external terminal;
Image recognition requesting means for requesting image recognition to an image recognizing unit for recognizing an image, wherein the image received by the image receiving means is a recognition target image,
When the recognition result of the recognition target image obtained from the image recognition unit in response to the image recognition request by the image recognition request unit is a recognition failure, the direction of the recognition target image is rotated to perform the image recognition. Means for re-requesting image recognition from the unit,
If the recognition result of the recognition target image acquired from the image recognition unit in response to the re-request for image recognition by the image recognition re-requesting unit is a successful recognition, the terminal based on the direction of the recognition target image. a terminal direction estimating means for estimating the direction of,
When the recognition result of the recognition target image acquired from the image recognition unit in response to the image recognition request by the image recognition request unit is a successful recognition, or when the image recognition requesting unit re-requests the image recognition. In a case where the recognition result of the recognition target image obtained from the image recognition unit in response is a successful recognition, a terminal screen including information representing the content of the recognized image is estimated by the terminal direction estimation unit. Terminal screen generating means for generating based on the direction of the corresponding terminal,
Terminal screen transmitting means for transmitting the terminal screen generated by the terminal screen generating means to the external terminal and displaying the terminal screen on the terminal;
A computer-readable program to function as a computer.