JP7477222B1 - Mobile terminal device, incoming call screen display method and program - Google Patents

Abstract

[Problem] In a mobile terminal, information and functions available when a video call is received are utilized to improve user convenience. [Solution] A mobile terminal device includes an incoming signal receiving unit that receives an incoming signal including a caller number, caller information, and preview video information when a video call is received, an incoming screen display unit that displays an incoming screen including the caller number, the caller information, and a caller information image extracted from the preview video information, and a banner notification sending unit that sends a banner notification including the caller information image extracted from the preview video information at regular intervals. [Selected Figure] Figure 1

Description

The present invention relates to a mobile terminal device, an incoming call screen display method, and a program.

For example, when an incoming call is received from a doorphone with a camera, the video data of the visitor captured by the doorphone with a camera is acquired and used as the incoming call screen data for the doorphone with a camera. Then, there is a technology that displays an incoming call screen according to the incoming call screen data of the doorphone with a camera to notify the user of an incoming call from the doorphone with a camera (for example, see Patent Document 1).

JP 2020-205486 A

The following analysis is provided by the present invention.

Video calls are being made between mobile terminal devices. Mobile terminal devices are typified by smartphone terminals and are broadly divided into iPhone (registered trademark) terminals that use iOS, developed by Apple, as their OS (Operating System), and terminals that use Android (registered trademark), the basic framework of which was developed by Google.

When making a video call on an iPhone, a VoIP (Voice over Internet Protocol) application (video call app) for making voice and/or video calls is run on the iPhone. iPhones come with a phone function application built into their OS (iOS), and a framework called CallKit is responsible for controlling incoming and outgoing calls in that application. Video call apps can also use CallKit for outgoing and incoming call control. However, when using CallKit, the incoming call screen will be the CallKit incoming call screen (also called the standard incoming call screen). When a call is received, this standard incoming call screen is generated by CallKit from information registered in the phone book. Therefore, when using CallKit, the information you want to display on the display screen when a call is received must be registered in the phone book beforehand.

If a video calling app does not use CallKit, the incoming call screen can be created freely. However, if CallKit is not used and an incoming call occurs on the OS's phone application during a call with a video calling app, the OS's phone application takes priority and the video call is interrupted. Also, if CallKit is not used and an incoming call is received while the device security is locked, the user must first release the security lock, significantly reducing usability. As such, it is desirable to use CallKit when operating a video calling app in order to manage conflicts with other calling applications and to answer incoming calls when the device security is locked.

According to the technology disclosed in Patent Document 1, when an incoming call is received from a camera-equipped doorphone, the video data of the visitor captured by the camera-equipped doorphone is stored as incoming call screen data in association with the camera-equipped doorphone's number information, and is displayed on the incoming call screen to notify the user of the incoming call. If the data is stored in the phone book on the iPhone device, CallKit allows the user to check the video of the caller before answering the video call.

However, since only one frame of the video can be displayed and the image is not necessarily optimal, the preview feels unsatisfactory. Furthermore, when a call is received between mobile terminal devices, the caller's voice may be received along with the video preview. However, even if the technology disclosed in Patent Document 1 is applied, this cannot be effectively utilized on iPhone terminals.

The present invention has been made in consideration of the above circumstances, and aims to provide a technology that contributes to improving user convenience by utilizing information and functions available on a mobile terminal device when a video call is received.

According to a first aspect of the present invention,
an incoming call signal receiving unit for receiving an incoming call signal including a caller number, caller information, and preview video information when a video call is received;
an incoming call screen display unit that displays an incoming call screen including the caller number, the caller information, and a caller information image extracted from the preview video information;
A banner notification sending section is provided for sending a banner notification including a sender information image extracted from the preview video information at regular intervals.

According to a second aspect of the present invention,
An incoming call screen display method executed by a mobile terminal device, comprising:
When a video call is received, an incoming call signal including a caller number, caller information, and preview video information is received;
An incoming call screen display method is provided which displays an incoming call screen including the caller number, the caller information, and a caller information image extracted from the preview video information, and transmits and displays a banner notification including the caller information image extracted from the preview video information at regular intervals.

According to a third aspect of the present invention,
Computer,
means for receiving an incoming call signal including a caller number, caller information, and preview video information when a video call is received;
means for displaying an incoming call screen including the caller number, the caller information, and a caller information image extracted from the preview video information;
A program is provided to function as a means for transmitting and displaying a banner notice including a sender information image extracted from the preview video information at regular intervals.

The program can be recorded on a computer-readable storage medium. The storage medium can be a non-transient medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. The present invention can also be embodied as a computer program product.

The present invention contributes to improving user convenience by utilizing information available when a video call is received from a mobile device.

1 is a block diagram showing a schematic configuration of a mobile terminal device according to the present disclosure. 1A and 1B are explanatory diagrams for explaining an example of an incoming call screen of the present disclosure, and FIG. 1C is an explanatory diagram for explaining an example of a standard incoming call screen provided in the CallKit. 1 is a diagram illustrating an example of an overall configuration of a telephone system according to a first embodiment; FIG. 2 is a diagram illustrating an example of a functional block of the mobile phone terminal according to the first embodiment. 5A and 5B are diagrams illustrating an example of an incoming call signal and a telephone directory according to the first embodiment, respectively; FIG. 1A is an explanatory diagram for explaining an example of an incoming call screen in the first embodiment, FIG. 1B and FIG. 1C are explanatory diagrams for explaining examples of a banner notification displayed on the incoming call screen in the first embodiment, and FIG. 1D is an explanatory diagram for explaining an example of a banner notification in the first embodiment. 11 is a flowchart of an incoming call screen display process according to the first embodiment. 11 is a flowchart of a banner notification process according to the first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a mobile phone terminal according to the first embodiment. FIG. 11 is an explanatory diagram for explaining an example of a banner notification to which a response receiving unit according to the second embodiment is added; FIG. 11 is a diagram illustrating an example of a functional block of a mobile phone terminal according to a second embodiment. 13A to 13D are diagrams illustrating examples of incoming call screens according to a modified example of the embodiment of the present invention.

Below, an overview of one embodiment of the present invention (hereinafter referred to as this embodiment) will be described with reference to the drawings. Note that the reference symbols in the drawings are given for convenience to each element as an example to aid understanding, and are not intended to limit the present invention to the illustrated form. Furthermore, the connection lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional lines. Unidirectional arrows are used to diagrammatically indicate the flow of the main signal (data), and do not exclude bidirectionality.

In addition, although there are ports and interfaces at the input/output connection points of each block in the diagram, they are not shown. In addition, in the following explanation, "A and/or B" is used to mean A or B, or A and B.

Before describing the embodiments of the present disclosure, an overview of the present disclosure will be described. In the present disclosure, a process that is not covered by a standard phone app operation UI that is pre-installed in a mobile terminal device is realized within the framework of the standard phone app operation UI. Specifically, a video preview of a video call is displayed on the standard incoming call screen of CallKit running on iOS.

Fig. 1 is a schematic diagram of a mobile terminal device 800 according to the present disclosure. The mobile terminal device 800 is, for example, a terminal (iOS terminal) equipped with iOS as the OS, and includes an incoming call signal receiving unit 810, a phone book registration unit 820, an incoming call screen display unit 830, and a banner notification sending unit 840.

When a video call is received, the incoming call signal receiving unit 810 receives the incoming call signal. The incoming call signal includes a caller number, caller information, and preview video information. The preview video information includes one or more caller information images (frames). A video call is an incoming call that requests a video call.

The phonebook registration unit 820 extracts a caller information image from the preview video information and registers the extracted caller information image in the phonebook 850 managed by the mobile terminal device 800. At this time, the caller information is registered in association with the caller number included in the incoming call signal. Furthermore, caller information may also be registered at the same time. This phonebook 850 is a phonebook that the CallKit provided in the iOS terminal accesses when receiving an incoming call. The CallKit is a framework that allows applications to use the UI (User Interface) functions required for making calls using VoIP.

The incoming call screen display unit 830 receives the caller number, refers to the telephone directory 850, and acquires the caller information image registered in association with the received caller number. If the caller information is registered, it is also acquired. The incoming call screen display unit 830 then generates an incoming call screen 300 including the caller number 311 (and the caller information) and the caller information image 321, as shown in FIG. 2(a), and displays it on the display screen of the mobile terminal device 800 to notify the user of the incoming call.

After displaying the incoming call screen 300, the banner notification sending unit 840 repeats the process of creating and displaying a banner notification 350 at a predetermined time interval (at every fixed time) until the user responds. As shown in FIG. 2(a), the banner notification 350 includes a caller information image 321 extracted from the preview video information as a banner image 351. The banner notification 350 is displayed in a banner display area on the display screen of the mobile terminal device 800.

For example, if the preview video information includes audio information, the banner notification sending unit 840 extracts the audio information for a predetermined period of time and obtains text information by voice recognition. The obtained text information may then be added to the banner notification 350 as banner text 352. Note that the voice recognition may be performed within the mobile terminal device 800, or may be sent to an external server, such as a voice recognition service server on the cloud, and the converted text may be received.

Furthermore, the banner notification sending unit 840 may add a response means (response receiving unit 360) including at least a video response, an audio response, and a rejection to the banner notification 350 and send it, as shown in FIG. 2(b).

In the present disclosure, when a call is received, the caller information image obtained from the preview video information is associated with the caller number and registered in the phone book 850. If the OS of the mobile terminal device 800 is iOS, processing by CallKit is possible. Therefore, even if an incoming call occurs in another phone function application, the other application is not given priority, and the call can be answered without releasing the security lock even if the call is received during a call security lock.

The CallKit acquires information registered in the phone book 850 and displays the incoming call screen 300. For this reason, it is necessary to register the information to be displayed on the incoming call screen 300 in the phone book 850 in advance in each mobile terminal device 800. However, in a usage environment in which the owner of the mobile terminal device 800 changes frequently, for example, it is troublesome to change and update the registered information every time. Furthermore, it is difficult to convey the nuances of an emergency situation, such as a still image or the first frame of an image, that are registered in advance in the phone book 850.

In addition, the standard incoming call screen 390 provided in CallKit does not provide a preview or voice response function for video incoming calls. As shown in FIG. 2(c), the standard incoming call screen 390 includes a text display area 391 in which a phone number and/or name is displayed in text, an image display area 392, and an instruction display area 393. Therefore, even in the case of a video incoming call, as shown in FIG. 2(c), text and a contact photo that has been registered in advance in the phone book 850 are displayed. Also, the instruction display area 393 displays only an accept button to answer or reject the video incoming call.

When used by, for example, a medical professional, such a display of the standard incoming call screen 390 makes it difficult to tell whether the call is a normal business call or an emergency call. In other words, because a preview of the other party's image cannot be displayed during a video call, the other party's preview video information is not transmitted during the video call, and there are frequent problems, particularly when a nurse, who is likely to be facing an emergency, contacts a doctor, where the caller misses an incoming call that should be answered or answers an incoming call that the medical professional does not want to answer. However, according to the present disclosure, preview video information of the caller is transmitted moment by moment during a video call, resolving such situations.

Furthermore, the preview video information may contain the caller's voice, but the iOS mobile terminal device 800 cannot play the voice over the ringtone. However, according to the present disclosure, the voice is converted to text and displayed as a banner notification 350, so the callee can also understand the contents of the caller's voice.

Furthermore, the standard incoming call screen 390 of the iOS mobile terminal device 800 generally has one answer button. For incoming video calls, only video incoming can be selected. Even if you want to respond by voice only, you cannot respond to voice-only calls. However, according to the present disclosure, a response acceptance section 360 is displayed and a variety of responses are accepted. Therefore, voice-only responses are also possible.

<<First embodiment>>
Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. Fig. 3 shows an example of the overall configuration of a telephone system 101 of the present embodiment. As shown in this figure, in this embodiment, a video call is made from a mobile phone terminal (sender terminal 100a) that is a call source to a mobile phone terminal (receiver terminal 100b) that is a call destination via a relay server 910.

The sender terminal 100a and the receiver terminal 100b are assumed to be mobile phone terminals equipped with iOS (for example, an iPhone terminal manufactured by Apple, etc.). Hereinafter, unless there is a need to distinguish between them, they will be represented by the mobile phone terminal 100. Furthermore, it is assumed that the mobile phone terminal 100 is equipped with a VoIP application (video calling application) of this embodiment. The video calling application is an application that controls voice calls, video calls, and the origination and reception of such calls. The relay server 910 is a VoIP relay server capable of relaying VoIP.

Furthermore, the recipient terminal 100b of this embodiment can access a voice recognition server 920 via a network 900 such as the Internet. The voice recognition server 920 is a server that recognizes voice and converts it into text.

Figure 4 is a functional block diagram of the mobile phone terminal 100 of this embodiment. As shown in this figure, the mobile phone terminal 100 of this embodiment is equipped with iOS as the OS 110. In addition, the mobile phone terminal 100 has a video calling application 120 installed, and is equipped with a phone book 130, a CallKit 140, a network interface (NW I/F) 150, and an external interface (external I/F) 160.

The video calling application 120 and CallKit 140 run on the OS 110. The phone book 130 is a database and can be accessed from the video calling application 120 and CallKit 140.

The video call application 120 includes an incoming call processing section that processes incoming calls and a call processing section that processes calls. The details of the incoming call processing section related to this embodiment are described below.

The incoming call processing unit of the video calling application 120 receives an incoming call (video incoming call) requesting a video call from another mobile phone terminal or the like, and generates an incoming call screen that realizes a video preview. The incoming call processing unit of the video calling application 120 of this embodiment includes an incoming call signal receiving unit 121, a registration data generating unit 122, a phone book operating unit 123, a CallKit operating unit 124, a banner notification control unit 125, and a timer 126.

The incoming signal receiving unit 121 receives an incoming signal 200 of a video call transmitted from another mobile phone terminal or the like via the NW I/F 150. As shown in FIG. 5(a), the incoming signal 200 includes a caller number 210, caller information 220, and preview video information 230. When the incoming signal 200 is an incoming signal of a video call, the incoming signal receiving unit 121 passes the caller number 210, caller information 220, and preview video information 230 contained in the incoming signal 200 to the registration data generating unit 122.

For example, when the incoming signal 200 contains preview video information 230, the incoming signal receiving unit 121 determines that the incoming call is a video call. In addition, the incoming signal 200 may contain information that specifies the type of incoming call. In this case, the incoming signal receiving unit 121 determines the type of incoming call based on the information that specifies the type of incoming call.

The registration data generating unit 122 generates registration data to be registered in the telephone directory 130 based on the information received from the incoming signal receiving unit 121. The registration data includes a caller number 210, caller information 220, and a caller information image. The caller information image is an image obtained from the preview video information 230. For example, one frame is extracted from the stream of the preview video information 230 to be used as the caller information image.

For example, when the preview video information 230 is provided as Web content, the registration data generation unit 122 analyzes the HTML (Hyper Text Markup Language) and identifies the image data (image file). Then, for example, the image data in the first frame is extracted as the caller information image.

The telephone directory operation unit 123 operates the data in the telephone directory 130. As shown in FIG. 5(b), a telephone number 131, a name 132, and an image 133 can be registered in the telephone directory 130.

The phone book 130 further includes a save area 134 and notes 135. The save area 134 is an area for temporarily storing image data. Furthermore, the notes 135 stores information for identifying the record type. In this embodiment, for example, if the record is a new record created for the video preview of this embodiment, a flag is set. Note that the save area 134 and notes 135 may be stored in a storage area separate from the phone book 130.

In this embodiment, when the telephone directory operation unit 123 receives registration data from the registration data generation unit 122, it determines whether the caller's information has already been registered. In this embodiment, for example, it compares the caller number 210 in the registration data with the telephone number 131 to determine whether a match has been registered. If a matching record has been registered, it temporarily saves the image 133 of that record to the save area 134, and registers the caller information image in image 133.

On the other hand, if the record is not registered, the registration data is used to create a record. The telephone directory operation unit 123 registers the caller number 210, caller information 220, and caller information image received from the registration data generation unit 122. For example, the caller number 210 is registered in the telephone number 131, part or all of the caller information 220 in the name 132, and the caller information image in the image 133. In this case, a flag is set in the notes 135 to indicate that the record has been registered for video preview.

At a predetermined timing, the telephone directory operation unit 123 deletes from the telephone directory 130 records for which a flag is set in the notes 135. In addition, at a predetermined timing, the telephone directory operation unit 123 returns images saved in the save area 134 to the images 133. The predetermined timing is, for example, when the incoming call processing ends, specifically, when the user answers the incoming call or when the user rejects the incoming call. Alternatively, the predetermined timing may be when the call for the incoming call ends.

After the phonebook operation unit 123 registers the registration data in the phonebook 130, the CallKit operation unit 124 notifies the CallKit 140 of the caller number 210 to notify the incoming call.

CallKit 140 is a framework that provides OS 110 with the functions of managing conflicts between applications that use the calling function, of allowing incoming calls during security lock, and of displaying an incoming call screen. When CallKit 140 is notified of an incoming call, it uses data from the phone book 130 to generate a standard incoming call screen 390 as an incoming call screen 300. Specifically, it obtains the name 132 and image 133 registered in association with the caller number (phone number 131) from the phone book 130. CallKit 140 generates the incoming call screen 300 including the phone number 131 and the obtained name 132 and image 133, and displays it on the display screen via the external interface (external I/F) 160.

Figure 6 (a) shows an example of an incoming call screen 300 displayed on the display screen. The incoming call screen 300 comprises a text display area 310, an image display area 320, and an instruction display area 330. The text display area 310, the image display area 320, and the instruction display area 330 correspond to the text display area 391, the image display area 392, and the instruction display area 393 of the standard incoming call screen 390, respectively.

In the text display area 310, the caller number (phone number 131) and data obtained from the name 132 in the telephone directory 130 are displayed as text data. Specifically, the caller's phone number 131 and the name 132, which is the caller's information, are displayed. Either the phone number 131 or the caller's information may be displayed. In addition, in the image display area 320, a caller information image is displayed. Furthermore, in the instruction display area 330, an acceptance button or the like is displayed to accept the user's intention to answer or reject the incoming call. Note that if the number of characters of the text data to be displayed in the text display area 310 exceeds the number of characters that can be displayed in the text area, the text data may be displayed, for example, by scrolling in the text display area 310.

The timer 126 measures time. In this embodiment, it starts measuring time when it receives the incoming signal 200, and notifies the banner notification control unit 125 at a predetermined time interval.

The banner notification control unit 125 generates a banner notification to be superimposed on the incoming call screen 300. In this embodiment, each time a notification is received from the timer 126, a predetermined frame of image is extracted from the preview video information 230 and used as a banner image. The frame to be extracted is, for example, the most recent frame. The image extraction method is the same as the extraction method used by the registration data generation unit 122. When generating a banner notification, the banner notification control unit 125 also determines whether the incoming call screen 300 is being displayed.

The banner notification control unit 125 of this embodiment further generates banner text. The banner text includes, for example, the caller number 210 included in the incoming call signal 200 and a predetermined message. The message is, for example, "Video call incoming." It may also include caller information 220.

Furthermore, the banner text of this embodiment may include text information obtained by converting audio information (audio data) included in the preview video information 230 into text. In this case, the banner notification control unit 125 extracts audio information for a predetermined period from the preview video information 230. Then, the extracted audio data is converted into text information to generate the banner text. If the device itself (mobile phone terminal 100) is equipped with a voice recognition application that recognizes and converts audio data into text, that voice data may be used. Alternatively, an external voice recognition service may be used.

For example, when using a cloud-based voice recognition service (a cloud service that transcribes voice data) as the external voice recognition service, the banner notification control unit 125 transmits voice data for a predetermined period of time via the NW I/F 150 to a voice recognition server 920 that provides a voice recognition service on the network 900, and uses the text information returned in response.

The specified period for extracting audio data may be set to a customizable length, for example, 5 seconds. For example, if the banner notification interval is 5 seconds, text conversion must be completed and returned to the video call application within 4 seconds. If the banner notification control unit 125 cannot detect text within 4 seconds, for example, it continues to accumulate audio data until the next interval. After accumulating audio data for a maximum of 10 seconds, it resets the audio data regardless of whether text detection was successful. The specified period may be the same as the specified interval for generating a banner notification, for example.

The banner notification control unit 125 places the generated banner image and banner text in a predetermined area at a predetermined time interval to generate a banner notification. Each time a banner notification is generated, it is sent via the external I/F 160 and displayed in a predetermined area on the display screen.

Figure 6(b) is a display example when banner notification 350 is superimposed on incoming call screen 300. Also, Figure 6(d) is a view showing only banner notification 350 enlarged. As shown in Figure 6(d), banner notification 350 comprises banner image 351 and banner text 352. As described above, if banner text 352 exceeds the number of characters that can be displayed in the display area, banner notification control unit 125 may display it by scrolling. Also, banner notification control unit 125 may change the size of banner notification 350 depending on the number of characters, as shown in Figure 6(c).

Specifically, the banner notification control unit 125 may adjust the size of the banner notification 350 so that all characters of the banner text 352 fit, for example. The banner notification control unit 125 may also adjust the font size of the display text displayed in the banner notification 350 so that all characters of the banner text 352 fit.

In the above, the mobile phone terminal 100 corresponds to the mobile terminal device 800 in FIG. 1. Also, the incoming call signal receiving unit 121 corresponds to the incoming call signal receiving unit 810, and the registration data generating unit 122 and the phone book operation unit 123 correspond to the phone book registration unit 820. The CallKit operation unit 124 and CallKit 140 correspond to the incoming call screen display unit 830. The banner notification control unit 125 corresponds to the banner notification sending unit 840. The phone book 130 corresponds to the phone book 850.

[Video Incoming Call Processing]
The flow of the process performed by the mobile phone terminal 100 of this embodiment when receiving a video call will be described below. In this embodiment, the incoming call screen display process and the banner notification process are performed in parallel. Each of these processes will be described below.

[Incoming call screen display process]
7 is a process flow of the incoming call screen display process according to the present embodiment. This process is started when the incoming call signal receiving unit 121 of the video calling application 120 receives the incoming call signal 200.

The incoming call signal receiving unit 121 analyzes the incoming call signal 200 and determines whether the incoming call is a call requesting a video call (video incoming call) (step S1101). For example, if the incoming call signal includes preview video information 230, the incoming call signal receiving unit 121 determines that the incoming call is a video incoming call.

If the call is not a video call, the process ends. In this case, CallKit 140 receives the call and performs call processing such as generating and displaying the standard incoming call screen 390.

When the incoming call signal receiving unit 121 determines that the incoming call is a video call (S1101; Yes), it passes the caller number 210, caller information 220, and preview video information 230 contained in the incoming call signal 200 to the registration data generating unit 122. The registration data generating unit 122 extracts one frame from the preview video information 230 (step S1102) and obtains a caller information image. Then, the caller number 210, caller information 220, and caller information image are treated as registration data (step S1103). The generated registration data is passed to the phone book operating unit 123.

The telephone directory operation unit 123 determines whether the caller record corresponding to the registration data is already registered in the telephone directory 130 (step S1104). In this embodiment, if a number matching the caller number 210 of the registration data is registered in the telephone number 131 of the telephone directory 130, it is determined that the caller record is already registered.

If it is determined that the record is already registered (S1104; Yes), the telephone directory operation unit 123 saves the image 133 of the record in question to the save area 134. Then, the caller information image included in the registration data is registered as the image 133.

On the other hand, if it is not determined that the data is already registered (S1104; No), the telephone directory operation unit 123 adds the registered data to the telephone directory 130 as a new record (step S1106). At this time, a flag is set in the notes 135.

When the operation by the phonebook operation unit 123 is completed, the CallKit operation unit 124 notifies the CallKit 140 of the caller number 210 and notifies the CallKit 140 of the incoming call. This causes the CallKit 140 to generate an incoming call screen 300 by referring to the phonebook 130 and display it on the display screen (step S1107).

After that, when CallKit 140 receives a response instruction from the user via the instruction display area 330 (step S1108), the phonebook operation unit 123 operates the phonebook 130 (step S1109), returns to the original state, and ends the process. Note that in this embodiment, the user's response is either to answer the call or to reject the call. Also, the operation of the phonebook 130 performed in step S1109 is, for example, to move an image registered in the save area 134 to image 133, and to delete a record with a flag set in remarks 135.

[Banner notification processing]
Next, the banner notification process will be described. Fig. 8 shows a process flow of the banner notification process of this embodiment. Like the incoming call screen display process, this process is also started when the incoming call signal receiving unit 121 of the video calling application 120 receives the incoming call signal 200. This process is executed independently of the incoming call screen display process. Hereinafter, the "predetermined period" for updating the banner notification 350 and the period for storing the voice data will be the same, and will be represented as T.

The banner notification process is performed when the incoming call signal receiving unit 121 determines during the incoming call screen display process that the incoming call is a video call (step S1101).

The banner notification control unit 125 clears the timer 126 and the accumulated audio data (step S1201).

Then, the banner notification control unit 125 extracts audio data from the preview video information 230 and starts storing it (step S1202). Then, after a predetermined period T has elapsed (step S1203), the banner notification control unit 125 converts the stored audio data into text data (step S1204). For example, the text data is obtained by sending the audio data to an external audio transcription cloud service (speech recognition server 920) or the like. If the device itself (mobile phone terminal 100) is equipped with the application, it may be used. In this way, the banner notification control unit 125 obtains the banner text 352.

The banner notification control unit 125 also extracts one frame from the preview video information 230 (step S1205) to obtain a banner image 351. Here, the banner notification control unit 125 extracts, for example, the latest frame of the preview video information 230 at that time.

The banner notification control unit 125 generates a banner notification 350 using the banner text 352 and the banner image 351, and displays it in a specified area on the display screen (step S1206).

Then, the banner notification control unit 125 determines whether the CallKit 140 has received a response instruction in the above-mentioned incoming call screen display process (step S1108). If a response instruction has been received (S1108; Yes), the banner notification 350 displayed on the display screen is erased (step S1207) and the process ends. On the other hand, if a response instruction has not been received, i.e., if the incoming call screen is still being displayed, the process returns to step S1201 and the process is repeated.

[Hardware configuration]
The following describes the hardware configuration of the mobile phone terminal 100. Fig. 9 is a diagram showing the hardware configuration of the mobile phone terminal 100. The mobile phone terminal 100 includes a processor 191, a storage unit 192, a ROM (Read Only Memory) 193, a RAM (Random Access Memory) 194, a communication interface (I/F: Interface) 195, a user interface 196, a microphone 197, and a speaker 198.

The communication I/F 195 is an interface for connecting the relay server 910 (see FIG. 3) and the mobile phone terminal 100 via wireless communication means. The mobile phone terminal 100 receives an incoming call signal from the relay server 910 via the communication I/F 195.

The user interface 196 includes a display device such as a display. The display device is configured as a touch panel and also serves as an input unit. In this embodiment, the incoming call screen 300 and the banner notification 350 are displayed on the display device such as a display. Also, in this embodiment, a video call is made. At this time, the video data is displayed on the display device.

The microphone 197 and speaker 198 are used for calls. The speaker 198 is also used for ringing incoming calls, etc.

The memory unit 192 holds various types of data. The ROM 193 is a non-volatile storage device. For example, a semiconductor storage device with a relatively small capacity, such as a flash memory, is used for the ROM 193. The programs executed by the processor 191 can be stored in the memory unit 192 or the ROM 193. In this embodiment, the telephone directory 130 is stored in the memory unit 192 or the ROM 193.

The above program can be stored using various types of non-transitory computer-readable media and supplied to the mobile phone terminal 100. The non-transitory computer-readable media includes various types of tangible storage media. Examples of the non-transitory computer-readable media include magnetic recording media such as a flexible disk, a magnetic tape, or a hard disk, optical recording media such as an optical disk, an optical disk medium such as a compact disk (CD) or a digital versatile disk (DVD), and semiconductor memories such as a mask ROM, a programmable ROM (PROM), an erasable PROM (EPROM), a flash ROM, or a RAM. The program may also be supplied to the computer using various types of temporary computer-readable media. Examples of the temporary computer-readable media include an electric signal, an optical signal, and an electromagnetic wave. The temporary computer-readable media can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

RAM 194 is a volatile storage device. Various semiconductor memory devices such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory) are used for RAM 194. RAM 194 can be used as an internal buffer for temporarily storing data, etc. Processor 191 loads a program stored in storage unit 192 or ROM 193 into RAM 194 and executes it. By processor 191 executing the program, functions such as video call application 120 and CallKit 140 shown in FIG. 4 are realized.

As described above, according to this embodiment, the caller information image obtained from the preview video information 230 is used as the background image displayed on the incoming call screen 300. This allows the user to see the image included in the preview video information 230, i.e., a real-time image of the caller, during an incoming video call, and to know what the caller is up to. According to the method of this embodiment, even if the mobile phone terminal 100 uses iOS as its OS, the above video preview can be obtained while making use of existing functions.

For example, when used to contact medical institutions, the caller can see at a glance whether the nurse making the call has an emergency situation or a less urgent work-related matter, and the caller can respond appropriately.

Furthermore, according to this embodiment, the banner image 351 on the banner notification 350 is updated at a predetermined time interval while a call is coming in. This allows you to grasp the caller's information in real time.

In addition, according to this embodiment, during an incoming call, the audio data included in the preview video information 230 is converted to text and displayed as banner text 352 on the banner notification 350. This makes it easier to understand the state of the caller and allows for a more appropriate decision as to whether or not to respond. Generally, mobile phone terminals 100 that use iOS as their OS are unable to play audio along with the ringtone. However, according to this embodiment, the content of the audio data included in the preview video information 230 can be understood.

The mobile phone terminal 100 of this embodiment achieves these functions using a standard incoming call screen 390 that can be processed by the CallKit 140. Therefore, it is possible to provide more useful information while utilizing the various functions realized by the CallKit 140.

As described above, according to this embodiment, convenience is improved by utilizing available information when a video call is received on the mobile phone terminal 100.

In general, when a video call is received on the standard incoming call screen 390 of an iOS mobile phone terminal 100, the user is unable to see the other party's real-time status as a preview, which often leads to confusion about how to respond, hindering effective use of the video call function. According to this embodiment, by displaying a preview image of the other party and visualizing the voice content, not only does it greatly improve the usability of the video call function, but it also greatly contributes to differentiating solutions in linking nurse call and intercom products, whose main function is video preview, with mobile phone terminals whose OS is iOS. In addition to linking nurse call and intercom products with VoIP apps, the introduction of this embodiment significantly improves the usability of iOS mobile phone terminals 100 in any system that provides video incoming calls.

For example, this is particularly useful when a nurse in a hospital room wants to use a VoIP app to send a video call to a doctor in a meeting.

<<Second embodiment>>
Next, a second embodiment to which the present invention is applied will be described. In this embodiment, a banner notification 350 is utilized to enable a response by video call and a response by voice only call to be selectively used. The following description of this embodiment will focus on the differences from the first embodiment.

The functional configuration and hardware configuration of the mobile phone terminal 100 of this embodiment are the same as those of the first embodiment. However, in this embodiment, the banner notification control unit 125 displays a response button in addition to the banner image 351 and banner text 352 as the banner notification 350. Then, a response is accepted via the response button.

In this embodiment, the banner notification control unit 125 adds a response receiving unit 360 to the banner notification 350, sends it via the external I/F 160, and displays it in a predetermined area on the display screen.

An example of a banner notification 350 to which a response receiving unit 360 has been added is shown in FIG. 10. As shown in this figure, the response receiving unit 360 has, for example, an area for receiving an instruction to make a video call (video response), an area for receiving an instruction to make a voice call (voice response), and an area for receiving an instruction to reject a response. Note that the response receiving unit 360 may also have areas for receiving other responses and instructions.

When the banner notification control unit 125 receives an instruction from a user, it performs processing according to the instruction. For example, when an instruction for a video response is received, it instructs the call processing unit 170 (see FIG. 11) of the video call application 120 to make a video call. When an instruction for a voice response is received, it instructs the call processing unit 170 to make a voice call. When an instruction for rejection is received, it instructs the CallKit 140 to reject the incoming call.

As described above, in this embodiment, the banner notification control unit 125 adds a response receiving unit 360 that receives response instructions including at least a video response, an audio response, and a rejection to the banner notification 350 and sends it out. Then, instructions from the user are received via the response receiving unit 360.

Conventionally, the standard incoming call screen 390 only has two instruction acceptance buttons. In other words, for an incoming video call, it is only possible to answer or reject the video call, and for an incoming voice call, it is only possible to answer or reject the voice call. However, according to this embodiment, an area for accepting response instructions is provided in the banner notification 350. Then, it is possible to transition from the banner notification 350 to a VoIP app (video call app). Therefore, even if an incoming video call is received, it is possible to respond by making a voice call.

In addition to the same effects as the first embodiment, this embodiment can provide a variety of response options. For example, in a situation where a voice call would be possible, the only option would be to reject the call in the past, but since a voice call response is possible, the possibility of being able to respond increases. This increases convenience.

<Modification 1>
In the second embodiment, the method of providing various response options is not limited to the above. For example, as shown in Fig. 12(a) , a message 353 indicating that a voice response is possible may be displayed in a banner notification 350. For example, when there is no response for a certain period of time, the banner notification control unit 125 may control the banner notification 350 to display the message 353 instead of displaying the banner image 351 and the banner text 352. Alternatively, the normal banner notification 350 (banner image 351 and banner text 352) and the message 353 may be displayed alternately at a predetermined time interval.

Also, as shown in FIG. 12(b), in addition to the message 353, the response acceptance section 360 may be displayed.

Furthermore, as shown in Figures 12(c) and 12(d), when a voice response is possible, a message 353 may be displayed in the banner notification 350 without displaying the caller information image.

<Modification 2>
In the above-described embodiments, the banner notification 350 includes both the banner image 351 and the banner text 352. However, this is not limiting. The information displayed at a predetermined time interval as the banner notification 350 may be either one of the two. Also, a banner image combined with text information may be used as the banner notification.

<Modification 3>
In addition, in each of the above embodiments, the registration data generating unit 122 extracts the image in the first frame of the preview video information 230 as the caller information image, but is not limited to this. For example, a simple image analysis process may be performed to extract image data identified as an image of a person as the caller information image.

<Modification 4>
In addition, in each of the above embodiments, the mobile phone terminal 100 is a mobile phone terminal (for example, an iPhone manufactured by Apple) whose OS is iOS. However, the mobile phone terminal 100 is not limited to this. For example, the mobile phone terminal 100 may be an iPad (registered trademark) that uses iPadOS. In addition, the mobile phone terminal 100 may be another mobile phone terminal that has a standard incoming call screen that does not provide a video preview.

<Modification 5>
In addition, when the mobile phone terminal 100 has a plurality of ring tones, a ring tone may be set for each caller number (telephone number 131). In this case, for example, the telephone directory 130 further has a setting field for setting a ring tone, and the ring tone is set in advance in the setting field.

The background color of the image display area 320, the display color of the banner notification 350, etc. may be changed for each caller number (telephone number 131). In this case, too, the telephone directory 130 is provided with a setting field for setting various colors, and the colors are set in advance in the setting field.

In the flowchart used in the above explanation, multiple steps (processes) are described in order, but the order in which each step is performed is not limited to the order described. For example, the order of the steps shown in the figure can be changed to the extent that does not interfere with the content, such as performing each process in parallel.

The programs stored in the storage unit 192 of the mobile phone terminal 100 can be provided as program products recorded on a non-transitory computer-readable storage medium. These can be used to store various programs recorded on a non-transitory computer-readable storage medium for the medium to long term.

Although each embodiment of the present invention has been described above, the present invention is not limited to the above-mentioned embodiments, and further modifications, substitutions, and adjustments can be made without departing from the basic technical concept of the present invention. For example, the network configurations and configurations of each element shown in each drawing are examples to aid in understanding the present invention, and are not limited to the configurations shown in these drawings.

Some or all of the above-described embodiments and modified examples can be described as in the following supplementary forms, but are not limited to the following.
(Appendix 1)
an incoming call signal receiving unit for receiving an incoming call signal including a caller number, caller information, and preview video information when a video call is received;
an incoming call screen display unit that displays an incoming call screen including the caller number, the caller information, and a caller information image extracted from the preview video information;
a banner notification sending section for sending a banner notification including a sender information image extracted from the preview video information at regular intervals.
(Appendix 2)
In the mobile terminal device according to claim 1,
It is preferable that the banner notification sending unit obtains text information from the preview video information by speech recognition, and adds the obtained text information to the banner notification and sends it.
(Appendix 3)
In the mobile terminal device according to claim 1 or 2,
It is preferable that the banner notification sending unit sends out a response means including at least a video response, an audio response, and a rejection, together with the banner notification.
(Appendix 4)
In the mobile terminal device according to claim 2,
It is preferable that the banner notification sending unit adjusts the size of the display area of the banner notification or the size of the displayed text according to the amount of the text information.
(Appendix 5)
In the mobile terminal device according to claim 2,
It is preferable that the banner notification sending unit synthesizes the text information with the sender information image and sends out the banner notification.
(Appendix 6)
In the mobile terminal device according to any one of Supplementary Note 1 to 5,
It is preferable that the banner notification sending section sends out the banner notification together with information on possible response means.
(Appendix 7)
In the mobile terminal device according to any one of Supplementary Notes 1 to 6,
The function of the incoming call screen display unit is preferably provided as part of the function of a framework that provides a function for managing conflicts between applications that use the call function and a function for enabling incoming calls during security lock.
(Appendix 8)
An incoming call screen display method executed by a mobile terminal device, comprising:
When a video call is received, an incoming call signal including a caller number, caller information, and preview video information is received;
An incoming call screen display method that displays an incoming call screen including the caller number, the caller information, and a caller information image extracted from the preview video information, and transmits and displays a banner notification including the caller information image extracted from the preview video information at regular intervals.
(Appendix 9)
Computer,
means for receiving an incoming call signal including a caller number, caller information, and preview video information when a video call is received;
means for displaying an incoming call screen including the caller number, the caller information, and a caller information image extracted from the preview video information;
a program for functioning as a means for transmitting and displaying a banner notification including a sender information image extracted from the preview video information at regular intervals;
In addition, each of the embodiments of Supplementary Notes 8 and 9 can be expanded into the embodiments of Supplementary Notes 2-7, similar to the embodiment of Supplementary Note 1.

The disclosures of the above patent documents and the like are incorporated herein by reference. Within the framework of the entire disclosure of the present invention (including the scope of the claims), and further based on the basic technical ideas, modifications and adjustments of the embodiments and examples are possible. Furthermore, within the framework of the disclosure of the present invention, various combinations and selections of the various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art would be able to make in accordance with the entire disclosure, including the scope of the claims, and the technical ideas. In particular, with regard to the numerical ranges described in this document, any numerical value or subrange included within the range should be interpreted as being specifically described, even if not otherwise specified.

100: mobile phone terminal, 100a: caller terminal, 100b: receiver terminal, 101: telephone system, 110: OS, 120: video call application, 121: incoming call signal receiving unit, 122: registration data generating unit, 123: phone book operation unit, 124: CallKit operation unit, 125: banner notification control unit, 126: timer, 130: phone book, 131: phone number, 132: name, 133: image, 134: save area, 135: remarks, 140: CallKit, 150: NW I/F, 160: external I/F, 170: call processing unit, 191: processor, 192: storage unit, 193: ROM, 194: RAM, 195: communication I/F, 196: user interface, 197: microphone, 198: speaker,
200: incoming call signal, 210: caller number, 220: caller information, 230: preview video information,
300: incoming call screen, 310: text display area, 311: caller number, 320: image display area, 321: caller information image, 330: instruction display area, 350: banner notification, 351: banner image, 352: banner text, 353: message, 360: response acceptance section, 390: standard incoming call screen, 391: text display area, 392: image display area, 393: instruction display area,
800: mobile terminal device, 810: incoming call signal receiving unit, 820: telephone directory registration unit, 830: incoming call screen display unit, 840: banner notification sending unit, 850: telephone directory,
900: network, 910: relay server, 920: voice recognition server

Claims (9)

an incoming call signal receiving unit for receiving an incoming call signal including a caller number, caller information, and preview video information when a video call is received;
an incoming call screen display unit that displays an incoming call screen including the caller number, the caller information, and a caller information image extracted from the preview video information;
a banner notification sending section for sending a banner notification including a sender information image extracted from the preview video information at regular intervals. 2. The mobile terminal device according to claim 1,
The banner notification sending unit obtains text information from the preview video information by speech recognition, and adds the obtained text information to the banner notification and sends it. 2. The mobile terminal device according to claim 1,
The banner notification sending unit sends the banner notification by adding a response means including at least a video response, a voice response, and a rejection. 3. The mobile terminal device according to claim 2,
A mobile terminal device, wherein the banner notification sending unit adjusts the size of a display area of the banner notification or the size of displayed text according to the amount of the text information. 3. The mobile terminal device according to claim 2,
The banner notification sending unit combines the text information with the sender information image and sends out the banner notification. 2. The mobile terminal device according to claim 1,
The banner notification sending unit adds information on possible response means to the banner notification and sends it. 2. The mobile terminal device according to claim 1,
A mobile terminal device in which the function of the incoming call screen display unit is provided as part of the function of a framework that provides a function for managing conflicts between applications that use call functions and a function for enabling incoming calls while security is locked. An incoming call screen display method executed by a mobile terminal device, comprising:
When a video call is received, an incoming call signal including a caller number, caller information, and preview video information is received;
An incoming call screen display method that displays an incoming call screen including the caller number, the caller information, and a caller information image extracted from the preview video information, and transmits and displays a banner notification including the caller information image extracted from the preview video information at regular intervals. Computer,
means for receiving an incoming call signal including a caller number, caller information, and preview video information when a video call is received;
means for displaying an incoming call screen including the caller number, the caller information, and a caller information image extracted from the preview video information;
a program for functioning as a means for transmitting and displaying a banner notification including a sender information image extracted from the preview video information at regular intervals;

Images