JP2005151019A - Portable terminal, its control method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enhance usability by solving a problem of the lowering of use feeling of a visitor using a mounted user interface. <P>SOLUTION: A camera state change is noticed to a camera controller using any one of the vibrator drive function, the illumination function or the voice output function of a portable telephone client 1-4 depending on the control behavior of a camera 1-3 on a network. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a portable terminal device including at least one of a vibrator driving unit, an illumination driving unit, and an audio output unit, and capable of controlling a camera connected via a network, a method for controlling the portable terminal device, and a program. It is.

  Conventionally, surveillance camera systems based on analog lines have been installed in banks, convenience stores, supermarkets, jewelry stores, stores in all industries such as consumer finance to prevent theft, theft and shoplifting. However, in recent years, remote camera systems based on digital signals such as TCP / IP have been put into practical use for transmission of image information in surveillance camera systems based on such analog lines. Since such a camera is a system based on a global protocol called TCP / IP, it is possible to use a global digital network such as the Internet as a global network, and to view images from a remote location. It is also possible to do it. Furthermore, broadcast live broadcasting is possible by opening these systems to the public, and it has been put to practical use for such purposes. In general, such a camera system is a network camera.

  Among these network cameras, a system that can not only observe images remotely but also control cameras such as pan, tilt, and zoom by controlling the camera platform has been put into practical use. ing.

  This type of network camera may be viewed simultaneously by a large number of people and at the same time, the camera may be controlled by the large number of people.

  The video can be viewed by a large number of people at the same time. However, with regard to camera control, if a large number of people control at the same time, a control conflict occurs, so that everyone cannot perform the intended camera control. In view of this, a system has been put into practical use in which a specific person for a certain period of time is given the right to allow camera control, and the right is distributed to the controller in turn. This system introduces the concept of camera control rights and assigns camera control rights only to specific people, allowing camera control to only one specific person and performing normal camera control. A viewer who wants to perform camera control requests a camera control right from the camera server, and can acquire the camera control right and operate the camera for a certain period of time. When someone has already acquired the camera control right, a mechanism is introduced in which the request for the camera control right is queued and the control right is allocated in order according to the queue every time a certain period of time elapses. Camera control is assigned to each viewer for a certain period of time, so that the camera can be controlled normally by a large number of viewers.

On the other hand, in recent years, mobile devices such as mobile phones are provided with a WWW browsing function and a mail transmission / reception function typified by iMode, and protocols such as TCP / IP and HTTP can be used. For this reason, video and digital information can be transmitted and received even with a mobile phone. A system for browsing the network camera with a mobile phone using these has been put into practical use (see Patent Document 1).
JP 2003-299069 A

  Mobile phones in recent years have been equipped with a variety of user interface devices that have not existed before. As an example, a vibrator is installed in a mobile phone, and a notification function by vibration, an illumination function for blinking lights such as LEDs, etc. are installed in addition to a display for viewing a phone number, World Wide Web, and an image. .

  Although these mobile phones have become highly functional in this way, when these mobile phones are used as clients for browsing a network camera, the environment is higher than when using a normal PC or a high-speed LAN. There is a difference and there is a problem.

  For example, mobile phones use relatively low-frequency radio compared to high-speed network connections such as fixed line connections, so the line speed is very slow and the server response to client requests is slow. Communication delay occurs.

  In addition, since the operation of one camera is shared by a large number of people, there is a possibility of waiting very much when acquiring the control right of the camera when the number of viewers is very large. This has also caused a significant decrease in the user experience.

  In particular, notifications such as acquisition of camera control rights that are required during camera control force the viewer waiting for acquisition of the camera control rights to wait. An operation in which the viewer gazes at a display that displays information and waits for these notifications by effectively notifying the viewer of the change by using a function installed in a mobile phone such as a vibrator. It becomes possible to release from.

  In order to solve such problems, the present invention uses a built-in user interface to solve a decrease in the user's feeling of use and improve usability, and control of the mobile terminal device For method and program purposes.

  In order to achieve the above object, a mobile terminal device of the present invention includes at least one of a vibrator driving unit, an illumination driving unit, and an audio output unit, and can control a camera connected via a network. And a notification means for notifying a camera controller of a camera state change using any one of the vibrator driving means, the illumination means, and the audio output means according to the behavior of camera control on the network. It is characterized by that.

  In addition, the mobile terminal device control method according to the present invention includes at least one of a vibrator drive unit, an illumination drive unit, and an audio output unit, and is a control method for a mobile terminal device capable of controlling a camera connected via a network. And according to the behavior of camera control on the network, the camera controller is notified of a camera state change using any one of the vibrator driving means, the illumination means, and the audio output means. To do.

  The program of the present invention includes at least one of a vibrator driving unit, an illumination driving unit, and an audio output unit, and executes a method for controlling a portable terminal device capable of controlling a camera connected via a network. A program that notifies a camera controller of a camera state change using one of the vibrator driving unit, the illumination unit, and the audio output unit according to the behavior of camera control on the network. And

  According to the present invention, in response to a change in the state of the camera control right, the camera controller can recognize the change in the state of the camera control right by the mobile phone client by a method other than staring at the image. It is not necessary to stare at the image just to confirm the acquisition of the camera, and the convenience of the camera controller can be improved.

  Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
In the first embodiment, for the behavior of the network camera server, the response state on the server side is determined on the client side according to the response speed of the behavior (time is set in advance in the mobile phone client). It is characterized by controlling the vibrator and illumination. This embodiment will be described with reference to the drawings.

  FIG. 1 is a configuration diagram showing the configuration of the entire system. As a whole, this system includes a camera device 1-3 to which a camera platform device is connected, a video server 1-2 that transfers video of the video device to a mobile phone client, and a camera control server 1-2 that controls the camera platform. A mobile phone client 1-4 that receives and displays those images and provides an interface that enables camera control, and a network 1-1 that connects them.

  The network 1-1 used here is a TCP / IP network used in the Internet. The physical configuration of this network may be a fixed line or wireless. A logical protocol other than TCP / IP (in the case of IPX / SPX or a cellular phone, a provider-specific network protocol) may be used. Since there are a wide variety of networks, in this embodiment, the network is hereinafter treated as an abstract digital signal communication network.

  The camera device 1-3 includes a camera device unit that is a part that captures video and a pan head device unit that directs the camera device unit in various directions. These two devices can receive information from the camera control server and perform operations such as pan / tilt / zoom. The captured video is sent to a video server, captured as digital data, and subjected to processing such as compression.

  Reference numeral 1-2 denotes a video server and a camera control server. The video server captures a captured video using an image from the camera device as an input source, digitizes the video, and compresses the image data. Then, the image data is transmitted to the mobile phone client having a video acquisition request.

  The camera control server receives a camera control right acquisition request and a camera control request from the connected mobile phone client. Then, the concept of camera control right is introduced, camera control is permitted only to a specific viewer, and the camera control information is notified to the pan head apparatus.

  There are examples in which the camera device, the pan head device, the video server, and the camera control server configured by 1-3 and 1-2 are put into practical use as one device, and need not be physically separated. In addition, there is an example in which video processing is not performed using an analog method but is completely processed using a digital method, but either one may be used in the present embodiment.

  1-4 indicates a mobile phone client. A mobile phone is composed of a device that constitutes a normal computer such as a central processing unit, and a mobile phone that provides a platform for running software has been put into practical use. These are called i-applis and are realized by providing an execution environment for a program language such as Java (registered trademark). Any software execution environment may be used as long as it can perform the same function.

  The mobile phone client can communicate with the network. In this embodiment, TCP / IP and the HTTP protocol that is an upper layer protocol thereof are used. However, any network protocol may be used as long as the same function can be achieved.

  FIG. 2 shows a use case of the system proposed in this embodiment. Reference numerals 2-1 and 2-2 denote image receivers who are viewers using the mobile phone client. These requests generate three use cases for the network camera system: image acquisition 2-4 and camera control request 2-5 and resolution 2-5 of the camera control right conflict associated therewith.

  In particular, since the competition between the camera control request and the camera control right occurs at the same time, the use case has a dependency relationship. For this reason, the present embodiment provides a system that solves two scenarios, namely, two scenarios, that is, video acquisition and camera control request (camera control right conflict).

  FIG. 3 shows a collaboration diagram regarding the acquisition of video in the two scenarios shown in FIG. Reference numerals 3-1 and 3-2 denote image recipients. These image receivers irregularly request (1) video requests from the video server indicated by 3-3. If there is no latest image data in response to these requests, the video server makes a (2) video acquisition request to the camera device shown in 3-4. Thereafter, the video server performs video acquisition of the camera device ((3) video acquisition). Further, the video server compresses the captured digital image data ((4) video image compression). Then, image data is transmitted to all mobile phone clients that are requesting video at this time ((5) video delivery). Upon receiving this video data, the mobile phone client displays (6) the captured video on the screen. The scenario of video acquisition can be achieved by such message notification and data transmission procedure.

  FIG. 4 shows a collaboration diagram, particularly regarding the camera control request (camera control rights conflict) in the two scenarios shown in FIG. Reference numerals 4-1 and 4-2 denote image receivers (camera controllers). Reference numeral 4-3 denotes a camera control server. And 4-4 has shown the pan head apparatus controlled with respect to the camera server.

  The scenario for solving the camera control request (camera control right conflict) is slightly more complicated than the scenario of video acquisition. The acquired video server video can be shared from all mobile phone clients at the same time, but the operation of the camera platform controlled by the camera control server cannot be shared at the same time ( Exclusive network resources). Therefore, in the present embodiment, this is solved by using the concept of camera control rights. The ownership of the camera control right at a certain moment is limited to one mobile phone client.

  Therefore, (1) when a request for camera control is generated from the mobile phone client, the camera control server performs the queuing process of the request for the camera control right in the order of the camera control right. Then, (3) a camera control right is returned to the mobile phone client that has requested all the camera control rights. This response is the earliest notification to the mobile phone client that requested the camera control right that the camera control right was successfully owned, and the other mobile phone clients were unsuccessfully owned by the camera control right. , The mobile phone client is notified of information on how long to wait ((4) camera control right notification).

  The mobile phone client notifies the camera control right by controlling the vibrator or illumination according to the response of the camera control right.

  The mobile phone client that has successfully acquired the camera control right receives the camera control input from the camera controller ((5) camera control), and sends a camera control request to the camera control server ((6) camera control information transmission). .

  This camera control information includes information such as pan / tilt / zoom, aperture control, gain control, filter control, preset control, piper control, and deflow control, which are general operation parameters for the camera. Then, the camera control server (7) receives the camera control information, creates the camera platform control information based on the camera control information, and (9) transmits the camera platform control signal to the camera platform. The target (10) pan head control is performed. Such a message notification or data transmission procedure makes it possible to achieve a scenario of a camera control request (camera control right conflict).

  FIG. 5 shows an internal configuration diagram of the video server and the camera control server. As shown in 5-2, the video server and the camera control server include an image output device, an input device, a central processing unit, a main storage device, an external storage device, an image input device, an external device signal transmission / reception device, a network interface, etc. These devices are combined by a bus line. This configuration is not much different from a general computer.

  As an external device, a display device indicated by 5-1 is connected to the image output device. Further, the image input device is connected to a camera device indicated by 5-3 so that an image can be input. In the present embodiment, an analog image signal is particularly premised, but the present invention is not limited to this, and a digital image signal may be used. In that case, the image input device is changed to match the signal. Further, the pan / tilt head device 5-3 is connected to the external device signal transmitting / receiving device, and the entire camera can be controlled by this connection. In the present embodiment, a signal line generally called RS232C is used. However, any interface that can perform the same function may be replaced.

  Transmission / reception of image information and transmission / reception of camera control information can be performed with a mobile phone client by connecting to an external TCP / IP network 5-4 using a network interface. As described above, any digital network may be used as this network. Since the video server and the camera control server are not significantly different in hardware configuration, the software may be executed on the same hardware, or the software may be configured on different hardware. I do not care.

  FIG. 6 shows an internal configuration diagram of the mobile phone client. If the mobile phone client is a communication device that can communicate with digital communication and World Wide Web or TCP / IP network, there is no particular problem even if it is replaced with a PDA or the like.

  The configuration of the mobile phone client 6-2 is basically the same as that of a normal computer. An image display device 6-4, an input device, a central processing unit, a main storage device, a vibrator driving device, an illumination control device 6-3, a wireless information transmitting / receiving device 6-1 and the like are included. A characteristic part of this embodiment is that a vibrator drive device and an illumination control device are mounted on the mobile phone client. By using such a device, it is possible to provide a user interface that improves usability for camera control using a mobile phone as a client.

  In addition, this mobile phone client can connect to the TCP / IP network 6-5 using a wireless information transmitting / receiving device. This network connection is often connected to the Internet or the like via a mobile phone company network, and there are few examples of direct connection. However, the logical configuration may be handled as being directly connected, and in the present embodiment, the description will be made on the assumption that the logical connection is made. These networks may be protocols other than TCP / IP as long as they are logical digital network connections, and may be in any physical form such as fixed lines and wireless.

  FIG. 7 shows a class configuration diagram of software inside the video server. Reference numerals 7-1, 7-2, 7-3, 7-5, and 7-6 denote classes and interfaces constituting the video server. All classes inherit the interface indicated by 7-2 and are used for transferring image data between classes.

  7-3 shows a client class. This class abstracts the mobile phone client and communicates with the mobile phone client. Further, a request from the mobile phone client is a message notification in the video server in the form of a notification from this class. The client class is instantiated in a one-to-one relationship with the mobile phone client. Therefore, a data distribution class is prepared to manage these objects efficiently. There is only one instance of this class, and all the client classes are registered in the data distribution class, so that image data can be transferred to each client class. The data distribution class collectively receives requests from the client class, and requests image data from the video compression class. The video compression class further makes the request to the video acquisition class. The video acquisition class captures video from the camera device 7-1 and creates digital image data of the video to be captured. This image data is notified to the video compression class. Image data is compressed in the video compression class. As this data compression algorithm, any algorithm can be used, and in this embodiment, the Jpeg format is used. After the image data is compressed, the compressed image data is notified to the data distribution class, and the image data is notified to the client class that requests the image. The video server can be operated by such a software configuration.

  FIG. 8 shows a collaboration diagram inside the video server. Furthermore, an object diagram of the video server is also shown, showing an internal object configuration. Reference numerals 8-3, 8-4, 8-5, 8-6, and 8-7 denote objects constituting the video server. Each of these objects represents an instantiated object of the class shown in FIG.

  By the operation of the image receiver in 8-2, the mobile phone client makes a request for (1) video to the video server. This request is received by the client object instantiated in a one-to-one relationship with the mobile phone client inside the video server. The client object notifies the data distribution object of the request (2) a video request. These video requests are managed by the data distribution object, and when the data distribution object determines that it is necessary to acquire the latest updated image, (3) a video request is made to the video compression object. Further, the video compression object notifies the video acquisition object of (4) video request. The video acquisition object that has received the notification of the video request performs (5) video acquisition initialization for the camera device 8-2. When initialization for video acquisition is completed, the video acquisition object actually (6) acquires video. Next, the video acquisition object notifies the video compression object of the captured image data (7). When receiving the image data, the video compression object compresses the image data (8), and notifies the compressed image data to the data distribution object (9). Next, the data distribution object that has received the video data notifies the requested client object of the video data (10). Finally, each client object receives the compressed image data. Send (11) to each mobile phone client. Thus, the image data obtained by the camera device can be displayed on the mobile phone client (12). With such an internal configuration, it is possible to realize the operation of the video server.

  FIG. 9 shows a class configuration diagram of software inside the camera control server.

  Reference numerals 9-1, 9-5, 9-6, 9-7, 9-8, 9-3, and 9-9 denote classes and interfaces constituting the camera control server. The client class is a class instantiated in a one-to-one relationship with the mobile phone client. Therefore, the mobile phone client is abstracted in the camera control server, and communication with the actual mobile phone client and requests from the mobile phone client are made through this class.

  These communication requests are all managed by the client manager class. The client manager class receives information such as camera control request from the client class, determines the necessity for notification of camera status changes to the client class, etc. Distribution.

  As interfaces, a camera control right status notification interface 9-6 and a camera control status notification interface 9-7 are defined. These interfaces are interfaces used by the camera control authority class and the camera control class to communicate with the client manager class. For this reason, the client manager class inherits this interface. Requests for camera control rights compiled from the client manager class are notified to camera control rights classes and camera control classes, respectively.

  The camera control right class is a class indicating a camera control right. For this reason, it holds which client class the current camera control right is in, the validity period of the camera control right, a list of requesters of the camera control right, and the like. Although the owner of the camera control right changes depending on the time, the elapse of time serving as a reference is notified to the camera control right class by the timer class 9-9. The change in the state of the camera control right is notified to the client manager through the camera control right state notification interface.

  The camera control class is a class that controls the camera platform device of the camera device. Therefore, the camera control class holds the state of the pan head device. Further, in response to a camera control request from the client manager, a camera control signal is actually sent to the pan head apparatus to perform arbitrary camera control. Further, the change of the state of the pan head apparatus is notified to the client manager by the camera control state notification interface. These client control class and camera control class state changes are received by the client manager, and camera-initiated changes are notified to each client class. The camera control server can be operated with such a software configuration.

  FIG. 10 shows a collaboration diagram inside the camera control server.

  Further, an object diagram of the camera control server is also shown, showing an internal object configuration. Reference numerals 10-1, 10-2, 10-5, 10-6, 10-7, and 10-8 denote objects constituting the camera control server. Each of these objects represents an instantiated object of the class shown in FIG.

  In the camera control server, the client object receives the camera control right request (1) from the mobile phone client. The request for the camera control right is notified to the client manager (2). Each mobile phone client's request will be posted to one client manager object. The camera control right request is requested (3) from the client manager object to the camera control right object. The camera control right object queues the camera control right request from the client manager object in a queue in the camera control right object, and organizes and grasps the camera control right request from each client object. Furthermore, the state of the camera control right is determined from the time elapsed (5) from the timer object. The state of the camera control right holds information of only one client that can operate the camera and other clients that wait for the camera control right, and at the same time, how long this state lasts. This change in camera control status is notified to the client manager (6). Receiving this, the client manager notifies the corresponding client object of the status of the camera control right (7). Further, the client object transmits (8) the status of the camera control right to the mobile phone client. Thus, the mobile phone client can display (9) the camera control right state.

  Since the mobile phone client holding the camera control right can perform camera control, when camera control is performed by the camera controller, the camera control information is transmitted to the client object (9). The camera control information is notified from the client object to the client manager object (10), and further notified to the camera control object (11). Then, the camera control object creates a control signal for the pan head apparatus 10-9 based on the camera control information, and notifies the pan head apparatus of the signal (12). Thus, camera control information from the mobile phone client is executed (13) by the pan head apparatus. In addition, the camera control object monitors changes in the state of the pan / tilt head device, and monitors changes in the state (13). If there is a change in the state of the pan / tilt head device, the camera control object notifies the client object of camera control state information (13). Upon receiving this notification, the client object notifies each client object of the information (14), and finally notifies the mobile phone client of information on the change in the camera control state (15). In this way, the camera control state can be displayed (16) by the mobile phone client.

  FIG. 11 shows a class diagram of the mobile phone client. 11-1, 11-2, 11-3, 11-4, 11-11, 11-12, 11-2, 11-5, 11-6, and 11-9 respectively indicate classes in the mobile phone client. Yes.

  The camera image display class is a class that holds the state of the image display device of the mobile phone client. Therefore, the camera image display class has an image display method and can display an image of the video server. The key input class is a class for monitoring input of a numeric key of a mobile phone client. Simultaneously with the input, the contents are notified to the camera control server class or the like. The camera control state display class is a class for displaying the state of the pan / tilt head device. In this embodiment, the camera control state display class is a class for displaying states such as pan / tilt / zoom. The camera control right state display class is a class for displaying the state of the camera control right on the image display device of the mobile phone client.

  The vibrator operation class is a class for controlling the vibrator driving device in the mobile phone client. The illumination operation class is a class for controlling the illumination device in the mobile phone client. The camera control right display manager class is a class for uniformly managing the UI of the camera control right. This class manages and manages images, vibrators, illuminations, etc. to notify the presence or absence of camera control rights. The timer class is a class for managing the passage of time in the mobile phone client.

  The video server class is a class that actually communicates with the video server and simultaneously holds the status of the video server. This class has a video request method and video reception event. When video data is actually received, it operates in the mobile phone client as a notification from this class. The camera control server class is a class that actually communicates with the camera control server and holds the state of the camera control server. It has methods and events related to camera control such as request and notification of camera control right and request and notification of camera control, and operates as a camera control class in the mobile phone client. In the mobile phone client, this embodiment is realized by using such a software configuration.

  FIG. 12 shows a collaboration diagram of the mobile phone client. This figure also shows an object diagram of the mobile phone client. 12-10, 12-4, 12-3, 12-6, 12-7, 12-8, 12-9, 12-10, and 12-11 respectively indicate objects in the mobile phone client. 11 is an instance of each class. In the mobile phone client, the video system and the camera control system operate independently.

  The video system is a relatively simple operation, and the video server object always first sends a video request (1) to the video server. On the other hand, when the video server can acquire the latest image data, the video server object receives (2) the image data. After the reception, the video data is decompressed in the video server object, and finally the image is displayed (3) on the camera image display object. By performing this cycle many times, it is possible to display a moving image.

  On the other hand, with respect to camera control, first, the camera controller 12-5 performs a camera control right request operation (1). This input is notified to the key input object, further to the camera control server object (2), and notified to the camera control server object. The camera control server object transmits this camera control right request to the camera control server. Thereafter, the camera control server transmits a response (4) regarding the acquisition contents of the camera control right, and the camera control server object receives the information. Information on the camera control right is notified (5) to the camera control right display manager object. This object controls the display and behavior of the camera control right in response to the time notification (6) from the timer object. This control method is an invention part of the present embodiment in the present proposal. The camera control right display object, the vibrator operation object, the illumination operation object, and the like are controlled according to the length of time until the camera control right is acquired (6) (7) (8). If the camera control right can be acquired, the UI for camera control is permitted, and the camera controller operates the UI that actually instructs camera control (10). The input is notified (11) to the camera control server object through the key input object. Further, the camera control server object transmits the camera control request to the camera control server (12), and the camera control server actually executes the pan head control. The change in the state of the camera platform at that time is notified (13) to the camera control server object. Further, the camera control information is notified to the camera control state display object, whereby the camera control state is displayed on the image. Thus, the present embodiment is executed by the object operating in the mobile phone client.

  FIG. 13 shows the state transition of the video server. The video server basically holds four states 13-1, 13-2, 13-3, and 13-4. The initialization state 13-1 shows the initialization state from the startup state of the video server. If the initialization of the video server is successful, the process proceeds to the image acquisition state 13-2. In the image acquisition state, an image is captured. When the image is captured normally, the image compression state 13-3 is entered.

  In the image compression state, the obtained video data is being compressed. When the image compression is normally performed, the image distribution state 13-4 is entered. In this state transition, when there is no client connection, the subsequent video distribution becomes meaningless. In this case, the state returns to the image acquisition state. In the image distribution state, the obtained image data is actually transmitted to each client. When the transmission of the latest image data to each client is completed, the state returns to the image acquisition state again, and thereafter such state transition is repeated.

  FIG. 14 shows the state transition of the camera control server. In particular, this figure shows the state transition of the client object in the camera control server and the state transition of the camera control right object. The client object holds the states 14-2, 14-3, and 14-4. 14-2 represents a state without camera control right. In this state, there is no request for the camera control right from the mobile phone client, and the camera control right is not assigned to the mobile phone client. When the camera operator receives a camera control right acquisition operation communication, the camera operator enters a camera control right waiting state. This state is when the control right is not assigned to the client object by the camera control right notification from the camera control right object. Furthermore, when a certain amount of time has passed or the client holding the camera control right is disconnected, the state of the camera control right object changes, and when the control right is assigned to the client object, the camera control is performed. It becomes possible. In this state, camera control is permitted, and the camera control request from the mobile phone client is valid.

  Further, 14-5, 14-6, and 14-7 show state transition diagrams of the camera control right object.

  The camera control right is generated from a state without a camera control right holder. In this state, there is no request for camera control right from any client object. If at least one camera control right request is received from the client object in this state, the state shifts to the camera control right holder present state. In this state, only camera control from the client object that requested the camera control right is valid. From this state, if there is a request for the camera control right from another client object, the state shifts to the camera control right holding company & camera control right waiting state. In this state, the camera control is enabled only for the camera holding the camera control right first, and all other camera controls are prohibited. In addition, when the mobile phone client is disconnected or when a certain period of time elapses when the camera control right holder is present, the camera control right holder is present, and the camera control right waiter is present, the camera is controlled accordingly. The state of the rights object changes. By having such a state transition, the concept of camera control right is realized.

  FIG. 15 shows a video-related state transition diagram of the mobile phone client.

  It is the video server object in the mobile phone client that holds this state. The actual display is performed by the camera image display object.

  Reference numerals 15-1, 15-2, 15-3, and 15-4 denote states of the mobile phone client. The mobile phone client maintains the initial state after startup. If the connection with the server is successful from this state, an image request to the video server is transmitted and at the same time an image acquisition wait state is entered. In this state, when image data is received from the video server, the image development state is entered. If the image data is further successfully developed, it is displayed on the image, and the image display state is entered. Then, the image request is transmitted again to the video server, and at the same time, the state shifts to an image acquisition waiting state. After this, this cycle is repeated many times. It is possible to display the image of the video server while repeating such state transitions.

  FIG. 16 shows a state transition diagram related to camera control of the mobile phone client. Reference numerals 16-2, 16-3, 16-4, 16-6, 16-7, and 16-5 denote states of the mobile phone client.

  After being activated, the mobile phone client is activated in a state without the camera control right. Thereafter, upon receiving a camera control right request operation from the camera controller, a camera control right request is made to the camera control server, and at the same time, the camera control right standby state is entered. In this state, the control right notification is received from the camera control server, and the next state change is determined by the passage of time when the camera control right is acquired. This branch of the state change is an invention point in the present embodiment.

  A threshold value n is set in advance for the mobile phone client. The threshold value n may be set by the camera controller or may be set in the system by default. The state transition destination is changed according to the camera control right state response timing of the camera control server from the camera control right standby state start time when the control right is requested. If the camera control right is immediately obtained, the camera control state display state is entered, and the camera control state is immediately entered. In this way, when the camera control right can be acquired without any problem, the camera shifts to a camera controllable state without showing a particularly large reaction.

  If the camera control right can be acquired after n seconds, the camera control right display state & vibrator operation state is selected in which sound, illumination, or vibrator is selected and notified. At this time, the sound, illumination, and vibrator operations may all be performed simultaneously or selectively. Then, the camera control is performed by shifting to a camera controllable state. If the camera control right cannot be obtained, the state shifts to the camera control non-acquisition display & illumination operation state in which one of sound, illumination, and vibrator is selected and notified. At this time, the sound, illumination, and vibrator operations may all be performed simultaneously or selectively. And it transfers to a state without a camera control right.

  In addition, it is possible to make different patterns for the operation of the vibrator, sound and illumination in the camera control right status display & rainy season operation state and the camera control right non-acquisition & vibrator operation state. As described above, it is possible to improve the convenience of the camera controller by changing the behavior of the mobile phone client according to the time taken to notify the acquisition of the control right from the camera control server.

  FIG. 17 shows a sequence diagram related to video distribution. Reference numerals 17-1 to 17-30 denote the contents of the operation sequences. When the image receiver first makes a connection request to the mobile phone client, the key input object is detected in 17-2 and this content is transmitted to the video server object. At 17-31, the video server object attempts to connect. If the connection fails, the sequence ends at 17-6. If successful, an image request is made to the video server in 17-7.

  On the other hand, the video server side initializes the client object by 17-3 and enters the client waiting state. At the same time, the data distribution object, the video compression object, and the video acquisition object are initialized.

  The client object receives the video request in 17-7, receives the content in 17-8, and notifies the data distribution object. Further, the data distribution object notifies the video compression object 17-21, and finally the video acquisition object is notified of the image acquisition request from the mobile phone client. Upon receiving this notification, the video acquisition object acquires an image of the camera device in 17-22 and notifies the video compression object of the image data. The video compression object compresses the image data in 17-27 and notifies the data distribution object of the compressed data. The data distribution object that has received the image data determines whether each client object needs the latest acquired image data. If there is no request for image data, the process returns to 17-20, and the image acquisition and compression of the latest camera device is performed again. If there is a client object requesting distribution of image data, the video data is notified to the client object in 17-26. Upon receiving the notification, the client object transmits video data to the mobile phone client at 17-25. The mobile phone client receives the image data at 17-24, decodes the video data at 17-28, notifies the image data to the camera image display object at 17-29, and receives the image data at 17-23. Is displayed. In this way, it is possible to display the image of the video server. Thereafter, the process returns to 17-7, and it is possible to repeatedly receive image data by repeating this cycle thereafter.

  FIG. 18 shows a sequence diagram regarding camera control. Reference numerals 18-1 to 18-32 denote the contents of the operation sequences, respectively. When the camera controller first makes a connection request to the mobile phone client, the key input object of the mobile phone client recognizes the operation 18-1. The key input object notifies the camera control server object of connection to the camera control server, and the camera control server object connects to the camera control server in 18-2.

  The camera control server detects this connection in the client object 18-3. In the camera control server, connection information is further notified to the client manager class and registered in the holding list of the client that is currently connected to the camera control server. If the connection is successful, the mobile phone client is ready to accept a camera control right request input 18-5. When there is a camera control right request operation from the camera controller from this state, the key input object transmits a request for acquiring the camera control right to the camera control server object 18-10. Further, at this time, a camera control right request is also notified to the camera control right display manager object 18-9.

  Then, the camera control right display manager notifies the camera control right display object of the current state of the camera control right and displays it 18-8.At 18-10, the request for the camera control right is transmitted to the camera control server, The client object receives the request and is further notified to the client manager object 18-11. The client manager object issues a camera control right request 18-16 to the camera control right object and notifies the camera control object to obtain the current camera state 18-15. At this time, the camera control right object changes the state such as the owner of the camera control right state, the waiting time of the waiting queue, the camera control right occupation time based on the timing notification 18-6 from the timer object. The status change is notified to the client manager object within 18-16.

  In this way, the client manager object can grasp the camera control right state and the camera control state. Necessary information is provided and notified to each registered client object 18-17. Upon receiving this notification, the client object transmits the camera control right state and the camera control state to the mobile phone client, and the camera control server object of the mobile phone client receives 18-18. The contents of this reception are notified to the camera control state display object and the camera control right state object, and the current camera control right and the state of the pan head apparatus are displayed on the mobile phone client. At this time, it is also sent to the camera control right display manager.

  At this time, if the user has the camera control right, the request for the camera control right is permitted from the camera controller, and the request from the key input object is received 18-22. The camera control request is transmitted 18-24 by the camera control server object, and is notified to the camera control object 18-24, 18-25, 18-26 via the client object and client manager object of the camera control server. Upon receiving this notification, the camera control object performs camera operations such as pan / tilt / zoom on the pan / tilt head device 18-27. Then, the camera client manager is notified of the change in the state of the pan head device which has been changed again at 18-28, and the camera control state is transmitted to the mobile phone client. In this way, camera control can be realized.

  If the camera control right cannot be acquired even in 18-23, the control waits for the control right from the camera control server in 18-30 and waits. When the camera control right can be acquired during this period, if the threshold value n seconds previously set in the mobile phone client is not exceeded, the control is transferred to 18-32. If the threshold value n is exceeded, The control is moved to 18-37, the vibrator is operated by the vibrator operation control object, the camera control right is displayed in 18-36, and the process proceeds to 18-22.

  Even if the camera control right cannot be acquired, the vibrator is operated at 18-33, and the state of the camera control right is displayed at 18-32. At this time, the operation of the vibrator may be performed by changing the illumination, or may be notified by sound. Using such a sequence, the mobile phone client can efficiently notify the camera controller, who is the user of the mobile phone client, of the camera control right.

  FIG. 19 shows the GUI of the mobile phone client. Reference numerals 19-2, 19-4, 19-5, 19-6, 19-3, and 19-7 indicate image transitions with respect to respective states of the mobile phone client.

  The first state is the state (A), and it is possible to input the connection destination of the video server in this state. In the present embodiment, selection can be made by a selection method, but an address or the like may be input. When the input is completed, the state (B) is entered. This state is a state in which connection to the server is performed. When the connection is successful and image acquisition is completed, the state (C) is entered. In this state, the video of the video server is displayed, but the control right is not acquired. From this state, when the camera controller requests a control right, the state (D) is entered. In the state (D), the camera control right is waiting from the camera control server. If the camera control right can be acquired and the camera can be controlled, the state (E) is entered. In this state, camera control is actually permitted. If the camera control right cannot be acquired, the state shifts to (F). In this state, a UI such as a vibrator, sound, or illumination is used to notify that the camera control right cannot be acquired. By implementing such a UI, it is possible to improve user convenience for waiting for camera control rights.

  As described above, in this embodiment, the response on the server side is determined on the client side according to the response speed of the network camera server (the time is set in advance in the mobile phone client). Thus, it becomes possible to control the vibrator and illumination, and to notify the camera controller of a change in the behavior of the efficient camera control right.

(Second embodiment)
Unlike the case of the first embodiment, the client side uniquely sets a timeout for the behavior on the server side, and notifies the camera control with a vibrator, illumination, etc. regarding the timeout.

  In the case of the first embodiment, the mobile phone client also makes a request for the camera control right in the case of the first embodiment by measuring the acquisition time until the camera control right in 18-39 of FIG.

If the camera server's response to the camera control right request immediately obtains the control right, nothing is done. If the camera control right can be acquired within the threshold of n seconds, the camera controller is notified of the use of the camera control right using sound, light, vibrator, or the like. When the control ticket could not be acquired, the logic was to notify the camera controller that the camera control right could not be acquired using sound, light, vibrator, etc.
In the second embodiment, the camera control right is requested first. If the camera server's response to the camera control right request immediately obtains the control right, nothing is done. If the camera control right can be acquired within the threshold of n seconds, the camera controller is notified of the use of the camera control right using sound, light, vibrator, or the like. If the server response exceeds the threshold of n seconds or if the control right cannot be acquired, the camera controller is informed that the camera control right cannot be acquired using sound, light, vibrator, etc. Make a notification. In the second embodiment, the counter that measures the passage of time at this time is basically based on the camera control server state change notification from the camera control server in the first embodiment. The behavior of the server is judged on the client side, and the timer is supported on the client side. That is, the operation of the vibrator or the like is determined under the initiative of the mobile phone client.

  As described above, even if the mobile phone client is changed as in the second embodiment, it is possible to notify the camera controller of a change in the behavior of the camera control right efficiently.

System Configuration Use case diagram System collaboration diagram (video) System collaboration diagram (camera control) Server internal configuration diagram Client internal configuration diagram Video server class diagram Video server collaboration diagram (class) Camera control server class diagram Camera control server collaboration diagram (class) Cell phone class diagram Mobile phone collaboration diagram (class) Video server state transition diagram Camera control server state transition diagram Cell phone state transition diagram (video) Cell phone state transition diagram (camera control system) Video delivery sequence diagram Camera control sequence diagram GUI diagram (example)

Explanation of symbols

1-1 TCP / IP network 1-2 Video server / camera control server 1-3 Camera device 1-4 Mobile phone client

Claims (5)

A portable terminal device comprising at least one of a vibrator driving means, an illumination driving means, and an audio output means, and capable of controlling a camera connected via a network,
According to the camera control behavior on the network, it has a notification means for notifying the camera controller of the camera state change using any of the vibrator driving means, the illumination means, and the audio output means. Mobile terminal device. The camera can be controlled by a mobile terminal device connected to a plurality of mobile terminal devices and having a camera control right,
The portable terminal device according to claim 1, wherein the notification unit notifies a camera controller of a change in the camera control right. It further has a detecting means for detecting a state change of the camera control right,
The portable terminal device according to claim 2, wherein the notification unit notifies the camera controller of a change in the camera control right in accordance with a detection result of the detection unit. A mobile terminal device control method comprising at least one of a vibrator drive means, an illumination drive means, and an audio output means, and capable of controlling a camera connected via a network,
A mobile terminal device that notifies a camera controller of a camera state change using any one of the vibrator driving unit, the illumination unit, and the audio output unit according to the behavior of camera control on a network. Control method. A program for executing a method for controlling a portable terminal device that includes at least one of a vibrator driving unit, an illumination driving unit, and an audio output unit and is capable of controlling a camera connected via a network,
A program for notifying a camera controller of a camera state change using any one of the vibrator driving unit, the illumination unit, and the audio output unit according to the behavior of camera control on the network.

Images