FI115277B - Arrangement of motion observation in mobile station - Google Patents

Description

11527 "

Providing motion detection on a mobile station

Field of the Invention

The invention relates to controlling a mobile station equipped with a camera, and in particular to providing motion detection in a mobile station operatively connected to the camera.

Background of the Invention

The importance of images in telecommunication devices has increased. Images taken with a digital camera using a mobile device may have been transmitted, for example, using e-mail to other communication devices. Data transmission standards that are more suitable for data transfer have been further developed. In particular, packet data transmission, such as General Packet Radio Service (GPRS), enhances data transmission over a mobile network. A multimedia messaging service (MMS) has also been developed for transferring messages containing images to and from mobile stations. There are also 15 mobile devices with a camera integrated in the mobile device. One device that combines mobile communication functionality with a camera is a wireless surveillance camera that can be configured to take pictures at specific time intervals and send them via MMS to a specified mobile station or email address. The camera also has a motion detector, which captures an image and transmits it when the motion detector detects motion. This type of camera can easily be set up to monitor, for example, a summer cottage and send a picture of a burglar in the cottage. The problem with known devices combining mobile functionality with a camera is that motion detection is implemented in infrared technology! | nilkalla. Infrared detection is quite unreliable; Even with certain clothing, motion detection can be prevented if the thermal power required by the infrared '· · ·' sensor is not radiated through the clothing. Also, no movement is detected through the glass.

I I *

I »I

. Brief Description of the Invention. It is an object of the invention to provide a method and apparatus for implementing the method so as to improve the functionality of motion detection in a mobile communication environment. The object of the invention is achieved by a method, a mobile station and a computer program product which are characterized by what is stated in the independent claims. Preferred embodiments of the invention are claimed in the dependent claims.

115277 2

The invention is based on the fact that in a mobile station having a functionally connected camera, motion is monitored based on a digital evaluation of differences between the images taken by the camera. The camera is controlled to determine the pixel values of the pixels to be tracked at at least two different time points to form at least two images. Differences between images are determined and goodwill is created based on differences between images. Goodwill is generally taken to mean a value derived from differences in the image information of any comparable image. The goodwill is compared to a predetermined threshold and the mobile station initiates at least one supervised operation if the goodwill reaches -10.

An advantage of the arrangement according to the invention is that the detection of motion is made more reliable. The solution according to the invention is completely different from the previous devices combining the camera and the mobile functionality, because in the solution the camera can repeatedly determine the image information from which the motion is directly evaluated. In this case, no separate sensor sensors are required to trigger the capture. Digital analysis of sequential images can very accurately detect motion in the imaging area. The mobile station according to the invention is very easy to arrange for a temporary monitoring need, since the mobile station which is otherwise included can provide the monitoring 20 independently and completely wirelessly. With motion detection based on the image information provided by the camera according to the invention, it is also possible for V to respond even less when motion is detected. Since the detection is not based on thermal radiation, non-radiant heat is also detected; If you do, the motion can also be tracked through the glass, i.e. the camera may be protected from the weather, the camera may be covered or moved and no errors will occur due to heat flows (eg when the electric heater is turned on). Because infrared sensors do not use general curtain detection technology, which requires a moving object to pass through one or more devices. through the plane or segment formed by the tea, there are no sectors or areas in the control area where motion detection is not possible, and motion is detected as quickly near and far.

According to a preferred embodiment of the invention, the mobile station has a sensitivity adjustment, whereby the threshold used for comparing the goodwill can be changed according to the situation. According to yet another embodiment 35, a sensitivity adjustment window is provided to the user which illustrates at least continuously comparing the difference between successive image tracking values with the goodwill to be updated on the basis of 115277 3 and said threshold value. The user is provided with the option of setting said threshold and setting said threshold based on input from the user. Sensitivity control can take into account different environments and the quality of motion monitored, and thus further improves motion detection reliability.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be further described in connection with preferred embodiments, with reference to the accompanying drawings, in which:

Figure 1 illustrates a mobile station with a camera attached thereto;

Figure 2 illustrates a method according to a preferred embodiment of the invention; and

Figure 3 illustrates a method for detecting motion according to a preferred embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

Figure 1 illustrates a mobile station MS with camera functionality CA. The mobile station generally refers to a device comprising means for implementing wireless communication. Such devices include devices capable of communicating with a PLMN, such as GSM / GPRS or a third 20-generation network (e.g., 3GPP). It is to be noted that the invention can also be applied to marine mobile stations in which the camera is not an integrated mobile station. the communication device, but the camera providing the image information is connected by a wired; or your mobile wireless device.

The mobile station MS comprises memory MEM, UI, I / O means for providing external communication, and a central processing unit (CPU) comprising one or more processors.

,: · The MEM has a non-volatile portion for storing applications and other data to be stored by the central processing unit CPU, and a volatile portion for use in temporary data processing. The mobile station MS illustrated in Figure 2 also comprises a camera CA which can operate what; 'Using any known digital imaging technique and convert •, f; in the left sensor to electrical charge the light, assign RGB values to the pixels, and convert them to digital. Note that in addition to still images 35, the camera CA may also be able to record video. The processing, such as compression, and storage of the image information in digital format 115277 4 is performed by the central processing unit CPU.

According to a preferred embodiment of the present invention, the mobile station MS executes a surveillance application MA which detects motion in the camera imaging area based on the image information received from the camera and provides a function associated with the surveillance application MA, for example an alarm. The computer program codes executed in the central processing unit CPU can provide the mobile station MS to implement a monitoring application MA, some embodiments of which are illustrated in Figures 2 and 10 3. The functions of the monitoring application MA may be divided into several separate and communicating software components. The computer program may be stored on any memory medium, such as a PC hard disk or a CD-ROM, from where it can be loaded into the memory MEM of the executing device CS. The computer program can also be downloaded over the network using, for example, 15 TCP / IP protocol stacks. It is also possible to use March solutions or a combination of hard and software solutions to implement the inventive tools.

Figure 2 illustrates a preferred method for providing a monitoring application MA. The monitoring application MA can be activated when the 20 mobile stations MS are placed in the correct location for monitoring. According to a preferred embodiment of the invention, the display of the mobile station MS can be utilized to find the correct imaging area, i.e. the picture is updated according to the position of the mobile station MS. This embodiment facilitates the knowledge; redirection. The monitoring can be started at the time set by the timer: \ j 25 or start at the time specified in the settings. : After this period, it may continue for the set time or until the monitoring is interrupted. According to one embodiment, the monitoring can be initiated or uninterrupted

also roams a short message SMS from another terminal

. , (Short Message Service). When the MS is activated in the mobile station; 30 watt, the camera CA is directed to 201 to capture images at a specified time interval, i.e., · · * to determine pixel values of pixels to be tracked and to transmit

· ·· They are digitally processed by the monitoring application. Camera in CA

: "is preferably controlled to use only a portion of the pixels used to take a typical image. The image may be smaller or the pixels of the images may be rasterized so that the image area is not reduced. Alternatively, the control application MA limits the number of pixels to be compared. 115277 5, the image-limiting embodiment can limit the need for processing capacity and thus also the battery capacity, which is important in mobile stations. Motion detection is also made faster when only a fraction of the available pixels are compared. Pictures are taken at the frame rate set by the mobile station 5 ) The frame rate can preferably be selected from the monitor application settings, for example, one of three options of 0.5, 1, and 10 seconds, etc. Sufficient frame rate is determined by, among other things, the speed of the motion being tracked. experiment with the adequacy of the selected frame rate and puts the frame at 10 frames in context.

Based on the image information 202 received from the camera CA, differences between successive images 203 are determined and a goodwill 204 is generated based on the differences, illustrating the magnitude of change between images. The differences between the images can be generated directly from uncompressed image information (pixel values) obtained from the camera CA or from compressed image information. Next, 205 goodwill is compared to a predetermined threshold and it is checked whether the goodwill reaches the threshold.

If the goodwill does not reach the 205 threshold, monitoring continues and proceeds to step 202 to wait for pixel values in the next image, which 20 can be compared to the previous image in step 203. If the goodwill 205 reaches the threshold, the mobile station MS 206 initiates In step 206, one or more of the following operations may be performed:! Motion can be used to guide the camera to take a final * ·, · 25 image, typically using a larger number of pixels than: just for motion detection. Preferably, pixel values of pixels corresponding to the selected image size and quality in the settings are received from the camera CA and typically the image is compressed into a JPEG format, for example.

When motion is detected, pictures can be taken according to the settings. If travel; : The 30 message MS is arranged to capture images at a sufficient density, there are - · 'multiple images of the motion trigger and a moving image can be generated. Phase: ·: Pixel values already optionally already set for monitoring are used (saving and / or transmitting the image), making the image even more relevant.

In step 206, the captured image may be stored by the mobile station MS;:; 35 to MEM. The settings may be set to a certain number of images to be • saved: or images may be stored until the memory becomes full.

115277 6

The captured image may be transmitted to another communication device using a data transmission service supported by the mobile station MS and the mobile communication network serving the mobile station MS. The image can be sent, for example, using MMS or as an e-mail. The mobile station may be connected to the network and send the image immediately after it has been taken. When using a circuit switched service, the mobile station makes a call to a specified number to which the image is transferred. The mobile station MS may have an address of another communications device, such as an IP address, to which the image is assigned. Instead of transmitting over a mobile network, the image may be transmitted to the other device 10 in other ways, for example, via short-range radio. The image may be transmitted, for example, to another user's terminal or control system device according to the monitoring application's connection settings.

Based on motion recognition and / or image capture, a notification may be sent to another communication device. As with the transmission of the image, the notification can be transmitted using any service provided by the mobile stations and the associated mobile network. A particularly effective way to send a notification is to use a short message, SMS, which is immediately forwarded to the specified number. If the number cannot be contacted, the short message is stored in the mobile station's network and forwarded to said number as soon as the terminal identified by the number is connected to the network. On the basis of the notification received, the user or, for example, a security guard, receives very quickly: j: information on the detected movement in the controlled area.

In step 206, a local alarm may be performed, whereupon the mobile station may play a strong audio alarm and / or use other effects available in the mobile station MS, such as screen blinking.

According to a preferred embodiment of the invention, the sensitivity of motion detection can be adjusted by changing the threshold value. The user can set the threshold himself or select one of the preset defaults.

': / * 30 There are several default values that can be set so that motion detection is sufficiently sensitive. The UI can be used to display a sensitivity adjustment • window showing real-time goodwill and set threshold. This sensitivity adjustment window makes it easy for the user to set the correct level of motion detection, ie the threshold, for different states and situations. For example, flicker in fluorescent lamps causes movement to be equated with; changes in the image information, whereby the threshold must be large enough to prevent this kind of noise from causing false motion detection. The window can display not only the current goodwill, but also minimum and maximum values for the monitoring time as bars and numeric values to further facilitate the setting of the correct threshold. The values may be scaled, for example, to a range of 0-100 to facilitate operation. An alternative to implementing a sensitivity control window is to plot a graph of goodwill over time. According to one embodiment, the sensitivity (threshold) adjustment can also be performed automatically by determining a sufficient threshold value of the goodwill measured during the adjustment time. This embodiment further facilitates the use of the device. The threshold adjustment 10 is preferably performed prior to the start of monitoring, whereby the adequacy of the threshold can be ascertained. Thus, for example, a threshold can be set so that the movement of the pet does not cause step 206 to be performed. Once the user has adjusted the threshold (in step 201), the monitoring application MA preferably waits for a predetermined delay time before monitoring begins and the camera-15 is directed to take pictures. There are also several preselection threshold options for monitoring application MA, such as sensitive, normal, and coarse. The user can directly select one of the presets, which further facilitates the start of the MA application.

Preferably, the MS also has a twilight function that takes into account the effect of light. In motion detection, the luminance value of the pixels can also be calculated from the RGB values of the pixels used to determine the sensitivity of the camera CA used, such as day or night. The luminance number can be calculated as

• as an average of the RGB values of the pixels in the image area. Camera CA.

: For example, three brightness settings can be set for 25 sensitivity settings, ·. · Degrees, dark, normal and bright, the middle of which acts as a hysterical region.

*% 'Camera CA is set to night position when the luminance value is in the dark area for a predetermined 1' ', and back to day position when the luminance level is in the bright area for a predefined period. Thus, the mobile station can also control the camera i 30 CA during tracking based on the image information.

According to a preferred embodiment, the mobile station MS,; maintaining a history record, log, of observations made during the tracking period. The log can record the time at which the movement occurred and other information such as maximum and minimum values.

Figure 3 illustrates a motion detection method that may also be applied to the monitoring application illustrated in connection with Figure 2 above. When the pixel values to be followed by 301 new images are received from the camera CA, the pixel value pairs An; Bn and An + 1; Bn + 1 of the new image, called image B, and the previous image, called image A, are determined. n and n + 1 represent successive pixels, so that in pixel value pairs An; Bn and An + 1; Bn + 1 represent the same consecutive pixels of two consecutive images. Differences between the pixel value pairs are determined 303, and then 304 is checked to see if there is a difference above the first threshold set in the pixel value pairs. The differences between the pixel pairs can be summed and the sum value compared to the first threshold value. According to a preferred embodiment, the difference detected in a pair of pixels is compared one pixel at a time to the first threshold value and both pairs of pixels must have a difference above the first threshold in order to proceed from step 304 to step 305. This embodiment neglects a single pixel. Preferably, the first threshold value is set by default in the monitoring application MA so that the pixel difference must be large enough.

If there is a difference in pairs of pixel values above the first threshold value, the value of the counter maintained in method 305 is incremented. If there is not a sufficient difference, proceed directly to step 306, where it is checked whether all the monitored pixel value pairs have been processed. If not all pixel values have been processed, then return to step 302 to determine the following pixel value pairs (An + 1; Bn + 1 and An + 2; Bn + 2).

After the inspection of step 306 has processed all the pairs of pixel values set to be tracked, a final counter value is obtained which illustrates. the amount of motion detected in the image pair. Then, it is possible to proceed to step 307,. : 25 which deals with the resulting final counter value. The counter value can be processed to achieve the desired sensitivity to the amount of movement. For this weighting, you can use a mathematical weighting function or retrieve a weighted value corresponding to the counter value from a pre-stored weighting table. For example, weighting can highlight the effect of a small movement: 30 changes in the amount of movement (goodwill). One way is to apply a logarithmic curve. As previously described, the user can: display a motion calibration window, thereby improving the visibility. For example, the counter value processed can be scaled to a value between 0 and 100 (which sometimes also sets the goodwill threshold). From the processed counter value, 35 is dipped 308 to see if it reaches a preset second threshold value. If the processed counter value does not exceed the second threshold value, no significant movement has occurred between the 115277 9 examined sequential images, whereby the counter is reset and can be processed to pixel values of the new received image to step 301. If the processed counter value reaches another threshold, already illustrated above with reference to Figure 2.

It is noted that step 307 is not necessary, but step 308 can directly compare the resulting counter value with a set second threshold value. When comparing FIG. 3 to FIG. 2, the processed counter value corresponds to the goodwill of FIG. 2 and the second threshold value used in step 308 for the motion detection used in step 205 of FIG. 2. The difference between the pixel value pairs determines the goodwill (increments the counter and processes the resulting counter value).

The advantage of this method is that sufficiently reliable and responsive motion detection can be implemented on devices with limited processing capacity. The number of pixel values to be tracked for motion detection can be adapted to the available processing capacity, taking into account battery capacity as well. In the motion detection method of Figure 3, pairs of pixels were used for comparison, any combination of close pixels can be used for comparison, or individual pixels can be directly compared. However, by comparing pixel pairs, it is possible to filter out random pixel noise.

: According to a preferred embodiment of the invention, the level of control; ,: The MA application tracks movement, and if a significant change occurs in movement,. : activates a function associated with the monitoring. This embodiment can be implemented using the embodiments already described above in connection with Figs. 2 and 3, since motion differences are also detected on the basis of the differences in the images. | reduction. However, when the difference exceeds a threshold value, the movement is reduced or stopped and, for example, a notification is sent to another communication device.

According to yet another preferred embodiment of the invention, the mobile communication monitoring application MA may also indicate that no movement has been detected. In this case, the MA is preferably set to monitor the motion for a time interval selected by the user, and if the motion is not detected in the time interval, for example, it sends a notification to another communication device.

Some exemplary situations in which the solution of the present invention could be applied are described below. The user can set their mobile 10 115277 mobile to follow the Garage at. 23.00, after which the mobile station sends a short message SMS to the other mobile station of the detected movement. The mobile station can be set in the hotel room towards the door, whereby the mobile station triggers a loud alarm if there is movement in the direction of the door.

5 The mobile station can be left in a convenient location when leaving home. The monitoring function of the mobile station can be activated for the short message SMS, after which the mobile station transmits an image if motion is detected. Some other example situations where the solution can be applied are access control and nature monitoring. According to a preferred embodiment, user profiles can be defined for these different states, for example, with suitable thresholds. The user can directly select a context profile based on the context, the settings of which are set in the monitoring application when starting MA monitoring.

It will be obvious to a person skilled in the art that as technology advances, the basic idea of the invention can be implemented in many different ways. The invention and its embodiments are thus not limited to the examples described above, but may vary within the scope of the claims.

1 '| 1 ·

Claims (10)

115277 A method for providing motion detection in a mobile device operatively connected to a camera, the method 5 controlling the camera to determine pixel values of the pixels to be tracked at at least two different times to form at least two images, determining image differences, generating goodwill based on image differences, and 10 initiating in the mobile station at least one control associated function in response to the goodwill reaching said threshold value, characterized by setting said threshold value by means of a sensitivity adjustment function. The method of claim 1, characterized by determining at least two pairs of pixel values in the images, wherein the pixel value pair comprises pixel values of corresponding pixels of the two images being compared, comparing differences between pixels in each pixel value pair, increasing counter value in response to adding a predetermined first threshold value, weighting the counter value in response to having processed all pixel value pairs of pixels to be monitored, checking if the weighted counter value exceeds a predetermined second threshold value, and starting at the specified second threshold. A method according to claim 1 or 2, characterized in that said function comprises taking a picture, wherein the camera is controlled to determine the pixel values set for taking the picture. A method according to any one of the preceding claims, characterized in that said function comprises storing an image, wherein: at least one image is stored in the memory of the mobile station. A method according to any one of the preceding claims, characterized in that said function comprises transmitting an image, wherein: 115277 is transmitting the image to at least one of the predetermined communication devices in response to recording the image. A method according to any one of the preceding claims, characterized in that said function comprises transmitting a notification, wherein: the notification is transmitted to at least one communication device in response to the image recording. A method according to any one of the preceding claims, characterized in that the user is presented with a sensitivity adjustment window which illustrates at least continuously the goodwill and said threshold value updated based on a comparison of differences between successive image tracking values, providing the user with a set threshold value; 15. 8. A mobile station operatively connected to a camera, characterized in that the mobile station is arranged to direct the camera to determine the pixel values of the pixels to be tracked at at least two different times to produce at least two images, the mobile station being arranged to determine based on retrieved pixel values, the mobile station is arranged to generate goodwill image differences; on the basis of i, 25 mobile stations are arranged to compare the goodwill with a predetermined threshold value, the mobile station is arranged to initiate at least one monitoring associated function in response to the goodwill reaching a predetermined threshold, and: 30 mobile stations further comprise a sensitivity control said threshold. A mobile station according to claim 8, characterized in that, *, said function is at least one of the following: audible alarm, new call; 35277, or sending a notification to at least one communication device. 10. A computer program product for controlling a mobile station, characterized in that the computer program product part comprises: 5 a computer program code portion for controlling a mobile station a computer program code portion for controlling a mobile station to generate goodwill based on image differences, a computer program code portion for controlling a mobile station to compare a goodwill with a predetermined threshold value, 15 a computer program code portion for controlling a mobile station to initiate at least one n a sensitivity adjustment function for setting said threshold. • · »•» · • 1 • »« il • »· • · • • • •» »1 *» · * I * ·. »· <I · * · - 'I 1» •> i »* \ \ 115277