KR20240027223A - Power saving method of portable electronic devices equipped with user walking accident prevention application - Google Patents

Abstract

The control method of a portable electronic device of the present invention includes step (a) of determining whether the tilt of the portable electronic device obtained by the posture detection unit falls within a preset tilt range, and detecting the position if the tilt falls within the tilt range. Step (b) of determining whether or not the user is walking based on a negative detection value, and (c) of detecting whether the display provided on the front of the portable electronic device is operating when it is determined that the person is walking in step (b); , If it is determined that the display is running in step (c), the user's gaze is detected based on the image acquired by the front camera, and the surrounding image is acquired by driving the rear camera based on the detection result. It includes a step (d) of detecting a walking caution object by inputting the surrounding image acquired through the rear camera into an artificial neural network learned based on the walking caution object learning data.

Description

Power saving method of portable electronic devices equipped with user walking accident prevention application}

The present invention relates to a method for reducing power consumption of a portable electronic device equipped with a pedestrian caution notification application.

In modern society, the number of people moving around while looking at their smartphones is increasing. As a result, a new word called smombie (a combination of the words smartphone and zombie) was coined. Additionally, side effects of smombie include collisions, falls, and traffic accidents.

Several studies are being conducted to prevent such accidents. For example, there is a method of installing a ramp on the side edge of a staircase and installing a pad step on the entrance side of the staircase, so that when a pedestrian steps on the pad step, the lamp operates to make the pedestrian aware that there is a staircase. However, this method not only has the possibility of not recognizing the ramp when a pedestrian is concentrating on a smartphone, but also has the problem of being expensive to install.

As another example, when a user reaches a crosswalk, a device installed at a traffic light or the like notifies the user with a voice that there is a crosswalk or uses a lamp to draw attention to the user. In this way, the pedestrian's attention is not directly alerted by what he or she is currently focusing on (i.e., the smart phone), but the pedestrian's attention is simply aroused in an indirect way through surrounding objects outside of the focus, so the pedestrian walking while concentrating on the smartphone Accidents due to this are continuing.

The problem that the present invention aims to solve is to detect pedestrian caution objects by inputting images acquired through a rear camera placed on the back of a portable electronic device into an artificial neural network learned based on walking caution object learning data, but to detect pedestrian caution objects. The present invention provides a control method for a portable electronic device that reduces unnecessary power consumption by ensuring that the rear camera for water detection is operated only when the user is looking at the screen of the portable electronic device while walking.

The control method of a portable electronic device of the present invention includes step (a) of determining whether the tilt of the portable electronic device obtained by the posture detection unit falls within a preset tilt range, and detecting the position if the tilt falls within the tilt range. Step (b) of determining whether or not the user is walking based on a negative detection value, and (c) of detecting whether the display provided on the front of the portable electronic device is operating when it is determined that the person is walking in step (b); , If it is determined that the display is running in step (c), the user's gaze is detected based on the image acquired by the front camera, and the surrounding image is acquired by driving the rear camera based on the detection result. It includes a step (d) of detecting a walking caution object by inputting the surrounding image acquired through the rear camera into an artificial neural network learned based on the walking caution object learning data.

The inclination range in step (a) may be defined as an inclination of 15 to 60 degrees with respect to the horizontal of the portable electronic device.

Step (b) includes calculating the moving speed based on the detection value of the position detection unit, and determining that the person is walking when the moving speed falls within a predetermined walking speed range defined between 1 and 7 km/h. May include steps.

Gaze detection in step (d) may include detecting the user's pupils in the image acquired through the front camera.

A computer-readable recording medium according to another aspect of the present invention records a program for executing the above method in a portable electronic device.

The control method of a portable electronic device of the present invention uses the rear camera to obtain images necessary for detecting pedestrian objects only when it is determined that the user is walking while looking at the screen displayed on the front of the portable electronic device, thereby reducing power consumption. Can be managed effectively.

In particular, there is an effect of saving approximately 30% of power compared to the case where pedestrian detection is always performed through the rear camera while the screen is output through the display.

In addition, safety accidents while walking can be prevented because the camera automatically detects surrounding pedestrian cautions and notifies the user of them. In particular, the detection of pedestrian cautions is achieved with very high accuracy by a previously learned artificial neural network. There is.

Figure 1 is a configuration diagram of a pedestrian caution notification system applied to a control method of a portable electronic device according to an embodiment of the present invention.
Figure 2 is a block diagram of a portable electronic device to which a control method according to an embodiment of the present invention is applied.
Figure 3 is a diagram showing a pedestrian caution notification application running on a portable electronic device.
Figure 4 is a flowchart showing the process of learning pedestrian cautions in the pedestrian caution notification method.
Figure 5 is a flowchart showing a pedestrian warning notification method.
Figure 6 is a bounding box loss graph verifying the learning results of the neural network.
Figure 7 is a class accuracy graph verifying the learning results of the neural network.
Figure 8 shows the results of detecting pedestrian caution objects through an artificial neural network.
Figure 9 is a structural diagram of the MobileNet standard model (MobileNet Body Architecture).
Figure 10 is a structural diagram of a pedestrian warning object detection model.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to be understood by those skilled in the art. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

1 is a configuration diagram of a pedestrian caution notification system applied to a control method of a portable electronic device according to an embodiment of the present invention. Figure 2 is a block diagram of a portable electronic device to which a control method according to an embodiment of the present invention is applied.

Referring to Figures 1 and 2, the pedestrian caution notification system 1 according to an embodiment of the present invention includes a portable electronic device 10 and a learning server that communicates with the portable electronic device 10 through a communication network 20. (30) may be included.

Portable electronic devices 10 include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, and slate PCs. , it may be a concept that includes a tablet PC, an ultrabook, etc.

The portable electronic device 10 includes a processor 11, a memory 12, an input unit 13, a communication unit 15, a display 16, a front camera 17, a rear camera 18, and an audio output unit 19. , may include a position detection unit 21 and/or a posture detection unit 22.

The memory 12 can store algorithms, applications, or programs for the operation of the processor 11, and can also temporarily store input/output data (for example, phone books, messages, still images, videos, etc.).

An artificial neural network for detecting pedestrian cautions may be stored in the memory 12. However, the artificial neural network is not limited to this, and may be linked with the electronic device 10 while stored in web storage on the Internet.

The memory 12 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), and RAM. (random access memory; RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic It may include at least one type of storage medium among disks and optical disks.

The input unit 13 generates input data in response to a control command for controlling the operation of the portable electronic device 10 issued by the user. The input unit 13 may be composed of a key pad, a dome switch, a touch pad (static pressure/electrostatic), a jog wheel, a jog switch, etc.

The communication unit 15 transmits and receives information through the communication network 20, thereby enabling communication with other devices (eg, learning server 30) connected to the communication network 20. The communication unit 15 may include at least one of a mobile communication module, a wireless Internet module, and a short-range communication module.

The display 16 displays (outputs) information processed by the electronic device 10. For example, a user interface (UI) or a graphic user interface (GUI) may be displayed on the display 16. The display 16 may form the front of the electronic device 10.

The display 16 may be a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), or a flexible display. ), a 3D display, and an electronic ink display (e-ink display).

Some of these displays may be transparent or light-transmissive so that the outside can be viewed through them. This may be called a transparent display, and representative examples of the transparent display include TOLED (Transparant OLED).

The front camera 17 may include an image sensor that acquires a digital still image or moving image, and the image acquired through the image sensor may be stored in the memory 12. The front camera 17 can acquire images in the direction the screen of the display 16 faces. The processor 11 can process the image and display it through the display 16. The front camera 18 may form the front of the electronic device 10.

The location detection unit 21 is used to obtain the location (or current location) of the portable electronic device 10. The location detection unit 21 may include at least one of a Global Positioning System (GPS) module and a Wireless Fidelity (WiFi) module. The GPS module can acquire the location of the portable electronic device 10 using signals sent from GPS satellites. Alternatively, the location of the portable electronic device 10 may be obtained based on information from the Wi-Fi module and a wireless AP (Wireless Access Point) that transmits or receives wireless signals.

The processor 11 can obtain the moving speed of the portable electronic device 10 based on the location information detected through the location detection unit 21. For example, the processor 11 may determine the moving speed of the portable electronic device 10 based on the satellite signal acquired by the GPS module.

The posture detection unit 22 detects the posture of the portable electronic device 10. For example, the posture detection unit 22 may detect the angle at which the display 16 is tilted relative to the horizontal (i.e., inclination).

The posture sensor 22 may include at least one of a magnetic sensor, a gravity sensor, a gyroscope sensor, a motion sensor, and an acceleration sensor. there is. For example, the acceleration sensor can measure gravity acting in the x, y, and z axes, and using the measured value, the processor 11 determines the angle at which the portable electronic device 10 is tilted with respect to the horizontal ( That is, the slope) can be obtained.

The rear camera 18 may include an image sensor that acquires a digital still image or moving image, and the image acquired through the image sensor may be stored in the memory 12. The processor 11 can process the image and display it through the display 16. The rear camera 18 may configure the rear of the electronic device 10.

The audio output unit 19 may output audio data received from the communication unit 15 or stored in the memory 12. The sound output unit 19 also outputs sound signals related to functions performed by the electronic device 10 (eg, call signal reception sound, message reception sound, etc.). In particular, when a pedestrian caution object is detected through an artificial neural network, the sound output unit 19 may output a notification sound notifying the user of this. This sound output unit 19 may include a receiver, speaker, buzzer, etc.

Meanwhile, the learning server 30 is a machine learning device that learns walking precautions. The learning server 30 may be a learning model based on an artificial neural network (ANN).

The artificial neural network is preferably a convolutional neural network (CNN), but is not necessarily limited thereto. As is already well known, a convolutional neural network is a deep neural network mainly used in image data processing, and can extract complex features by reducing the bias of input data using a convolution layer and a pooling layer.

In CNN, the input layer is followed by convolution, relu, and pooling layers, followed by a fully connected layer, and finally, the result is output through a softmax layer. The convolution filter is sequentially applied to each part of the image as the inner product. Because this has the advantage of sharing weights and reducing the size of input data.

CNN shows excellent performance in image search and classification, but it uses a lot of resources during training. Therefore, there is no problem in operating it on a deep learning machine with excellent computational performance, for example, the learning server 30.

However, in order to apply the artificial neural network to the portable electronic device 10 in a mobile environment, the parameters must be reduced due to limitations in supported resources, so it can be applied only by reducing it to a minimum, and performance must also be maintained.

In this respect, a Mobilenet-CNN model with a non-linear placement standard and ReLU or Leaky ReLU function applied behind layers except the last fully connected layer (FC Layer) is applied to detect pedestrian objects for each frame. It can be configured to do so.

While the screen is displayed through the display 16 disposed on the front of the smartphone 10, an image may be acquired by the rear camera 18 disposed on the rear of the smartphone 10. That is, while the user is walking while looking at the screen, the rear camera 18 acquires images around the user.

Considering the general form of walking while holding the smartphone 10 with one hand and looking at the screen with one's head down, it can be said that the image acquired by the rear camera 18 is usually acquired from the floor in front of the pedestrian.

The processor 11 inputs the image acquired by the rear camera 18 into an artificial neural network and detects pedestrian cautions (e.g., stairs, crosswalks, traffic lights, street trees, sidewalk boundaries, etc.) from the image. . This walking caution detection process is carried out in the background process of the program driven by the processor 11, and at this time, the working process (or foreground process) displays a predetermined screen on the display 16. can be assigned. Here, the screen displayed on the display 16 may be a separate application (eg, messenger, web browser, game, etc.) that is currently running.

The operation of the rear camera 18 to detect pedestrian objects can preferably be performed while the user is walking while a predetermined screen is displayed on the display 16.

Meanwhile, the processor 11 extracts image frames at preset intervals from the video acquired by the rear camera 18, and inputs the extracted images into the CNN to detect pedestrian cautions. The processor 11 preferably extracts an image frame every 100 ms from the video, but is not necessarily limited to this.

Figure 4 is a flowchart showing the process of learning pedestrian cautions in the pedestrian caution notification method. Referring to FIG. 4, the learning of the artificial neural network in one embodiment of the present invention includes a preprocessing step (S110) in which a number of images prepared for learning walking cautions are processed to input them into the CNN, and the preprocessed images are input to the CNN. It may include a step (S120) of learning walking precautions by inputting them. This learning of walking precautions can be performed in the learning server 30.

First, prepare a number of images depicting pedestrian warning signs such as stairs, crosswalks, street trees, and sidewalk boundaries. These images can be taken directly or collected through Internet sites.

Bounding box work is performed on the learning or training images prepared in this way. This process may include a step (S110) of displaying a bounding box on a walking caution object in the prepared images and labeling information on the walking caution object indicated by the bounding box. These operations include labellmg, CVAT, LabelMe, Labelbox, VoTT, imglab, YOLO Mark, PixelAnotaionTool, OpenLabeling, imagetagger, Alturos.ImageAnnotation, DeepLabel, MedTagger, Turktools, Pixie, OpenLabeler, Anno-Mage, CATMAID, makesense.ai, LOST, This can be done manually or automatically using known applications such as annotorious and sloth. This preprocessed data can be used as a ground truth for training an artificial neural network.

The learning step (S120) is a step of learning walking precautions by inputting the labeled images prepared in step S110 into an artificial neural network. As described above, the artificial neural network may be CNN, but it is not necessarily limited to this, and other known ANNs are also possible.

Figure 6 is a bounding box loss graph verifying the learning results of the neural network. Figure 7 is a class accuracy graph verifying the learning results of the neural network.

After passing the neural network and verifying the learned data, different formulas were applied to the Bounding Box's Loss and Class Accuracy. Mean Squared Error was applied to the Bounding Box, and Categorical Crossentropy was applied to the Class.

As shown in Figure 6, Training Loss is 5.3 and Validation Loss is 103.1. The reason for this is that sometimes the entire object, such as a stair or crosswalk, is included in the bonding box, and in the case of a rotating or curved staircase, only a part of it is included in the bounding box. If you check the training curve, you can see that loss is rapidly decreasing due to relatively clear classification of walking cautions.

And, referring to Figure 7, Class Accuracy converges to 1 during training and shows an accuracy of 0.978 during validation. These results can also be seen in Figure 8.

Meanwhile, when learning or training in step S110 is completed, the training results can be transferred to a smart phone and applied to detecting pedestrian cautions. Since portable electronic products (especially smartphones) still lack computing power or resources compared to machines dedicated to deep learning, they detect pedestrian cautions based on the results learned from the learning server 30. do.

In particular, smartphones have limited resources, so they have the disadvantage of not being able to recognize and express images at the same time. Therefore, it is appropriate to recognize walking cautions using an artificial neural network in the background and display only the results to the user.

Figure 5 is a flowchart showing a pedestrian warning notification method. The steps shown in FIG. 5 may be performed in portable electronic device 10. Specifically, the pedestrian caution notification method according to an embodiment of the present invention includes the steps of determining whether the tilt of the portable electronic device 10 obtained by the posture detection unit 22 falls within a preset tilt range (S201 to S202) If the inclination falls within the inclination range, a step of determining whether or not to walk based on the detection value of the position detection unit 21 (S203 to S204); and, in the step of determining whether to walk, it is determined that the person is walking. In this case, in the step of detecting whether the display 16 provided on the front of the portable electronic device 10 is running (S205), and if it is determined that the display 16 is running in the step of detecting whether the display is running, the front Steps (S207 to S208) of detecting the user's gaze based on the image acquired by the camera 17 and acquiring surrounding images by driving the rear camera 18 based on the detection result, and acquiring the surrounding image using the rear camera 18 It may include a step (S209 to S211) of detecting a walking caution object by inputting the surrounding image acquired through ) into an artificial neural network learned based on the walking caution object learning data.

More specifically, the walking caution notification method according to an embodiment of the present invention includes a posture detection step (S01), a tilt determination step (S202), a movement detection step (S203), a gait determination step (S204), and a display drive detection step. (S205), user gaze detection step (S206), rear camera operation step (S207), surrounding image acquisition step (S208), walking caution object detection step (S209), walking caution object determination step (S210), warning notification step ( S211) may be included.

In the posture detection step (S201), the tilt A of the portable electronic device 10 may be detected by the posture detection unit 22. The tilt (A) may be defined as the angle at which the portable electronic device 10 is tilted with respect to the horizontal. For example, when the portable electronic device 10 is placed horizontally (i.e., when the screen is facing vertically upward), it can be defined as 0 degrees.

In the tilt determination step (S202), the processor 11 determines whether the tilt (A) of the portable electronic device 10 detected through the posture detection unit 22 falls within the preset tilt range (A1 to A2). This may be a step. Here, the tilt range is the angle at which the portable electronic device 10 is tilted relative to the horizontal when the user is looking at the screen of the portable electronic device 10 while walking, and is preferably between 15 degrees and 15 degrees. 60 degrees, but it is not necessarily limited to this.

Meanwhile, if it is determined in step S202 that the slope A does not fall within the slope range ('No' in S202), the processor 11 may control the rear camera 18 to be in an off state. There is (S212). Here, 'stop driving' means changing the operation to stop if the rear camera 18 is currently running (ON), and to continue to stop driving if the rear camera 18 is currently stopped. .

If it is determined that the slope A falls within the slope range A1 to A2 in the slope determination step S202, the movement detection step S203 may be performed.

The movement detection step (S203) may include the processor 11 determining the movement speed of the portable electronic device 10 based on the value detected through the position detection unit 21. For example, the processor 11 may determine the moving speed based on location information obtained through the GPS module. Alternatively, the processor 11 may detect movement speed based on information obtained through the posture detection unit 22.

The walking determination step (S204) may include the processor 11 determining whether the user is currently walking based on the movement speed (V) obtained in step S203. The processor 11 may determine that the user is currently walking if the movement speed (V) obtained in step S203 falls within the preset walking speed range (V1 to V2).

Meanwhile, if it is determined in step S204 that the movement speed (V) does not fall within the walking speed range ('No' in S204), the processor 11 controls the rear camera 18 to be in the operation stop (OFF) state. You can do it (S212).

In step S212, the processor 11 can control not only the rear camera 18 but also the front camera 17 to be in an off state.

Depending on the embodiment, in step S204, the processor 11 is in a state in which the movement speed (V) does not fall within the walking speed range for a preset time (for example, a stationary state in which the movement speed (V) is 0). Only when the operation is maintained longer than a set time, at least one of the front camera 17 and the rear camera 18 can be controlled to stop operation (OFF).

In cases where a walking user frequently stops walking for a short period of time, there is an effect of preventing excessive power consumption by repeating driving and stopping of the front and/or rear cameras 17 and 18 each time.

The user's walking speed range may be defined within the range of +-3Km/h based on 4Km/h. Preferably, the walking speed range may be defined in a section including 4Km/h.

If it is determined in the walking determination step (S204) that the movement speed (V) falls within the walking speed range (V1 to V2), the display driving detection step (S205) may be performed.

The display driving detection step (S205) is a step of detecting whether the display 16 is currently being driven (ON), and can be detected by the processor 11 based on the input or output of the display 16.

Meanwhile, if the display 16 is determined to be in a non-driving state (OFF) in step S205 ('No' in S205), the processor 11 controls the rear camera 18 to be in a non-driving state (OFF). (S212).

If it is determined that the display 16 is driving in the display driving detection step (S205), the user gaze detection step (S206) may be performed.

The user gaze detection step (S206) may be a step for estimating whether the user is looking at the screen of the display 16.

Step S206 may include detecting the user's gaze based on the image acquired by the front camera 17. Specifically, in step S206, the processor 11 may include the step of acquiring an image through the front camera 17 and detecting the user's pupils from the obtained image. In step S206, the processor 11 may extract features corresponding to the user's pupils from the acquired image. However, it is not limited to this, and step S206 may be performed based on another known eye tracking technique.

In step S206, if it is determined that the user is looking at the screen of the display 16, the processor 11 can drive the rear camera 18 (S207).

Meanwhile, if it is determined in step S206 that the user is not looking at the screen of the display 16 ('No' in S206), the processor 11 may control step S212 to be performed. As described above, in step S212, at least one of the front camera 17 and the rear camera 18 may be controlled to be in an off state.

Depending on the embodiment, in step S206, the processor 11 performs step S212 only when the state in which no gaze is detected (i.e., the state in which the user is not looking at the screen of the display 16) is maintained for a preset time or more. can be controlled to be carried out.

In cases where a walking user frequently takes his/her gaze off the screen of the display (16) for a short period of time, excessive power consumption can be prevented by repeatedly starting and stopping the front and/or rear cameras (17, 18) each time. There is a possible effect.

The surrounding image acquisition step (S208) is a step of acquiring the surrounding image using the rear camera 18 operated in step S207. At this time, display of the screen through the display 16 may be performed in a foreground process, and acquisition and processing of surrounding images through the rear camera 18 may be performed in a background process.

In step S209, the image acquired in step S208 is input to an artificial neural network learned based on the walking caution object learning data to detect the walking caution object. The artificial neural network is preferably a CNN, and may be constructed based on results learned by the learning server 30 as described above.

Here, the CNN-based image detection process can be performed in substantially the same way as the classification of pedestrian objects during the learning or training process in the learning server 30 described above.

Specifically, the image acquired in step S20 is input to the CNN, and pedestrian caution objects are detected based on the results obtained as the output of the artificial neural network through an artificial intelligence-based classification task (S209).

Afterwards, when it is determined that a pedestrian caution object has been detected (S210), a step of notifying the user of the electronic device 10 may be performed so that he/she can recognize it (S211).

The notification in step S211 may be made through at least one of the audio output unit 19 and the display 16. Since the process of detecting a pedestrian caution object is performed while the pedestrian is concentrating on the screen of the display 16, a message indicating that a pedestrian caution object has been found is output through the screen of the display 16, or an image of the pedestrian caution object is output. It is desirable to display .

Figure 9 is a structural diagram of the MobileNet standard model (MobileNet Body Architecture). Figure 10 is a structural diagram of a pedestrian warning object detection model.

Referring to Figures 9 and 10, the artificial neural network for detecting pedestrian cautions according to an embodiment of the present invention can be designed to be usable on mobile devices, and in this respect, the number of parameters can be dramatically reduced while simultaneously Under the premise that performance should not deteriorate, we adopted a neural network being studied in academia and the business world and implemented transfer-learning.

In more detail, the 'Artificial Neural Network for Detecting Pedestrian Observations' is a network of n layers that constitute the Mobilenet v1 standard model (see Figure 9) announced by Google Inc., starting from the n-2nd. The final nth layer was removed, and classification and object detection were performed using the results of the n-3th layer. Here, classification uses a global average pooling layer, and object detection can use a convolutional layer. For reference, what is shown in Figure 9 corresponds to the standard specifications, and in actual use, the stride, filter shape, and input size can be changed appropriately. As a result of various experiments, the applicant found that the fastest learning occurred and good results were obtained when the image input to the neural network was 128*128*3, and this was applied to the examples.

As shown in Figures 9 and 10, 'Artificial Neural Network for Detecting Pedestrian Observations' does not use the Average Pooling Layer, Fully Connected Layer, and Softmax Layer of the mobile network, and the results from the 13th Convolutional Layer are common. It may include an input classifier and object detector.

Here, the classifier classifies pedestrian objects detected in the image, includes one Average Pooling Layer and at least one Dense Layer, and can be configured to obtain class information. As in the example, the number of dense layers is preferably four, but it is not necessarily limited to this.

The object detector detects pedestrian objects in the image and includes at least one convolutional layer, at least one flatten layer, and at least one dense layer, and may be configured to obtain information about the bounding box. In the embodiment, there is one Convolutional Layer, Flatten Layer, and Dense Layer each, but it is not necessarily limited thereto.

Meanwhile, during actual learning, MobileNet is not learned, and only newly added layers for classification/object detection can be learned.

Meanwhile, Tensorflow-lite can be used to apply the above-described 'artificial neural network for detecting pedestrian cautions' to portable electronic devices (e.g., smartphones). In Tensorflow-lite, image frames are extracted in 100ms units while images are acquired by the rear camera 18 in the background thread, and the extracted images can be sent to the detection thread. In Detection Thread, pedestrian cautions can be determined based on learning data already stored in Tensorflow Lite (i.e., what was learned in the above-mentioned learning process).

The computer-readable recording medium according to the present invention records a program for causing a developing device to execute the above-described developing method. In this specification, the computer-readable recording medium includes a non-transitory computer recording medium (e.g., various main memory devices or auxiliary memory devices) or a radio signal (transitory computer recording medium) (e.g. For example, data signals available through a network) are included.

Claims (5)

Step (a) of determining whether the tilt of the portable electronic device obtained by the posture detection unit falls within a preset tilt range;
Step (b) of determining whether to walk based on the detection value of the position sensor when the slope is within the slope range;
If it is determined that the person is walking in step (b), step (c) of detecting whether the display provided on the front of the portable electronic device is running;
If it is determined that the display is running in step (c), the user's gaze is detected based on the image acquired by the front camera, and the rear camera is driven based on the detection result to obtain surrounding images. Step (d); and
A method of controlling a portable electronic device including step (e) of detecting a pedestrian caution object by inputting the surrounding image acquired through the rear camera into an artificial neural network learned based on the pedestrian caution object learning data. According to claim 1,
The slope range in step (a) is,
A method of controlling a portable electronic device defined as an inclination of the portable electronic device between 15 and 60 degrees relative to the horizontal. According to claim 1,
In step (b),
Obtaining a moving speed based on the detection value of the position sensor; and
A method of controlling a portable electronic device, including the step of determining that the mobile electronic device is walking when the moving speed falls within a predetermined walking speed range defined between 1 and 7 km/h. According to claim 1,
Gaze detection in step (d) above is,
A method of controlling a portable electronic device including detecting the user's pupils from the image acquired through the front camera. A computer-readable recording medium recording a program for executing the method according to any one of claims 1 to 4 in a portable electronic device.